Mercurial > public > mercurial-scm > hg-stable

annotate mercurial/verify.py @ 42198:496ac8a02380

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
verify: introduce an experimental --full flag The flag currently has no effect, see next changeset for details. We introduce the flag as experimental to keep the freedom of changing our mind on the final UI. Note: this patch highlight a small but in `hg help`. An option section is generated even if no option are visible.
author Pierre-Yves David <pierre-yves.david@octobus.net>
date Wed, 17 Apr 2019 01:11:09 +0200
parents 57539e5ea2e0
children 7755b89cadaf
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
2778
fdc232d8a193 Move repo.verify
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
1 # verify.py - repository integrity checking for Mercurial
fdc232d8a193 Move repo.verify
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
2 #
4635
63b9d2deed48 Updated copyright notices and add "and others" to "hg version"
Thomas Arendsen Hein <thomas@intevation.de>
parents: 4395
diff changeset
3 # Copyright 2006, 2007 Matt Mackall <mpm@selenic.com>
2778
fdc232d8a193 Move repo.verify
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
4 #
8225
46293a0c7e9f updated license to be explicit about GPL version 2
Martin Geisler <mg@lazybytes.net>
parents: 8209
diff changeset
5 # This software may be used and distributed according to the terms of the
10263
25e572394f5c Update license to GPLv2+
Matt Mackall <mpm@selenic.com>
parents: 9690
diff changeset
6 # GNU General Public License version 2 or any later version.
2778
fdc232d8a193 Move repo.verify
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
7
25991
d21d1774c73b verify: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25846
diff changeset
8 from __future__ import absolute_import
d21d1774c73b verify: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25846
diff changeset
9
17860
a45b33f12627 verify: fix all doubled-slash sites (issue3665)
Bryan O'Sullivan <bryano@fb.com>
parents: 17851
diff changeset
10 import os
25991
d21d1774c73b verify: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25846
diff changeset
11
d21d1774c73b verify: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25846
diff changeset
12 from .i18n import _
d21d1774c73b verify: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25846
diff changeset
13 from .node import (
d21d1774c73b verify: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25846
diff changeset
14 nullid,
d21d1774c73b verify: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25846
diff changeset
15 short,
d21d1774c73b verify: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25846
diff changeset
16 )
d21d1774c73b verify: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25846
diff changeset
17
d21d1774c73b verify: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25846
diff changeset
18 from . import (
d21d1774c73b verify: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25846
diff changeset
19 error,
35585
35fb3367f72d py3: use pycompat.bytestr() instead of str()
Pulkit Goyal <7895pulkit@gmail.com>
parents: 33499
diff changeset
20 pycompat,
25991
d21d1774c73b verify: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25846
diff changeset
21 revlog,
d21d1774c73b verify: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25846
diff changeset
22 util,
d21d1774c73b verify: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25846
diff changeset
23 )
2778
fdc232d8a193 Move repo.verify
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
24
42197
57539e5ea2e0 verify: introduce a notion of "level"
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41882
diff changeset
25 VERIFY_DEFAULT = 0
42198
496ac8a02380 verify: introduce an experimental --full flag
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 42197
diff changeset
26 VERIFY_FULL = 1
42197
57539e5ea2e0 verify: introduce a notion of "level"
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41882
diff changeset
27
57539e5ea2e0 verify: introduce a notion of "level"
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41882
diff changeset
28 def verify(repo, level=None):
27849
900d36a3e4dd with: use context manager in verify
Bryan O'Sullivan <bryano@fb.com>
parents: 27695
diff changeset
29 with repo.lock():
42197
57539e5ea2e0 verify: introduce a notion of "level"
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41882
diff changeset
30 v = verifier(repo, level)
57539e5ea2e0 verify: introduce a notion of "level"
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41882
diff changeset
31 return v.verify()
4915
97b734fb9c6f Use try/finally pattern to cleanup locks and transactions
Matt Mackall <mpm@selenic.com>
parents: 4635
diff changeset
32
17860
a45b33f12627 verify: fix all doubled-slash sites (issue3665)
Bryan O'Sullivan <bryano@fb.com>
parents: 17851
diff changeset
33 def _normpath(f):
a45b33f12627 verify: fix all doubled-slash sites (issue3665)
Bryan O'Sullivan <bryano@fb.com>
parents: 17851
diff changeset
34 # under hg < 2.4, convert didn't sanitize paths properly, so a
a45b33f12627 verify: fix all doubled-slash sites (issue3665)
Bryan O'Sullivan <bryano@fb.com>
parents: 17851
diff changeset
35 # converted repo may contain repeated slashes
a45b33f12627 verify: fix all doubled-slash sites (issue3665)
Bryan O'Sullivan <bryano@fb.com>
parents: 17851
diff changeset
36 while '//' in f:
a45b33f12627 verify: fix all doubled-slash sites (issue3665)
Bryan O'Sullivan <bryano@fb.com>
parents: 17851
diff changeset
37 f = f.replace('//', '/')
a45b33f12627 verify: fix all doubled-slash sites (issue3665)
Bryan O'Sullivan <bryano@fb.com>
parents: 17851
diff changeset
38 return f
a45b33f12627 verify: fix all doubled-slash sites (issue3665)
Bryan O'Sullivan <bryano@fb.com>
parents: 17851
diff changeset
39
27443
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
40 class verifier(object):
42197
57539e5ea2e0 verify: introduce a notion of "level"
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41882
diff changeset
41 def __init__(self, repo, level=None):
27444
6647401858ab verify: move widely used variables into class members
Durham Goode <durham@fb.com>
parents: 27443
diff changeset
42 self.repo = repo.unfiltered()
6647401858ab verify: move widely used variables into class members
Durham Goode <durham@fb.com>
parents: 27443
diff changeset
43 self.ui = repo.ui
39943
fec944719324 narrow: move support for `hg verify` into core
Martin von Zweigbergk <martinvonz@google.com>
parents: 39877
diff changeset
44 self.match = repo.narrowmatch()
42197
57539e5ea2e0 verify: introduce a notion of "level"
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41882
diff changeset
45 if level is None:
57539e5ea2e0 verify: introduce a notion of "level"
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41882
diff changeset
46 level = VERIFY_DEFAULT
57539e5ea2e0 verify: introduce a notion of "level"
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41882
diff changeset
47 self._level = level
27444
6647401858ab verify: move widely used variables into class members
Durham Goode <durham@fb.com>
parents: 27443
diff changeset
48 self.badrevs = set()
27453
8462d7f2c4fe verify: clean up weird error/warning lists
Matt Mackall <mpm@selenic.com>
parents: 27450
diff changeset
49 self.errors = 0
8462d7f2c4fe verify: clean up weird error/warning lists
Matt Mackall <mpm@selenic.com>
parents: 27450
diff changeset
50 self.warnings = 0
27444
6647401858ab verify: move widely used variables into class members
Durham Goode <durham@fb.com>
parents: 27443
diff changeset
51 self.havecl = len(repo.changelog) > 0
39272
73cf21b2e8a6 manifest: add getstorage() to manifestlog and use it globally
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38403
diff changeset
52 self.havemf = len(repo.manifestlog.getstorage(b'')) > 0
27444
6647401858ab verify: move widely used variables into class members
Durham Goode <durham@fb.com>
parents: 27443
diff changeset
53 self.revlogv1 = repo.changelog.version != revlog.REVLOGV0
37302
00f18dd1d3d6 verify: remove dependence on repo.changectx()
Martin von Zweigbergk <martinvonz@google.com>
parents: 36606
diff changeset
54 self.lrugetctx = util.lrucachefunc(repo.__getitem__)
27444
6647401858ab verify: move widely used variables into class members
Durham Goode <durham@fb.com>
parents: 27443
diff changeset
55 self.refersmf = False
27445
cc178057ab49 verify: move fncachewarned up to a class variable
Durham Goode <durham@fb.com>
parents: 27444
diff changeset
56 self.fncachewarned = False
32328
a2ab9ebcd85b verify: add a config option to skip certain flag processors
Jun Wu <quark@fb.com>
parents: 32290
diff changeset
57 # developer config: verify.skipflags
a2ab9ebcd85b verify: add a config option to skip certain flag processors
Jun Wu <quark@fb.com>
parents: 32290
diff changeset
58 self.skipflags = repo.ui.configint('verify', 'skipflags')
37417
76d2115cb817 verify: allow suppressing warnings about extra files
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37410
diff changeset
59 self.warnorphanstorefiles = True
27444
6647401858ab verify: move widely used variables into class members
Durham Goode <durham@fb.com>
parents: 27443
diff changeset
60
41867
c66037fb1bc5 verify: make the `warn` method private
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41866
diff changeset
61 def _warn(self, msg):
41866
a58748300e61 verify: document the `warn` method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 40437
diff changeset
62 """record a "warning" level issue"""
27446
6b2c1a1871a6 verify: move warn() to a class level function
Durham Goode <durham@fb.com>
parents: 27445
diff changeset
63 self.ui.warn(msg + "\n")
27453
8462d7f2c4fe verify: clean up weird error/warning lists
Matt Mackall <mpm@selenic.com>
parents: 27450
diff changeset
64 self.warnings += 1
27446
6b2c1a1871a6 verify: move warn() to a class level function
Durham Goode <durham@fb.com>
parents: 27445
diff changeset
65
41869
7eaf4b1ac2a3 verify: make `err` a private method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41868
diff changeset
66 def _err(self, linkrev, msg, filename=None):
41868
9534f3cb6b6c verify: document the `err` method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41867
diff changeset
67 """record a "error" level issue"""
27447
d1b91c10ce70 verify: move err() to be a class function
Durham Goode <durham@fb.com>
parents: 27446
diff changeset
68 if linkrev is not None:
d1b91c10ce70 verify: move err() to be a class function
Durham Goode <durham@fb.com>
parents: 27446
diff changeset
69 self.badrevs.add(linkrev)
36223
acc8e6e52af6 py3: use "%d" to convert integers to bytes
Pulkit Goyal <7895pulkit@gmail.com>
parents: 35585
diff changeset
70 linkrev = "%d" % linkrev
27447
d1b91c10ce70 verify: move err() to be a class function
Durham Goode <durham@fb.com>
parents: 27446
diff changeset
71 else:
d1b91c10ce70 verify: move err() to be a class function
Durham Goode <durham@fb.com>
parents: 27446
diff changeset
72 linkrev = '?'
d1b91c10ce70 verify: move err() to be a class function
Durham Goode <durham@fb.com>
parents: 27446
diff changeset
73 msg = "%s: %s" % (linkrev, msg)
d1b91c10ce70 verify: move err() to be a class function
Durham Goode <durham@fb.com>
parents: 27446
diff changeset
74 if filename:
d1b91c10ce70 verify: move err() to be a class function
Durham Goode <durham@fb.com>
parents: 27446
diff changeset
75 msg = "%s@%s" % (filename, msg)
d1b91c10ce70 verify: move err() to be a class function
Durham Goode <durham@fb.com>
parents: 27446
diff changeset
76 self.ui.warn(" " + msg + "\n")
27453
8462d7f2c4fe verify: clean up weird error/warning lists
Matt Mackall <mpm@selenic.com>
parents: 27450
diff changeset
77 self.errors += 1
27447
d1b91c10ce70 verify: move err() to be a class function
Durham Goode <durham@fb.com>
parents: 27446
diff changeset
78
41871
cfe08588d711 verify: make the `exc` method private
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41870
diff changeset
79 def _exc(self, linkrev, msg, inst, filename=None):
41870
5df8475c5343 verify: document the `exc` method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41869
diff changeset
80 """record exception raised during the verify process"""
36606
d85ef895d5f6 verify: fix exception formatting bug in Python 3
Augie Fackler <augie@google.com>
parents: 36372
diff changeset
81 fmsg = pycompat.bytestr(inst)
d85ef895d5f6 verify: fix exception formatting bug in Python 3
Augie Fackler <augie@google.com>
parents: 36372
diff changeset
82 if not fmsg:
d85ef895d5f6 verify: fix exception formatting bug in Python 3
Augie Fackler <augie@google.com>
parents: 36372
diff changeset
83 fmsg = pycompat.byterepr(inst)
41869
7eaf4b1ac2a3 verify: make `err` a private method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41868
diff changeset
84 self._err(linkrev, "%s: %s" % (msg, fmsg), filename)
27448
f4f2179077cb verify: move exc() function onto class
Durham Goode <durham@fb.com>
parents: 27447
diff changeset
85
41879
1f412223f5bb verify: rename the `checklog` to `_checkrevlog`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41878
diff changeset
86 def _checkrevlog(self, obj, name, linkrev):
41878
08d977451f26 verify: document the `checklog` method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41876
diff changeset
87 """verify high level property of a revlog
08d977451f26 verify: document the `checklog` method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41876
diff changeset
88
08d977451f26 verify: document the `checklog` method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41876
diff changeset
89 - revlog is present,
08d977451f26 verify: document the `checklog` method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41876
diff changeset
90 - revlog is non-empty,
08d977451f26 verify: document the `checklog` method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41876
diff changeset
91 - sizes (index and data) are correct,
08d977451f26 verify: document the `checklog` method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41876
diff changeset
92 - revlog's format version is correct.
08d977451f26 verify: document the `checklog` method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41876
diff changeset
93 """
27642
f6457349985b verify: move checklog() onto class
Durham Goode <durham@fb.com>
parents: 27453
diff changeset
94 if not len(obj) and (self.havecl or self.havemf):
41869
7eaf4b1ac2a3 verify: make `err` a private method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41868
diff changeset
95 self._err(linkrev, _("empty or missing %s") % name)
27642
f6457349985b verify: move checklog() onto class
Durham Goode <durham@fb.com>
parents: 27453
diff changeset
96 return
f6457349985b verify: move checklog() onto class
Durham Goode <durham@fb.com>
parents: 27453
diff changeset
97
f6457349985b verify: move checklog() onto class
Durham Goode <durham@fb.com>
parents: 27453
diff changeset
98 d = obj.checksize()
f6457349985b verify: move checklog() onto class
Durham Goode <durham@fb.com>
parents: 27453
diff changeset
99 if d[0]:
f6457349985b verify: move checklog() onto class
Durham Goode <durham@fb.com>
parents: 27453
diff changeset
100 self.err(None, _("data length off by %d bytes") % d[0], name)
f6457349985b verify: move checklog() onto class
Durham Goode <durham@fb.com>
parents: 27453
diff changeset
101 if d[1]:
f6457349985b verify: move checklog() onto class
Durham Goode <durham@fb.com>
parents: 27453
diff changeset
102 self.err(None, _("index contains %d extra bytes") % d[1], name)
f6457349985b verify: move checklog() onto class
Durham Goode <durham@fb.com>
parents: 27453
diff changeset
103
f6457349985b verify: move checklog() onto class
Durham Goode <durham@fb.com>
parents: 27453
diff changeset
104 if obj.version != revlog.REVLOGV0:
f6457349985b verify: move checklog() onto class
Durham Goode <durham@fb.com>
parents: 27453
diff changeset
105 if not self.revlogv1:
41867
c66037fb1bc5 verify: make the `warn` method private
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41866
diff changeset
106 self._warn(_("warning: `%s' uses revlog format 1") % name)
27642
f6457349985b verify: move checklog() onto class
Durham Goode <durham@fb.com>
parents: 27453
diff changeset
107 elif self.revlogv1:
41867
c66037fb1bc5 verify: make the `warn` method private
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41866
diff changeset
108 self._warn(_("warning: `%s' uses revlog format 0") % name)
27642
f6457349985b verify: move checklog() onto class
Durham Goode <durham@fb.com>
parents: 27453
diff changeset
109
41876
9e737ca539f6 verify: make `checkentry` a private method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41875
diff changeset
110 def _checkentry(self, obj, i, node, seen, linkrevs, f):
41875
00c9fde75c1a verify: document the `checkentry` method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41874
diff changeset
111 """verify a single revlog entry
00c9fde75c1a verify: document the `checkentry` method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41874
diff changeset
112
00c9fde75c1a verify: document the `checkentry` method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41874
diff changeset
113 arguments are:
00c9fde75c1a verify: document the `checkentry` method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41874
diff changeset
114 - obj: the source revlog
00c9fde75c1a verify: document the `checkentry` method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41874
diff changeset
115 - i: the revision number
00c9fde75c1a verify: document the `checkentry` method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41874
diff changeset
116 - node: the revision node id
00c9fde75c1a verify: document the `checkentry` method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41874
diff changeset
117 - seen: nodes previously seen for this revlog
00c9fde75c1a verify: document the `checkentry` method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41874
diff changeset
118 - linkrevs: [changelog-revisions] introducing "node"
00c9fde75c1a verify: document the `checkentry` method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41874
diff changeset
119 - f: string label ("changelog", "manifest", or filename)
00c9fde75c1a verify: document the `checkentry` method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41874
diff changeset
120
00c9fde75c1a verify: document the `checkentry` method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41874
diff changeset
121 Performs the following checks:
00c9fde75c1a verify: document the `checkentry` method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41874
diff changeset
122 - linkrev points to an existing changelog revision,
00c9fde75c1a verify: document the `checkentry` method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41874
diff changeset
123 - linkrev points to a changelog revision that introduces this revision,
00c9fde75c1a verify: document the `checkentry` method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41874
diff changeset
124 - linkrev points to the lowest of these changesets,
00c9fde75c1a verify: document the `checkentry` method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41874
diff changeset
125 - both parents exist in the revlog,
00c9fde75c1a verify: document the `checkentry` method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41874
diff changeset
126 - the revision is not duplicated.
00c9fde75c1a verify: document the `checkentry` method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41874
diff changeset
127
00c9fde75c1a verify: document the `checkentry` method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41874
diff changeset
128 Return the linkrev of the revision (or None for changelog's revisions).
00c9fde75c1a verify: document the `checkentry` method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41874
diff changeset
129 """
27643
62ce86fcfd06 verify: move checkentry() to be a class function
Durham Goode <durham@fb.com>
parents: 27642
diff changeset
130 lr = obj.linkrev(obj.rev(node))
62ce86fcfd06 verify: move checkentry() to be a class function
Durham Goode <durham@fb.com>
parents: 27642
diff changeset
131 if lr < 0 or (self.havecl and lr not in linkrevs):
62ce86fcfd06 verify: move checkentry() to be a class function
Durham Goode <durham@fb.com>
parents: 27642
diff changeset
132 if lr < 0 or lr >= len(self.repo.changelog):
62ce86fcfd06 verify: move checkentry() to be a class function
Durham Goode <durham@fb.com>
parents: 27642
diff changeset
133 msg = _("rev %d points to nonexistent changeset %d")
62ce86fcfd06 verify: move checkentry() to be a class function
Durham Goode <durham@fb.com>
parents: 27642
diff changeset
134 else:
62ce86fcfd06 verify: move checkentry() to be a class function
Durham Goode <durham@fb.com>
parents: 27642
diff changeset
135 msg = _("rev %d points to unexpected changeset %d")
41869
7eaf4b1ac2a3 verify: make `err` a private method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41868
diff changeset
136 self._err(None, msg % (i, lr), f)
27643
62ce86fcfd06 verify: move checkentry() to be a class function
Durham Goode <durham@fb.com>
parents: 27642
diff changeset
137 if linkrevs:
62ce86fcfd06 verify: move checkentry() to be a class function
Durham Goode <durham@fb.com>
parents: 27642
diff changeset
138 if f and len(linkrevs) > 1:
62ce86fcfd06 verify: move checkentry() to be a class function
Durham Goode <durham@fb.com>
parents: 27642
diff changeset
139 try:
62ce86fcfd06 verify: move checkentry() to be a class function
Durham Goode <durham@fb.com>
parents: 27642
diff changeset
140 # attempt to filter down to real linkrevs
62ce86fcfd06 verify: move checkentry() to be a class function
Durham Goode <durham@fb.com>
parents: 27642
diff changeset
141 linkrevs = [l for l in linkrevs
62ce86fcfd06 verify: move checkentry() to be a class function
Durham Goode <durham@fb.com>
parents: 27642
diff changeset
142 if self.lrugetctx(l)[f].filenode() == node]
62ce86fcfd06 verify: move checkentry() to be a class function
Durham Goode <durham@fb.com>
parents: 27642
diff changeset
143 except Exception:
62ce86fcfd06 verify: move checkentry() to be a class function
Durham Goode <durham@fb.com>
parents: 27642
diff changeset
144 pass
41867
c66037fb1bc5 verify: make the `warn` method private
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41866
diff changeset
145 self._warn(_(" (expected %s)") % " ".join
c66037fb1bc5 verify: make the `warn` method private
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41866
diff changeset
146 (map(pycompat.bytestr, linkrevs)))
27643
62ce86fcfd06 verify: move checkentry() to be a class function
Durham Goode <durham@fb.com>
parents: 27642
diff changeset
147 lr = None # can't be trusted
62ce86fcfd06 verify: move checkentry() to be a class function
Durham Goode <durham@fb.com>
parents: 27642
diff changeset
148
62ce86fcfd06 verify: move checkentry() to be a class function
Durham Goode <durham@fb.com>
parents: 27642
diff changeset
149 try:
62ce86fcfd06 verify: move checkentry() to be a class function
Durham Goode <durham@fb.com>
parents: 27642
diff changeset
150 p1, p2 = obj.parents(node)
62ce86fcfd06 verify: move checkentry() to be a class function
Durham Goode <durham@fb.com>
parents: 27642
diff changeset
151 if p1 not in seen and p1 != nullid:
41869
7eaf4b1ac2a3 verify: make `err` a private method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41868
diff changeset
152 self._err(lr, _("unknown parent 1 %s of %s") %
27643
62ce86fcfd06 verify: move checkentry() to be a class function
Durham Goode <durham@fb.com>
parents: 27642
diff changeset
153 (short(p1), short(node)), f)
62ce86fcfd06 verify: move checkentry() to be a class function
Durham Goode <durham@fb.com>
parents: 27642
diff changeset
154 if p2 not in seen and p2 != nullid:
41869
7eaf4b1ac2a3 verify: make `err` a private method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41868
diff changeset
155 self._err(lr, _("unknown parent 2 %s of %s") %
27643
62ce86fcfd06 verify: move checkentry() to be a class function
Durham Goode <durham@fb.com>
parents: 27642
diff changeset
156 (short(p2), short(node)), f)
62ce86fcfd06 verify: move checkentry() to be a class function
Durham Goode <durham@fb.com>
parents: 27642
diff changeset
157 except Exception as inst:
41871
cfe08588d711 verify: make the `exc` method private
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41870
diff changeset
158 self._exc(lr, _("checking parents of %s") % short(node), inst, f)
27643
62ce86fcfd06 verify: move checkentry() to be a class function
Durham Goode <durham@fb.com>
parents: 27642
diff changeset
159
62ce86fcfd06 verify: move checkentry() to be a class function
Durham Goode <durham@fb.com>
parents: 27642
diff changeset
160 if node in seen:
41869
7eaf4b1ac2a3 verify: make `err` a private method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41868
diff changeset
161 self._err(lr, _("duplicate revision %d (%d)") % (i, seen[node]), f)
27643
62ce86fcfd06 verify: move checkentry() to be a class function
Durham Goode <durham@fb.com>
parents: 27642
diff changeset
162 seen[node] = i
62ce86fcfd06 verify: move checkentry() to be a class function
Durham Goode <durham@fb.com>
parents: 27642
diff changeset
163 return lr
62ce86fcfd06 verify: move checkentry() to be a class function
Durham Goode <durham@fb.com>
parents: 27642
diff changeset
164
27444
6647401858ab verify: move widely used variables into class members
Durham Goode <durham@fb.com>
parents: 27443
diff changeset
165 def verify(self):
41872
e8c4a9f5b986 verify: minimal documentation for `verifier.verify`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41871
diff changeset
166 """verify the content of the Mercurial repository
e8c4a9f5b986 verify: minimal documentation for `verifier.verify`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41871
diff changeset
167
e8c4a9f5b986 verify: minimal documentation for `verifier.verify`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41871
diff changeset
168 This method run all verifications, displaying issues as they are found.
e8c4a9f5b986 verify: minimal documentation for `verifier.verify`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41871
diff changeset
169
41873
567892b4306c verify: explicitly return 0 if no error are encountered
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41872
diff changeset
170 return 1 if any error have been encountered, 0 otherwise."""
41874
4da2261e949b verify: add some inline documentation to the top level `verify` method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41873
diff changeset
171 # initial validation and generic report
27444
6647401858ab verify: move widely used variables into class members
Durham Goode <durham@fb.com>
parents: 27443
diff changeset
172 repo = self.repo
27443
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
173 ui = repo.ui
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
174 if not repo.url().startswith('file:'):
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
175 raise error.Abort(_("cannot verify bundle or remote repos"))
6752
e79a8f36c2a5 verify: lots of refactoring
Matt Mackall <mpm@selenic.com>
parents: 6751
diff changeset
176
27443
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
177 if os.path.exists(repo.sjoin("journal")):
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
178 ui.warn(_("abandoned transaction found - run hg recover\n"))
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
179
27648
e72e669dd51f verify: get rid of some unnecessary local variables
Durham Goode <durham@fb.com>
parents: 27647
diff changeset
180 if ui.verbose or not self.revlogv1:
27443
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
181 ui.status(_("repository uses revlog format %d\n") %
27648
e72e669dd51f verify: get rid of some unnecessary local variables
Durham Goode <durham@fb.com>
parents: 27647
diff changeset
182 (self.revlogv1 and 1 or 0))
27443
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
183
41874
4da2261e949b verify: add some inline documentation to the top level `verify` method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41873
diff changeset
184 # data verification
27695
fb0cc863d172 verify: replace "output parameters" by return values
Martin von Zweigbergk <martinvonz@google.com>
parents: 27648
diff changeset
185 mflinkrevs, filelinkrevs = self._verifychangelog()
fb0cc863d172 verify: replace "output parameters" by return values
Martin von Zweigbergk <martinvonz@google.com>
parents: 27648
diff changeset
186 filenodes = self._verifymanifest(mflinkrevs)
28111
06205989264b verify: move cross-checking of changeset/manifest out of _crosscheckfiles()
Martin von Zweigbergk <martinvonz@google.com>
parents: 28007
diff changeset
187 del mflinkrevs
06205989264b verify: move cross-checking of changeset/manifest out of _crosscheckfiles()
Martin von Zweigbergk <martinvonz@google.com>
parents: 28007
diff changeset
188 self._crosscheckfiles(filelinkrevs, filenodes)
27647
2c2858f3c1bb verify: move changelog verificaiton to its own function
Durham Goode <durham@fb.com>
parents: 27646
diff changeset
189 totalfiles, filerevisions = self._verifyfiles(filenodes, filelinkrevs)
2c2858f3c1bb verify: move changelog verificaiton to its own function
Durham Goode <durham@fb.com>
parents: 27646
diff changeset
190
41874
4da2261e949b verify: add some inline documentation to the top level `verify` method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41873
diff changeset
191 # final report
39506
f1186c292d03 verify: make output less confusing (issue5924)
Meirambek Omyrzak <meirambek77@gmail.com>
parents: 39272
diff changeset
192 ui.status(_("checked %d changesets with %d changes to %d files\n") %
f1186c292d03 verify: make output less confusing (issue5924)
Meirambek Omyrzak <meirambek77@gmail.com>
parents: 39272
diff changeset
193 (len(repo.changelog), filerevisions, totalfiles))
27647
2c2858f3c1bb verify: move changelog verificaiton to its own function
Durham Goode <durham@fb.com>
parents: 27646
diff changeset
194 if self.warnings:
2c2858f3c1bb verify: move changelog verificaiton to its own function
Durham Goode <durham@fb.com>
parents: 27646
diff changeset
195 ui.warn(_("%d warnings encountered!\n") % self.warnings)
2c2858f3c1bb verify: move changelog verificaiton to its own function
Durham Goode <durham@fb.com>
parents: 27646
diff changeset
196 if self.fncachewarned:
2c2858f3c1bb verify: move changelog verificaiton to its own function
Durham Goode <durham@fb.com>
parents: 27646
diff changeset
197 ui.warn(_('hint: run "hg debugrebuildfncache" to recover from '
2c2858f3c1bb verify: move changelog verificaiton to its own function
Durham Goode <durham@fb.com>
parents: 27646
diff changeset
198 'corrupt fncache\n'))
2c2858f3c1bb verify: move changelog verificaiton to its own function
Durham Goode <durham@fb.com>
parents: 27646
diff changeset
199 if self.errors:
2c2858f3c1bb verify: move changelog verificaiton to its own function
Durham Goode <durham@fb.com>
parents: 27646
diff changeset
200 ui.warn(_("%d integrity errors encountered!\n") % self.errors)
27648
e72e669dd51f verify: get rid of some unnecessary local variables
Durham Goode <durham@fb.com>
parents: 27647
diff changeset
201 if self.badrevs:
27647
2c2858f3c1bb verify: move changelog verificaiton to its own function
Durham Goode <durham@fb.com>
parents: 27646
diff changeset
202 ui.warn(_("(first damaged changeset appears to be %d)\n")
27648
e72e669dd51f verify: get rid of some unnecessary local variables
Durham Goode <durham@fb.com>
parents: 27647
diff changeset
203 % min(self.badrevs))
27647
2c2858f3c1bb verify: move changelog verificaiton to its own function
Durham Goode <durham@fb.com>
parents: 27646
diff changeset
204 return 1
41873
567892b4306c verify: explicitly return 0 if no error are encountered
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41872
diff changeset
205 return 0
27647
2c2858f3c1bb verify: move changelog verificaiton to its own function
Durham Goode <durham@fb.com>
parents: 27646
diff changeset
206
27695
fb0cc863d172 verify: replace "output parameters" by return values
Martin von Zweigbergk <martinvonz@google.com>
parents: 27648
diff changeset
207 def _verifychangelog(self):
41880
2f1f475e9646 verify: document `_verifychangelog`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41879
diff changeset
208 """verify the changelog of a repository
2f1f475e9646 verify: document `_verifychangelog`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41879
diff changeset
209
2f1f475e9646 verify: document `_verifychangelog`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41879
diff changeset
210 The following checks are performed:
2f1f475e9646 verify: document `_verifychangelog`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41879
diff changeset
211 - all of `_checkrevlog` checks,
2f1f475e9646 verify: document `_verifychangelog`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41879
diff changeset
212 - all of `_checkentry` checks (for each revisions),
2f1f475e9646 verify: document `_verifychangelog`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41879
diff changeset
213 - each revision can be read.
2f1f475e9646 verify: document `_verifychangelog`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41879
diff changeset
214
2f1f475e9646 verify: document `_verifychangelog`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41879
diff changeset
215 The function returns some of the data observed in the changesets as a
2f1f475e9646 verify: document `_verifychangelog`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41879
diff changeset
216 (mflinkrevs, filelinkrevs) tuples:
2f1f475e9646 verify: document `_verifychangelog`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41879
diff changeset
217 - mflinkrevs: is a { manifest-node -> [changelog-rev] } mapping
2f1f475e9646 verify: document `_verifychangelog`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41879
diff changeset
218 - filelinkrevs: is a { file-path -> [changelog-rev] } mapping
2f1f475e9646 verify: document `_verifychangelog`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41879
diff changeset
219
2f1f475e9646 verify: document `_verifychangelog`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41879
diff changeset
220 If a matcher was specified, filelinkrevs will only contains matched
2f1f475e9646 verify: document `_verifychangelog`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41879
diff changeset
221 files.
2f1f475e9646 verify: document `_verifychangelog`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41879
diff changeset
222 """
27647
2c2858f3c1bb verify: move changelog verificaiton to its own function
Durham Goode <durham@fb.com>
parents: 27646
diff changeset
223 ui = self.ui
2c2858f3c1bb verify: move changelog verificaiton to its own function
Durham Goode <durham@fb.com>
parents: 27646
diff changeset
224 repo = self.repo
30900
5249b6470de9 verify: replace _validpath() by matcher
Martin von Zweigbergk <martinvonz@google.com>
parents: 30403
diff changeset
225 match = self.match
27647
2c2858f3c1bb verify: move changelog verificaiton to its own function
Durham Goode <durham@fb.com>
parents: 27646
diff changeset
226 cl = repo.changelog
2c2858f3c1bb verify: move changelog verificaiton to its own function
Durham Goode <durham@fb.com>
parents: 27646
diff changeset
227
27443
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
228 ui.status(_("checking changesets\n"))
27695
fb0cc863d172 verify: replace "output parameters" by return values
Martin von Zweigbergk <martinvonz@google.com>
parents: 27648
diff changeset
229 mflinkrevs = {}
fb0cc863d172 verify: replace "output parameters" by return values
Martin von Zweigbergk <martinvonz@google.com>
parents: 27648
diff changeset
230 filelinkrevs = {}
27443
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
231 seen = {}
41879
1f412223f5bb verify: rename the `checklog` to `_checkrevlog`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41878
diff changeset
232 self._checkrevlog(cl, "changelog", 0)
38403
1249475f0bd6 verify: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 38402
diff changeset
233 progress = ui.makeprogress(_('checking'), unit=_('changesets'),
1249475f0bd6 verify: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 38402
diff changeset
234 total=len(repo))
27443
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
235 for i in repo:
38403
1249475f0bd6 verify: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 38402
diff changeset
236 progress.update(i)
27443
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
237 n = cl.node(i)
41876
9e737ca539f6 verify: make `checkentry` a private method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41875
diff changeset
238 self._checkentry(cl, i, n, seen, [i], "changelog")
2778
fdc232d8a193 Move repo.verify
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
239
27443
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
240 try:
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
241 changes = cl.read(n)
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
242 if changes[0] != nullid:
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
243 mflinkrevs.setdefault(changes[0], []).append(i)
27444
6647401858ab verify: move widely used variables into class members
Durham Goode <durham@fb.com>
parents: 27443
diff changeset
244 self.refersmf = True
27443
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
245 for f in changes[3]:
30900
5249b6470de9 verify: replace _validpath() by matcher
Martin von Zweigbergk <martinvonz@google.com>
parents: 30403
diff changeset
246 if match(f):
27443
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
247 filelinkrevs.setdefault(_normpath(f), []).append(i)
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
248 except Exception as inst:
27444
6647401858ab verify: move widely used variables into class members
Durham Goode <durham@fb.com>
parents: 27443
diff changeset
249 self.refersmf = True
41871
cfe08588d711 verify: make the `exc` method private
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41870
diff changeset
250 self._exc(i, _("unpacking changeset %s") % short(n), inst)
38403
1249475f0bd6 verify: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 38402
diff changeset
251 progress.complete()
27695
fb0cc863d172 verify: replace "output parameters" by return values
Martin von Zweigbergk <martinvonz@google.com>
parents: 27648
diff changeset
252 return mflinkrevs, filelinkrevs
2778
fdc232d8a193 Move repo.verify
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
253
28205
53f42c8d5f71 verify: show progress while verifying dirlogs
Martin von Zweigbergk <martinvonz@google.com>
parents: 28204
diff changeset
254 def _verifymanifest(self, mflinkrevs, dir="", storefiles=None,
38402
0ddbe03c5aaa verify: use progress helper for subdirectory progress
Martin von Zweigbergk <martinvonz@google.com>
parents: 37417
diff changeset
255 subdirprogress=None):
41881
5ad5a70df2f7 verify: document the `_verifymanifest` method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41880
diff changeset
256 """verify the manifestlog content
5ad5a70df2f7 verify: document the `_verifymanifest` method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41880
diff changeset
257
5ad5a70df2f7 verify: document the `_verifymanifest` method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41880
diff changeset
258 Inputs:
5ad5a70df2f7 verify: document the `_verifymanifest` method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41880
diff changeset
259 - mflinkrevs: a {manifest-node -> [changelog-revisions]} mapping
5ad5a70df2f7 verify: document the `_verifymanifest` method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41880
diff changeset
260 - dir: a subdirectory to check (for tree manifest repo)
5ad5a70df2f7 verify: document the `_verifymanifest` method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41880
diff changeset
261 - storefiles: set of currently "orphan" files.
5ad5a70df2f7 verify: document the `_verifymanifest` method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41880
diff changeset
262 - subdirprogress: a progress object
5ad5a70df2f7 verify: document the `_verifymanifest` method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41880
diff changeset
263
5ad5a70df2f7 verify: document the `_verifymanifest` method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41880
diff changeset
264 This function checks:
5ad5a70df2f7 verify: document the `_verifymanifest` method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41880
diff changeset
265 * all of `_checkrevlog` checks (for all manifest related revlogs)
5ad5a70df2f7 verify: document the `_verifymanifest` method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41880
diff changeset
266 * all of `_checkentry` checks (for all manifest related revisions)
5ad5a70df2f7 verify: document the `_verifymanifest` method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41880
diff changeset
267 * nodes for subdirectory exists in the sub-directory manifest
5ad5a70df2f7 verify: document the `_verifymanifest` method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41880
diff changeset
268 * each manifest entries have a file path
5ad5a70df2f7 verify: document the `_verifymanifest` method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41880
diff changeset
269 * each manifest node refered in mflinkrevs exist in the manifest log
5ad5a70df2f7 verify: document the `_verifymanifest` method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41880
diff changeset
270
5ad5a70df2f7 verify: document the `_verifymanifest` method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41880
diff changeset
271 If tree manifest is in use and a matchers is specified, only the
5ad5a70df2f7 verify: document the `_verifymanifest` method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41880
diff changeset
272 sub-directories matching it will be verified.
5ad5a70df2f7 verify: document the `_verifymanifest` method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41880
diff changeset
273
5ad5a70df2f7 verify: document the `_verifymanifest` method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41880
diff changeset
274 return a two level mapping:
5ad5a70df2f7 verify: document the `_verifymanifest` method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41880
diff changeset
275 {"path" -> { filenode -> changelog-revision}}
5ad5a70df2f7 verify: document the `_verifymanifest` method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41880
diff changeset
276
5ad5a70df2f7 verify: document the `_verifymanifest` method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41880
diff changeset
277 This mapping primarily contains entries for every files in the
5ad5a70df2f7 verify: document the `_verifymanifest` method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41880
diff changeset
278 repository. In addition, when tree-manifest is used, it also contains
5ad5a70df2f7 verify: document the `_verifymanifest` method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41880
diff changeset
279 sub-directory entries.
5ad5a70df2f7 verify: document the `_verifymanifest` method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41880
diff changeset
280
5ad5a70df2f7 verify: document the `_verifymanifest` method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41880
diff changeset
281 If a matcher is provided, only matching paths will be included.
5ad5a70df2f7 verify: document the `_verifymanifest` method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41880
diff changeset
282 """
27646
8f43793382c6 verify: move manifest verification to its own function
Durham Goode <durham@fb.com>
parents: 27645
diff changeset
283 repo = self.repo
8f43793382c6 verify: move manifest verification to its own function
Durham Goode <durham@fb.com>
parents: 27645
diff changeset
284 ui = self.ui
30900
5249b6470de9 verify: replace _validpath() by matcher
Martin von Zweigbergk <martinvonz@google.com>
parents: 30403
diff changeset
285 match = self.match
30309
f65faa4422c8 manifest: remove manifest.readshallowdelta
Durham Goode <durham@fb.com>
parents: 28467
diff changeset
286 mfl = self.repo.manifestlog
39272
73cf21b2e8a6 manifest: add getstorage() to manifestlog and use it globally
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38403
diff changeset
287 mf = mfl.getstorage(dir)
27646
8f43793382c6 verify: move manifest verification to its own function
Durham Goode <durham@fb.com>
parents: 27645
diff changeset
288
28203
7297e9e13a8a verify: check directory manifests
Martin von Zweigbergk <martinvonz@google.com>
parents: 28115
diff changeset
289 if not dir:
7297e9e13a8a verify: check directory manifests
Martin von Zweigbergk <martinvonz@google.com>
parents: 28115
diff changeset
290 self.ui.status(_("checking manifests\n"))
7297e9e13a8a verify: check directory manifests
Martin von Zweigbergk <martinvonz@google.com>
parents: 28115
diff changeset
291
27695
fb0cc863d172 verify: replace "output parameters" by return values
Martin von Zweigbergk <martinvonz@google.com>
parents: 27648
diff changeset
292 filenodes = {}
28203
7297e9e13a8a verify: check directory manifests
Martin von Zweigbergk <martinvonz@google.com>
parents: 28115
diff changeset
293 subdirnodes = {}
27443
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
294 seen = {}
28115
bd279da57e4b verify: extract "manifest" constant into variable
Martin von Zweigbergk <martinvonz@google.com>
parents: 28114
diff changeset
295 label = "manifest"
28203
7297e9e13a8a verify: check directory manifests
Martin von Zweigbergk <martinvonz@google.com>
parents: 28115
diff changeset
296 if dir:
7297e9e13a8a verify: check directory manifests
Martin von Zweigbergk <martinvonz@google.com>
parents: 28115
diff changeset
297 label = dir
28204
962921c330b0 verify: check for orphaned dirlogs
Martin von Zweigbergk <martinvonz@google.com>
parents: 28203
diff changeset
298 revlogfiles = mf.files()
962921c330b0 verify: check for orphaned dirlogs
Martin von Zweigbergk <martinvonz@google.com>
parents: 28203
diff changeset
299 storefiles.difference_update(revlogfiles)
38402
0ddbe03c5aaa verify: use progress helper for subdirectory progress
Martin von Zweigbergk <martinvonz@google.com>
parents: 37417
diff changeset
300 if subdirprogress: # should be true since we're in a subdirectory
0ddbe03c5aaa verify: use progress helper for subdirectory progress
Martin von Zweigbergk <martinvonz@google.com>
parents: 37417
diff changeset
301 subdirprogress.increment()
27444
6647401858ab verify: move widely used variables into class members
Durham Goode <durham@fb.com>
parents: 27443
diff changeset
302 if self.refersmf:
27443
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
303 # Do not check manifest if there are only changelog entries with
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
304 # null manifests.
41879
1f412223f5bb verify: rename the `checklog` to `_checkrevlog`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41878
diff changeset
305 self._checkrevlog(mf, label, 0)
38403
1249475f0bd6 verify: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 38402
diff changeset
306 progress = ui.makeprogress(_('checking'), unit=_('manifests'),
1249475f0bd6 verify: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 38402
diff changeset
307 total=len(mf))
27443
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
308 for i in mf:
28203
7297e9e13a8a verify: check directory manifests
Martin von Zweigbergk <martinvonz@google.com>
parents: 28115
diff changeset
309 if not dir:
38403
1249475f0bd6 verify: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 38402
diff changeset
310 progress.update(i)
27443
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
311 n = mf.node(i)
41876
9e737ca539f6 verify: make `checkentry` a private method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41875
diff changeset
312 lr = self._checkentry(mf, i, n, seen, mflinkrevs.get(n, []), label)
27443
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
313 if n in mflinkrevs:
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
314 del mflinkrevs[n]
28203
7297e9e13a8a verify: check directory manifests
Martin von Zweigbergk <martinvonz@google.com>
parents: 28115
diff changeset
315 elif dir:
41869
7eaf4b1ac2a3 verify: make `err` a private method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41868
diff changeset
316 self._err(lr, _("%s not in parent-directory manifest") %
28203
7297e9e13a8a verify: check directory manifests
Martin von Zweigbergk <martinvonz@google.com>
parents: 28115
diff changeset
317 short(n), label)
27443
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
318 else:
41869
7eaf4b1ac2a3 verify: make `err` a private method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41868
diff changeset
319 self._err(lr, _("%s not in changesets") % short(n), label)
2778
fdc232d8a193 Move repo.verify
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
320
27443
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
321 try:
30309
f65faa4422c8 manifest: remove manifest.readshallowdelta
Durham Goode <durham@fb.com>
parents: 28467
diff changeset
322 mfdelta = mfl.get(dir, n).readdelta(shallow=True)
f65faa4422c8 manifest: remove manifest.readshallowdelta
Durham Goode <durham@fb.com>
parents: 28467
diff changeset
323 for f, fn, fl in mfdelta.iterentries():
27443
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
324 if not f:
41869
7eaf4b1ac2a3 verify: make `err` a private method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41868
diff changeset
325 self._err(lr, _("entry without name in manifest"))
28203
7297e9e13a8a verify: check directory manifests
Martin von Zweigbergk <martinvonz@google.com>
parents: 28115
diff changeset
326 elif f == "/dev/null": # ignore this in very old repos
7297e9e13a8a verify: check directory manifests
Martin von Zweigbergk <martinvonz@google.com>
parents: 28115
diff changeset
327 continue
7297e9e13a8a verify: check directory manifests
Martin von Zweigbergk <martinvonz@google.com>
parents: 28115
diff changeset
328 fullpath = dir + _normpath(f)
7297e9e13a8a verify: check directory manifests
Martin von Zweigbergk <martinvonz@google.com>
parents: 28115
diff changeset
329 if fl == 't':
30900
5249b6470de9 verify: replace _validpath() by matcher
Martin von Zweigbergk <martinvonz@google.com>
parents: 30403
diff changeset
330 if not match.visitdir(fullpath):
5249b6470de9 verify: replace _validpath() by matcher
Martin von Zweigbergk <martinvonz@google.com>
parents: 30403
diff changeset
331 continue
28203
7297e9e13a8a verify: check directory manifests
Martin von Zweigbergk <martinvonz@google.com>
parents: 28115
diff changeset
332 subdirnodes.setdefault(fullpath + '/', {}).setdefault(
7297e9e13a8a verify: check directory manifests
Martin von Zweigbergk <martinvonz@google.com>
parents: 28115
diff changeset
333 fn, []).append(lr)
7297e9e13a8a verify: check directory manifests
Martin von Zweigbergk <martinvonz@google.com>
parents: 28115
diff changeset
334 else:
30900
5249b6470de9 verify: replace _validpath() by matcher
Martin von Zweigbergk <martinvonz@google.com>
parents: 30403
diff changeset
335 if not match(fullpath):
5249b6470de9 verify: replace _validpath() by matcher
Martin von Zweigbergk <martinvonz@google.com>
parents: 30403
diff changeset
336 continue
28203
7297e9e13a8a verify: check directory manifests
Martin von Zweigbergk <martinvonz@google.com>
parents: 28115
diff changeset
337 filenodes.setdefault(fullpath, {}).setdefault(fn, lr)
27443
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
338 except Exception as inst:
41871
cfe08588d711 verify: make the `exc` method private
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41870
diff changeset
339 self._exc(lr, _("reading delta %s") % short(n), inst, label)
28203
7297e9e13a8a verify: check directory manifests
Martin von Zweigbergk <martinvonz@google.com>
parents: 28115
diff changeset
340 if not dir:
38403
1249475f0bd6 verify: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 38402
diff changeset
341 progress.complete()
27443
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
342
28111
06205989264b verify: move cross-checking of changeset/manifest out of _crosscheckfiles()
Martin von Zweigbergk <martinvonz@google.com>
parents: 28007
diff changeset
343 if self.havemf:
41882
9c5a6af74afa verify: small refactoring and documentation in `_verifymanifest`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41881
diff changeset
344 # since we delete entry in `mflinkrevs` during iteration, any
9c5a6af74afa verify: small refactoring and documentation in `_verifymanifest`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41881
diff changeset
345 # remaining entries are "missing". We need to issue errors for them.
9c5a6af74afa verify: small refactoring and documentation in `_verifymanifest`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41881
diff changeset
346 changesetpairs = [(c, m) for m in mflinkrevs for c in mflinkrevs[m]]
9c5a6af74afa verify: small refactoring and documentation in `_verifymanifest`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41881
diff changeset
347 for c, m in sorted(changesetpairs):
28203
7297e9e13a8a verify: check directory manifests
Martin von Zweigbergk <martinvonz@google.com>
parents: 28115
diff changeset
348 if dir:
41869
7eaf4b1ac2a3 verify: make `err` a private method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41868
diff changeset
349 self._err(c, _("parent-directory manifest refers to unknown"
7eaf4b1ac2a3 verify: make `err` a private method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41868
diff changeset
350 " revision %s") % short(m), label)
28203
7297e9e13a8a verify: check directory manifests
Martin von Zweigbergk <martinvonz@google.com>
parents: 28115
diff changeset
351 else:
41869
7eaf4b1ac2a3 verify: make `err` a private method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41868
diff changeset
352 self._err(c, _("changeset refers to unknown revision %s") %
7eaf4b1ac2a3 verify: make `err` a private method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41868
diff changeset
353 short(m), label)
28203
7297e9e13a8a verify: check directory manifests
Martin von Zweigbergk <martinvonz@google.com>
parents: 28115
diff changeset
354
7297e9e13a8a verify: check directory manifests
Martin von Zweigbergk <martinvonz@google.com>
parents: 28115
diff changeset
355 if not dir and subdirnodes:
7297e9e13a8a verify: check directory manifests
Martin von Zweigbergk <martinvonz@google.com>
parents: 28115
diff changeset
356 self.ui.status(_("checking directory manifests\n"))
28204
962921c330b0 verify: check for orphaned dirlogs
Martin von Zweigbergk <martinvonz@google.com>
parents: 28203
diff changeset
357 storefiles = set()
28205
53f42c8d5f71 verify: show progress while verifying dirlogs
Martin von Zweigbergk <martinvonz@google.com>
parents: 28204
diff changeset
358 subdirs = set()
28204
962921c330b0 verify: check for orphaned dirlogs
Martin von Zweigbergk <martinvonz@google.com>
parents: 28203
diff changeset
359 revlogv1 = self.revlogv1
962921c330b0 verify: check for orphaned dirlogs
Martin von Zweigbergk <martinvonz@google.com>
parents: 28203
diff changeset
360 for f, f2, size in repo.store.datafiles():
962921c330b0 verify: check for orphaned dirlogs
Martin von Zweigbergk <martinvonz@google.com>
parents: 28203
diff changeset
361 if not f:
41869
7eaf4b1ac2a3 verify: make `err` a private method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41868
diff changeset
362 self._err(None, _("cannot decode filename '%s'") % f2)
28204
962921c330b0 verify: check for orphaned dirlogs
Martin von Zweigbergk <martinvonz@google.com>
parents: 28203
diff changeset
363 elif (size > 0 or not revlogv1) and f.startswith('meta/'):
962921c330b0 verify: check for orphaned dirlogs
Martin von Zweigbergk <martinvonz@google.com>
parents: 28203
diff changeset
364 storefiles.add(_normpath(f))
28205
53f42c8d5f71 verify: show progress while verifying dirlogs
Martin von Zweigbergk <martinvonz@google.com>
parents: 28204
diff changeset
365 subdirs.add(os.path.dirname(f))
38402
0ddbe03c5aaa verify: use progress helper for subdirectory progress
Martin von Zweigbergk <martinvonz@google.com>
parents: 37417
diff changeset
366 subdirprogress = ui.makeprogress(_('checking'), unit=_('manifests'),
0ddbe03c5aaa verify: use progress helper for subdirectory progress
Martin von Zweigbergk <martinvonz@google.com>
parents: 37417
diff changeset
367 total=len(subdirs))
28204
962921c330b0 verify: check for orphaned dirlogs
Martin von Zweigbergk <martinvonz@google.com>
parents: 28203
diff changeset
368
28203
7297e9e13a8a verify: check directory manifests
Martin von Zweigbergk <martinvonz@google.com>
parents: 28115
diff changeset
369 for subdir, linkrevs in subdirnodes.iteritems():
28205
53f42c8d5f71 verify: show progress while verifying dirlogs
Martin von Zweigbergk <martinvonz@google.com>
parents: 28204
diff changeset
370 subdirfilenodes = self._verifymanifest(linkrevs, subdir, storefiles,
38402
0ddbe03c5aaa verify: use progress helper for subdirectory progress
Martin von Zweigbergk <martinvonz@google.com>
parents: 37417
diff changeset
371 subdirprogress)
28203
7297e9e13a8a verify: check directory manifests
Martin von Zweigbergk <martinvonz@google.com>
parents: 28115
diff changeset
372 for f, onefilenodes in subdirfilenodes.iteritems():
7297e9e13a8a verify: check directory manifests
Martin von Zweigbergk <martinvonz@google.com>
parents: 28115
diff changeset
373 filenodes.setdefault(f, {}).update(onefilenodes)
28111
06205989264b verify: move cross-checking of changeset/manifest out of _crosscheckfiles()
Martin von Zweigbergk <martinvonz@google.com>
parents: 28007
diff changeset
374
28204
962921c330b0 verify: check for orphaned dirlogs
Martin von Zweigbergk <martinvonz@google.com>
parents: 28203
diff changeset
375 if not dir and subdirnodes:
38402
0ddbe03c5aaa verify: use progress helper for subdirectory progress
Martin von Zweigbergk <martinvonz@google.com>
parents: 37417
diff changeset
376 subdirprogress.complete()
37417
76d2115cb817 verify: allow suppressing warnings about extra files
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37410
diff changeset
377 if self.warnorphanstorefiles:
76d2115cb817 verify: allow suppressing warnings about extra files
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37410
diff changeset
378 for f in sorted(storefiles):
41867
c66037fb1bc5 verify: make the `warn` method private
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41866
diff changeset
379 self._warn(_("warning: orphan data file '%s'") % f)
28204
962921c330b0 verify: check for orphaned dirlogs
Martin von Zweigbergk <martinvonz@google.com>
parents: 28203
diff changeset
380
27695
fb0cc863d172 verify: replace "output parameters" by return values
Martin von Zweigbergk <martinvonz@google.com>
parents: 27648
diff changeset
381 return filenodes
27645
df8973e1fb45 verify: move file cross checking to its own function
Durham Goode <durham@fb.com>
parents: 27644
diff changeset
382
28111
06205989264b verify: move cross-checking of changeset/manifest out of _crosscheckfiles()
Martin von Zweigbergk <martinvonz@google.com>
parents: 28007
diff changeset
383 def _crosscheckfiles(self, filelinkrevs, filenodes):
27645
df8973e1fb45 verify: move file cross checking to its own function
Durham Goode <durham@fb.com>
parents: 27644
diff changeset
384 repo = self.repo
df8973e1fb45 verify: move file cross checking to its own function
Durham Goode <durham@fb.com>
parents: 27644
diff changeset
385 ui = self.ui
27443
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
386 ui.status(_("crosschecking files in changesets and manifests\n"))
2778
fdc232d8a193 Move repo.verify
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
387
28111
06205989264b verify: move cross-checking of changeset/manifest out of _crosscheckfiles()
Martin von Zweigbergk <martinvonz@google.com>
parents: 28007
diff changeset
388 total = len(filelinkrevs) + len(filenodes)
40437
d2ff0af6e959 verify: provide unit to ui.makeprogress()
Anton Shestakov <av6@dwimlabs.net>
parents: 39943
diff changeset
389 progress = ui.makeprogress(_('crosschecking'), unit=_('files'),
d2ff0af6e959 verify: provide unit to ui.makeprogress()
Anton Shestakov <av6@dwimlabs.net>
parents: 39943
diff changeset
390 total=total)
27645
df8973e1fb45 verify: move file cross checking to its own function
Durham Goode <durham@fb.com>
parents: 27644
diff changeset
391 if self.havemf:
27443
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
392 for f in sorted(filelinkrevs):
38403
1249475f0bd6 verify: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 38402
diff changeset
393 progress.increment()
27443
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
394 if f not in filenodes:
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
395 lr = filelinkrevs[f][0]
41869
7eaf4b1ac2a3 verify: make `err` a private method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41868
diff changeset
396 self._err(lr, _("in changeset but not in manifest"), f)
6892
dab95717058d verify: check repo.store
Adrian Buehlmann <adrian@cadifra.com>
parents: 6889
diff changeset
397
27645
df8973e1fb45 verify: move file cross checking to its own function
Durham Goode <durham@fb.com>
parents: 27644
diff changeset
398 if self.havecl:
27443
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
399 for f in sorted(filenodes):
38403
1249475f0bd6 verify: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 38402
diff changeset
400 progress.increment()
27443
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
401 if f not in filelinkrevs:
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
402 try:
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
403 fl = repo.file(f)
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
404 lr = min([fl.linkrev(fl.rev(n)) for n in filenodes[f]])
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
405 except Exception:
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
406 lr = None
41869
7eaf4b1ac2a3 verify: make `err` a private method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41868
diff changeset
407 self._err(lr, _("in manifest but not in changeset"), f)
27443
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
408
38403
1249475f0bd6 verify: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 38402
diff changeset
409 progress.complete()
8291
f5c1a9094e41 verify: avoid exception on missing file revlog
Henrik Stuart <hg@hstuart.dk>
parents: 8225
diff changeset
410
27644
331e5c28f5f0 verify: move filelog verification to its own function
Durham Goode <durham@fb.com>
parents: 27643
diff changeset
411 def _verifyfiles(self, filenodes, filelinkrevs):
331e5c28f5f0 verify: move filelog verification to its own function
Durham Goode <durham@fb.com>
parents: 27643
diff changeset
412 repo = self.repo
331e5c28f5f0 verify: move filelog verification to its own function
Durham Goode <durham@fb.com>
parents: 27643
diff changeset
413 ui = self.ui
331e5c28f5f0 verify: move filelog verification to its own function
Durham Goode <durham@fb.com>
parents: 27643
diff changeset
414 lrugetctx = self.lrugetctx
331e5c28f5f0 verify: move filelog verification to its own function
Durham Goode <durham@fb.com>
parents: 27643
diff changeset
415 revlogv1 = self.revlogv1
331e5c28f5f0 verify: move filelog verification to its own function
Durham Goode <durham@fb.com>
parents: 27643
diff changeset
416 havemf = self.havemf
27443
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
417 ui.status(_("checking files\n"))
8291
f5c1a9094e41 verify: avoid exception on missing file revlog
Henrik Stuart <hg@hstuart.dk>
parents: 8225
diff changeset
418
27443
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
419 storefiles = set()
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
420 for f, f2, size in repo.store.datafiles():
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
421 if not f:
41869
7eaf4b1ac2a3 verify: make `err` a private method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41868
diff changeset
422 self._err(None, _("cannot decode filename '%s'") % f2)
28007
fb92927f9775 treemanifests: fix streaming clone
Martin von Zweigbergk <martinvonz@google.com>
parents: 27964
diff changeset
423 elif (size > 0 or not revlogv1) and f.startswith('data/'):
27443
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
424 storefiles.add(_normpath(f))
6892
dab95717058d verify: check repo.store
Adrian Buehlmann <adrian@cadifra.com>
parents: 6889
diff changeset
425
39847
97986c9c69d3 verify: start to abstract file verification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39793
diff changeset
426 state = {
39850
e6d3d39cc1c7 revlog: use proper version comparison during verify
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39847
diff changeset
427 # TODO this assumes revlog storage for changelog.
39877
733db72f0f54 revlog: move revision verification out of verify
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39850
diff changeset
428 'expectedversion': self.repo.changelog.version & 0xFFFF,
733db72f0f54 revlog: move revision verification out of verify
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39850
diff changeset
429 'skipflags': self.skipflags,
733db72f0f54 revlog: move revision verification out of verify
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39850
diff changeset
430 # experimental config: censor.policy
733db72f0f54 revlog: move revision verification out of verify
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39850
diff changeset
431 'erroroncensored': ui.config('censor', 'policy') == 'abort',
39847
97986c9c69d3 verify: start to abstract file verification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39793
diff changeset
432 }
97986c9c69d3 verify: start to abstract file verification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39793
diff changeset
433
27443
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
434 files = sorted(set(filenodes) | set(filelinkrevs))
27644
331e5c28f5f0 verify: move filelog verification to its own function
Durham Goode <durham@fb.com>
parents: 27643
diff changeset
435 revisions = 0
38403
1249475f0bd6 verify: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 38402
diff changeset
436 progress = ui.makeprogress(_('checking'), unit=_('files'),
1249475f0bd6 verify: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 38402
diff changeset
437 total=len(files))
27443
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
438 for i, f in enumerate(files):
38403
1249475f0bd6 verify: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 38402
diff changeset
439 progress.update(i, item=f)
6892
dab95717058d verify: check repo.store
Adrian Buehlmann <adrian@cadifra.com>
parents: 6889
diff changeset
440 try:
27443
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
441 linkrevs = filelinkrevs[f]
6892
dab95717058d verify: check repo.store
Adrian Buehlmann <adrian@cadifra.com>
parents: 6889
diff changeset
442 except KeyError:
27443
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
443 # in manifest but not in changelog
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
444 linkrevs = []
2778
fdc232d8a193 Move repo.verify
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
445
27443
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
446 if linkrevs:
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
447 lr = linkrevs[0]
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
448 else:
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
449 lr = None
2778
fdc232d8a193 Move repo.verify
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
450
3744
d626fc9e3985 verify: add rename link checking
Matt Mackall <mpm@selenic.com>
parents: 3473
diff changeset
451 try:
27443
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
452 fl = repo.file(f)
39793
b63dee7bd0d9 global: replace most uses of RevlogError with StorageError (API)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39506
diff changeset
453 except error.StorageError as e:
41869
7eaf4b1ac2a3 verify: make `err` a private method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41868
diff changeset
454 self._err(lr, _("broken revlog! (%s)") % e, f)
27443
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
455 continue
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
456
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
457 for ff in fl.files():
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
458 try:
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
459 storefiles.remove(ff)
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
460 except KeyError:
37417
76d2115cb817 verify: allow suppressing warnings about extra files
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37410
diff changeset
461 if self.warnorphanstorefiles:
41867
c66037fb1bc5 verify: make the `warn` method private
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41866
diff changeset
462 self._warn(_(" warning: revlog '%s' not in fncache!") %
37417
76d2115cb817 verify: allow suppressing warnings about extra files
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37410
diff changeset
463 ff)
76d2115cb817 verify: allow suppressing warnings about extra files
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37410
diff changeset
464 self.fncachewarned = True
27443
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
465
39847
97986c9c69d3 verify: start to abstract file verification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39793
diff changeset
466 if not len(fl) and (self.havecl or self.havemf):
41869
7eaf4b1ac2a3 verify: make `err` a private method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41868
diff changeset
467 self._err(lr, _("empty or missing %s") % f)
39847
97986c9c69d3 verify: start to abstract file verification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39793
diff changeset
468 else:
39877
733db72f0f54 revlog: move revision verification out of verify
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39850
diff changeset
469 # Guard against implementations not setting this.
733db72f0f54 revlog: move revision verification out of verify
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39850
diff changeset
470 state['skipread'] = set()
39847
97986c9c69d3 verify: start to abstract file verification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39793
diff changeset
471 for problem in fl.verifyintegrity(state):
39877
733db72f0f54 revlog: move revision verification out of verify
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39850
diff changeset
472 if problem.node is not None:
733db72f0f54 revlog: move revision verification out of verify
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39850
diff changeset
473 linkrev = fl.linkrev(fl.rev(problem.node))
733db72f0f54 revlog: move revision verification out of verify
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39850
diff changeset
474 else:
733db72f0f54 revlog: move revision verification out of verify
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39850
diff changeset
475 linkrev = None
733db72f0f54 revlog: move revision verification out of verify
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39850
diff changeset
476
39847
97986c9c69d3 verify: start to abstract file verification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39793
diff changeset
477 if problem.warning:
41867
c66037fb1bc5 verify: make the `warn` method private
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41866
diff changeset
478 self._warn(problem.warning)
39847
97986c9c69d3 verify: start to abstract file verification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39793
diff changeset
479 elif problem.error:
41869
7eaf4b1ac2a3 verify: make `err` a private method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41868
diff changeset
480 self._err(linkrev if linkrev is not None else lr,
7eaf4b1ac2a3 verify: make `err` a private method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41868
diff changeset
481 problem.error, f)
39847
97986c9c69d3 verify: start to abstract file verification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39793
diff changeset
482 else:
97986c9c69d3 verify: start to abstract file verification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39793
diff changeset
483 raise error.ProgrammingError(
97986c9c69d3 verify: start to abstract file verification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39793
diff changeset
484 'problem instance does not set warning or error '
97986c9c69d3 verify: start to abstract file verification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39793
diff changeset
485 'attribute: %s' % problem.msg)
97986c9c69d3 verify: start to abstract file verification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39793
diff changeset
486
27443
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
487 seen = {}
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
488 for i in fl:
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
489 revisions += 1
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
490 n = fl.node(i)
41876
9e737ca539f6 verify: make `checkentry` a private method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41875
diff changeset
491 lr = self._checkentry(fl, i, n, seen, linkrevs, f)
27443
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
492 if f in filenodes:
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
493 if havemf and n not in filenodes[f]:
41869
7eaf4b1ac2a3 verify: make `err` a private method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41868
diff changeset
494 self._err(lr, _("%s not in manifests") % (short(n)), f)
6534
9b35a9f34675 verify: check copy source revlog and nodeid
Patrick Mezard <pmezard@gmail.com>
parents: 6211
diff changeset
495 else:
27443
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
496 del filenodes[f][n]
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
497
39877
733db72f0f54 revlog: move revision verification out of verify
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39850
diff changeset
498 if n in state['skipread']:
733db72f0f54 revlog: move revision verification out of verify
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39850
diff changeset
499 continue
3744
d626fc9e3985 verify: add rename link checking
Matt Mackall <mpm@selenic.com>
parents: 3473
diff changeset
500
27443
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
501 # check renames
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
502 try:
39877
733db72f0f54 revlog: move revision verification out of verify
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39850
diff changeset
503 # This requires resolving fulltext (at least on revlogs). We
733db72f0f54 revlog: move revision verification out of verify
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39850
diff changeset
504 # may want ``verifyintegrity()`` to pass a set of nodes with
733db72f0f54 revlog: move revision verification out of verify
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39850
diff changeset
505 # rename metadata as an optimization.
733db72f0f54 revlog: move revision verification out of verify
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39850
diff changeset
506 rp = fl.renamed(n)
27443
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
507 if rp:
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
508 if lr is not None and ui.verbose:
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
509 ctx = lrugetctx(lr)
36372
a4d41ba4ad23 verify: don't reimplement any()
Martin von Zweigbergk <martinvonz@google.com>
parents: 36223
diff changeset
510 if not any(rp[0] in pctx for pctx in ctx.parents()):
41867
c66037fb1bc5 verify: make the `warn` method private
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41866
diff changeset
511 self._warn(_("warning: copy source of '%s' not"
27446
6b2c1a1871a6 verify: move warn() to a class level function
Durham Goode <durham@fb.com>
parents: 27445
diff changeset
512 " in parents of %s") % (f, ctx))
27443
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
513 fl2 = repo.file(rp[0])
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
514 if not len(fl2):
41869
7eaf4b1ac2a3 verify: make `err` a private method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41868
diff changeset
515 self._err(lr,
7eaf4b1ac2a3 verify: make `err` a private method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41868
diff changeset
516 _("empty or missing copy source revlog "
7eaf4b1ac2a3 verify: make `err` a private method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41868
diff changeset
517 "%s:%s") % (rp[0],
7eaf4b1ac2a3 verify: make `err` a private method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41868
diff changeset
518 short(rp[1])),
7eaf4b1ac2a3 verify: make `err` a private method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41868
diff changeset
519 f)
27443
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
520 elif rp[1] == nullid:
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
521 ui.note(_("warning: %s@%s: copy source"
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
522 " revision is nullid %s:%s\n")
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
523 % (f, lr, rp[0], short(rp[1])))
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
524 else:
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
525 fl2.rev(rp[1])
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
526 except Exception as inst:
41871
cfe08588d711 verify: make the `exc` method private
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41870
diff changeset
527 self._exc(lr, _("checking rename of %s") % short(n),
cfe08588d711 verify: make the `exc` method private
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41870
diff changeset
528 inst, f)
6892
dab95717058d verify: check repo.store
Adrian Buehlmann <adrian@cadifra.com>
parents: 6889
diff changeset
529
27443
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
530 # cross-check
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
531 if f in filenodes:
30403
b667b78099eb verify: avoid shadowing two variables with a list comprehension
Augie Fackler <augie@google.com>
parents: 30385
diff changeset
532 fns = [(v, k) for k, v in filenodes[f].iteritems()]
27443
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
533 for lr, node in sorted(fns):
41869
7eaf4b1ac2a3 verify: make `err` a private method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41868
diff changeset
534 self._err(lr, _("manifest refers to unknown revision %s") %
7eaf4b1ac2a3 verify: make `err` a private method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41868
diff changeset
535 short(node), f)
38403
1249475f0bd6 verify: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 38402
diff changeset
536 progress.complete()
27443
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
537
37417
76d2115cb817 verify: allow suppressing warnings about extra files
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37410
diff changeset
538 if self.warnorphanstorefiles:
76d2115cb817 verify: allow suppressing warnings about extra files
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37410
diff changeset
539 for f in sorted(storefiles):
41867
c66037fb1bc5 verify: make the `warn` method private
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41866
diff changeset
540 self._warn(_("warning: orphan data file '%s'") % f)
27443
937e73a6e4ff verify: move verify logic into a class
Durham Goode <durham@fb.com>
parents: 26900
diff changeset
541
27644
331e5c28f5f0 verify: move filelog verification to its own function
Durham Goode <durham@fb.com>
parents: 27643
diff changeset
542 return len(files), revisions