mercurial-scm/hg-stable: mercurial/verify.py annotate

annotate mercurial/verify.py @ 42197:57539e5ea2e0

verify: introduce a notion of "level" Some checks are slower than others, to help the user to run the checks he needs, we are about to introduce new flag to select faster vs deeper runs. This put the scaffolding in place to do this.

author	Pierre-Yves David <pierre-yves.david@octobus.net>
date	Wed, 17 Apr 2019 01:12:21 +0200
parents	9c5a6af74afa
children	496ac8a02380

rev	line source
2778 fdc232d8a193 Move repo.verify Matt Mackall <mpm@selenic.com> parents: diff changeset	1 # verify.py - repository integrity checking for Mercurial
fdc232d8a193 Move repo.verify Matt Mackall <mpm@selenic.com> parents: diff changeset	2 #
4635 63b9d2deed48 Updated copyright notices and add "and others" to "hg version" Thomas Arendsen Hein <thomas@intevation.de> parents: 4395 diff changeset	3 # Copyright 2006, 2007 Matt Mackall <mpm@selenic.com>
2778 fdc232d8a193 Move repo.verify Matt Mackall <mpm@selenic.com> parents: diff changeset	4 #
8225 46293a0c7e9f updated license to be explicit about GPL version 2 Martin Geisler <mg@lazybytes.net> parents: 8209 diff changeset	5 # This software may be used and distributed according to the terms of the
10263 25e572394f5c Update license to GPLv2+ Matt Mackall <mpm@selenic.com> parents: 9690 diff changeset	6 # GNU General Public License version 2 or any later version.
2778 fdc232d8a193 Move repo.verify Matt Mackall <mpm@selenic.com> parents: diff changeset	7
25991 d21d1774c73b verify: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 25846 diff changeset	8 from __future__ import absolute_import
d21d1774c73b verify: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 25846 diff changeset	9
17860 a45b33f12627 verify: fix all doubled-slash sites (issue3665) Bryan O'Sullivan <bryano@fb.com> parents: 17851 diff changeset	10 import os
25991 d21d1774c73b verify: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 25846 diff changeset	11
d21d1774c73b verify: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 25846 diff changeset	12 from .i18n import _
d21d1774c73b verify: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 25846 diff changeset	13 from .node import (
d21d1774c73b verify: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 25846 diff changeset	14 nullid,
d21d1774c73b verify: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 25846 diff changeset	15 short,
d21d1774c73b verify: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 25846 diff changeset	16 )
d21d1774c73b verify: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 25846 diff changeset	17
d21d1774c73b verify: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 25846 diff changeset	18 from . import (
d21d1774c73b verify: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 25846 diff changeset	19 error,
35585 35fb3367f72d py3: use pycompat.bytestr() instead of str() Pulkit Goyal <7895pulkit@gmail.com> parents: 33499 diff changeset	20 pycompat,
25991 d21d1774c73b verify: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 25846 diff changeset	21 revlog,
d21d1774c73b verify: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 25846 diff changeset	22 util,
d21d1774c73b verify: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 25846 diff changeset	23 )
2778 fdc232d8a193 Move repo.verify Matt Mackall <mpm@selenic.com> parents: diff changeset	24
42197 57539e5ea2e0 verify: introduce a notion of "level" Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41882 diff changeset	25 VERIFY_DEFAULT = 0
57539e5ea2e0 verify: introduce a notion of "level" Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41882 diff changeset	26
57539e5ea2e0 verify: introduce a notion of "level" Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41882 diff changeset	27 def verify(repo, level=None):
27849 900d36a3e4dd with: use context manager in verify Bryan O'Sullivan <bryano@fb.com> parents: 27695 diff changeset	28 with repo.lock():
42197 57539e5ea2e0 verify: introduce a notion of "level" Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41882 diff changeset	29 v = verifier(repo, level)
57539e5ea2e0 verify: introduce a notion of "level" Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41882 diff changeset	30 return v.verify()
4915 97b734fb9c6f Use try/finally pattern to cleanup locks and transactions Matt Mackall <mpm@selenic.com> parents: 4635 diff changeset	31
17860 a45b33f12627 verify: fix all doubled-slash sites (issue3665) Bryan O'Sullivan <bryano@fb.com> parents: 17851 diff changeset	32 def _normpath(f):
a45b33f12627 verify: fix all doubled-slash sites (issue3665) Bryan O'Sullivan <bryano@fb.com> parents: 17851 diff changeset	33 # under hg < 2.4, convert didn't sanitize paths properly, so a
a45b33f12627 verify: fix all doubled-slash sites (issue3665) Bryan O'Sullivan <bryano@fb.com> parents: 17851 diff changeset	34 # converted repo may contain repeated slashes
a45b33f12627 verify: fix all doubled-slash sites (issue3665) Bryan O'Sullivan <bryano@fb.com> parents: 17851 diff changeset	35 while '//' in f:
a45b33f12627 verify: fix all doubled-slash sites (issue3665) Bryan O'Sullivan <bryano@fb.com> parents: 17851 diff changeset	36 f = f.replace('//', '/')
a45b33f12627 verify: fix all doubled-slash sites (issue3665) Bryan O'Sullivan <bryano@fb.com> parents: 17851 diff changeset	37 return f
a45b33f12627 verify: fix all doubled-slash sites (issue3665) Bryan O'Sullivan <bryano@fb.com> parents: 17851 diff changeset	38
27443 937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	39 class verifier(object):
42197 57539e5ea2e0 verify: introduce a notion of "level" Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41882 diff changeset	40 def __init__(self, repo, level=None):
27444 6647401858ab verify: move widely used variables into class members Durham Goode <durham@fb.com> parents: 27443 diff changeset	41 self.repo = repo.unfiltered()
6647401858ab verify: move widely used variables into class members Durham Goode <durham@fb.com> parents: 27443 diff changeset	42 self.ui = repo.ui
39943 fec944719324 narrow: move support for `hg verify` into core Martin von Zweigbergk <martinvonz@google.com> parents: 39877 diff changeset	43 self.match = repo.narrowmatch()
42197 57539e5ea2e0 verify: introduce a notion of "level" Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41882 diff changeset	44 if level is None:
57539e5ea2e0 verify: introduce a notion of "level" Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41882 diff changeset	45 level = VERIFY_DEFAULT
57539e5ea2e0 verify: introduce a notion of "level" Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41882 diff changeset	46 self._level = level
27444 6647401858ab verify: move widely used variables into class members Durham Goode <durham@fb.com> parents: 27443 diff changeset	47 self.badrevs = set()
27453 8462d7f2c4fe verify: clean up weird error/warning lists Matt Mackall <mpm@selenic.com> parents: 27450 diff changeset	48 self.errors = 0
8462d7f2c4fe verify: clean up weird error/warning lists Matt Mackall <mpm@selenic.com> parents: 27450 diff changeset	49 self.warnings = 0
27444 6647401858ab verify: move widely used variables into class members Durham Goode <durham@fb.com> parents: 27443 diff changeset	50 self.havecl = len(repo.changelog) > 0
39272 73cf21b2e8a6 manifest: add getstorage() to manifestlog and use it globally Gregory Szorc <gregory.szorc@gmail.com> parents: 38403 diff changeset	51 self.havemf = len(repo.manifestlog.getstorage(b'')) > 0
27444 6647401858ab verify: move widely used variables into class members Durham Goode <durham@fb.com> parents: 27443 diff changeset	52 self.revlogv1 = repo.changelog.version != revlog.REVLOGV0
37302 00f18dd1d3d6 verify: remove dependence on repo.changectx() Martin von Zweigbergk <martinvonz@google.com> parents: 36606 diff changeset	53 self.lrugetctx = util.lrucachefunc(repo.__getitem__)
27444 6647401858ab verify: move widely used variables into class members Durham Goode <durham@fb.com> parents: 27443 diff changeset	54 self.refersmf = False
27445 cc178057ab49 verify: move fncachewarned up to a class variable Durham Goode <durham@fb.com> parents: 27444 diff changeset	55 self.fncachewarned = False
32328 a2ab9ebcd85b verify: add a config option to skip certain flag processors Jun Wu <quark@fb.com> parents: 32290 diff changeset	56 # developer config: verify.skipflags
a2ab9ebcd85b verify: add a config option to skip certain flag processors Jun Wu <quark@fb.com> parents: 32290 diff changeset	57 self.skipflags = repo.ui.configint('verify', 'skipflags')
37417 76d2115cb817 verify: allow suppressing warnings about extra files Gregory Szorc <gregory.szorc@gmail.com> parents: 37410 diff changeset	58 self.warnorphanstorefiles = True
27444 6647401858ab verify: move widely used variables into class members Durham Goode <durham@fb.com> parents: 27443 diff changeset	59
41867 c66037fb1bc5 verify: make the `warn` method private Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41866 diff changeset	60 def _warn(self, msg):
41866 a58748300e61 verify: document the `warn` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 40437 diff changeset	61 """record a "warning" level issue"""
27446 6b2c1a1871a6 verify: move warn() to a class level function Durham Goode <durham@fb.com> parents: 27445 diff changeset	62 self.ui.warn(msg + "\n")
27453 8462d7f2c4fe verify: clean up weird error/warning lists Matt Mackall <mpm@selenic.com> parents: 27450 diff changeset	63 self.warnings += 1
27446 6b2c1a1871a6 verify: move warn() to a class level function Durham Goode <durham@fb.com> parents: 27445 diff changeset	64
41869 7eaf4b1ac2a3 verify: make `err` a private method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41868 diff changeset	65 def _err(self, linkrev, msg, filename=None):
41868 9534f3cb6b6c verify: document the `err` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41867 diff changeset	66 """record a "error" level issue"""
27447 d1b91c10ce70 verify: move err() to be a class function Durham Goode <durham@fb.com> parents: 27446 diff changeset	67 if linkrev is not None:
d1b91c10ce70 verify: move err() to be a class function Durham Goode <durham@fb.com> parents: 27446 diff changeset	68 self.badrevs.add(linkrev)
36223 acc8e6e52af6 py3: use "%d" to convert integers to bytes Pulkit Goyal <7895pulkit@gmail.com> parents: 35585 diff changeset	69 linkrev = "%d" % linkrev
27447 d1b91c10ce70 verify: move err() to be a class function Durham Goode <durham@fb.com> parents: 27446 diff changeset	70 else:
d1b91c10ce70 verify: move err() to be a class function Durham Goode <durham@fb.com> parents: 27446 diff changeset	71 linkrev = '?'
d1b91c10ce70 verify: move err() to be a class function Durham Goode <durham@fb.com> parents: 27446 diff changeset	72 msg = "%s: %s" % (linkrev, msg)
d1b91c10ce70 verify: move err() to be a class function Durham Goode <durham@fb.com> parents: 27446 diff changeset	73 if filename:
d1b91c10ce70 verify: move err() to be a class function Durham Goode <durham@fb.com> parents: 27446 diff changeset	74 msg = "%s@%s" % (filename, msg)
d1b91c10ce70 verify: move err() to be a class function Durham Goode <durham@fb.com> parents: 27446 diff changeset	75 self.ui.warn(" " + msg + "\n")
27453 8462d7f2c4fe verify: clean up weird error/warning lists Matt Mackall <mpm@selenic.com> parents: 27450 diff changeset	76 self.errors += 1
27447 d1b91c10ce70 verify: move err() to be a class function Durham Goode <durham@fb.com> parents: 27446 diff changeset	77
41871 cfe08588d711 verify: make the `exc` method private Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41870 diff changeset	78 def _exc(self, linkrev, msg, inst, filename=None):
41870 5df8475c5343 verify: document the `exc` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41869 diff changeset	79 """record exception raised during the verify process"""
36606 d85ef895d5f6 verify: fix exception formatting bug in Python 3 Augie Fackler <augie@google.com> parents: 36372 diff changeset	80 fmsg = pycompat.bytestr(inst)
d85ef895d5f6 verify: fix exception formatting bug in Python 3 Augie Fackler <augie@google.com> parents: 36372 diff changeset	81 if not fmsg:
d85ef895d5f6 verify: fix exception formatting bug in Python 3 Augie Fackler <augie@google.com> parents: 36372 diff changeset	82 fmsg = pycompat.byterepr(inst)
41869 7eaf4b1ac2a3 verify: make `err` a private method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41868 diff changeset	83 self._err(linkrev, "%s: %s" % (msg, fmsg), filename)
27448 f4f2179077cb verify: move exc() function onto class Durham Goode <durham@fb.com> parents: 27447 diff changeset	84
41879 1f412223f5bb verify: rename the `checklog` to `_checkrevlog` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41878 diff changeset	85 def _checkrevlog(self, obj, name, linkrev):
41878 08d977451f26 verify: document the `checklog` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41876 diff changeset	86 """verify high level property of a revlog
08d977451f26 verify: document the `checklog` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41876 diff changeset	87
08d977451f26 verify: document the `checklog` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41876 diff changeset	88 - revlog is present,
08d977451f26 verify: document the `checklog` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41876 diff changeset	89 - revlog is non-empty,
08d977451f26 verify: document the `checklog` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41876 diff changeset	90 - sizes (index and data) are correct,
08d977451f26 verify: document the `checklog` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41876 diff changeset	91 - revlog's format version is correct.
08d977451f26 verify: document the `checklog` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41876 diff changeset	92 """
27642 f6457349985b verify: move checklog() onto class Durham Goode <durham@fb.com> parents: 27453 diff changeset	93 if not len(obj) and (self.havecl or self.havemf):
41869 7eaf4b1ac2a3 verify: make `err` a private method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41868 diff changeset	94 self._err(linkrev, _("empty or missing %s") % name)
27642 f6457349985b verify: move checklog() onto class Durham Goode <durham@fb.com> parents: 27453 diff changeset	95 return
f6457349985b verify: move checklog() onto class Durham Goode <durham@fb.com> parents: 27453 diff changeset	96
f6457349985b verify: move checklog() onto class Durham Goode <durham@fb.com> parents: 27453 diff changeset	97 d = obj.checksize()
f6457349985b verify: move checklog() onto class Durham Goode <durham@fb.com> parents: 27453 diff changeset	98 if d[0]:
f6457349985b verify: move checklog() onto class Durham Goode <durham@fb.com> parents: 27453 diff changeset	99 self.err(None, _("data length off by %d bytes") % d[0], name)
f6457349985b verify: move checklog() onto class Durham Goode <durham@fb.com> parents: 27453 diff changeset	100 if d[1]:
f6457349985b verify: move checklog() onto class Durham Goode <durham@fb.com> parents: 27453 diff changeset	101 self.err(None, _("index contains %d extra bytes") % d[1], name)
f6457349985b verify: move checklog() onto class Durham Goode <durham@fb.com> parents: 27453 diff changeset	102
f6457349985b verify: move checklog() onto class Durham Goode <durham@fb.com> parents: 27453 diff changeset	103 if obj.version != revlog.REVLOGV0:
f6457349985b verify: move checklog() onto class Durham Goode <durham@fb.com> parents: 27453 diff changeset	104 if not self.revlogv1:
41867 c66037fb1bc5 verify: make the `warn` method private Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41866 diff changeset	105 self._warn(_("warning: `%s' uses revlog format 1") % name)
27642 f6457349985b verify: move checklog() onto class Durham Goode <durham@fb.com> parents: 27453 diff changeset	106 elif self.revlogv1:
41867 c66037fb1bc5 verify: make the `warn` method private Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41866 diff changeset	107 self._warn(_("warning: `%s' uses revlog format 0") % name)
27642 f6457349985b verify: move checklog() onto class Durham Goode <durham@fb.com> parents: 27453 diff changeset	108
41876 9e737ca539f6 verify: make `checkentry` a private method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41875 diff changeset	109 def _checkentry(self, obj, i, node, seen, linkrevs, f):
41875 00c9fde75c1a verify: document the `checkentry` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41874 diff changeset	110 """verify a single revlog entry
00c9fde75c1a verify: document the `checkentry` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41874 diff changeset	111
00c9fde75c1a verify: document the `checkentry` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41874 diff changeset	112 arguments are:
00c9fde75c1a verify: document the `checkentry` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41874 diff changeset	113 - obj: the source revlog
00c9fde75c1a verify: document the `checkentry` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41874 diff changeset	114 - i: the revision number
00c9fde75c1a verify: document the `checkentry` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41874 diff changeset	115 - node: the revision node id
00c9fde75c1a verify: document the `checkentry` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41874 diff changeset	116 - seen: nodes previously seen for this revlog
00c9fde75c1a verify: document the `checkentry` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41874 diff changeset	117 - linkrevs: [changelog-revisions] introducing "node"
00c9fde75c1a verify: document the `checkentry` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41874 diff changeset	118 - f: string label ("changelog", "manifest", or filename)
00c9fde75c1a verify: document the `checkentry` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41874 diff changeset	119
00c9fde75c1a verify: document the `checkentry` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41874 diff changeset	120 Performs the following checks:
00c9fde75c1a verify: document the `checkentry` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41874 diff changeset	121 - linkrev points to an existing changelog revision,
00c9fde75c1a verify: document the `checkentry` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41874 diff changeset	122 - linkrev points to a changelog revision that introduces this revision,
00c9fde75c1a verify: document the `checkentry` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41874 diff changeset	123 - linkrev points to the lowest of these changesets,
00c9fde75c1a verify: document the `checkentry` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41874 diff changeset	124 - both parents exist in the revlog,
00c9fde75c1a verify: document the `checkentry` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41874 diff changeset	125 - the revision is not duplicated.
00c9fde75c1a verify: document the `checkentry` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41874 diff changeset	126
00c9fde75c1a verify: document the `checkentry` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41874 diff changeset	127 Return the linkrev of the revision (or None for changelog's revisions).
00c9fde75c1a verify: document the `checkentry` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41874 diff changeset	128 """
27643 62ce86fcfd06 verify: move checkentry() to be a class function Durham Goode <durham@fb.com> parents: 27642 diff changeset	129 lr = obj.linkrev(obj.rev(node))
62ce86fcfd06 verify: move checkentry() to be a class function Durham Goode <durham@fb.com> parents: 27642 diff changeset	130 if lr < 0 or (self.havecl and lr not in linkrevs):
62ce86fcfd06 verify: move checkentry() to be a class function Durham Goode <durham@fb.com> parents: 27642 diff changeset	131 if lr < 0 or lr >= len(self.repo.changelog):
62ce86fcfd06 verify: move checkentry() to be a class function Durham Goode <durham@fb.com> parents: 27642 diff changeset	132 msg = _("rev %d points to nonexistent changeset %d")
62ce86fcfd06 verify: move checkentry() to be a class function Durham Goode <durham@fb.com> parents: 27642 diff changeset	133 else:
62ce86fcfd06 verify: move checkentry() to be a class function Durham Goode <durham@fb.com> parents: 27642 diff changeset	134 msg = _("rev %d points to unexpected changeset %d")
41869 7eaf4b1ac2a3 verify: make `err` a private method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41868 diff changeset	135 self._err(None, msg % (i, lr), f)
27643 62ce86fcfd06 verify: move checkentry() to be a class function Durham Goode <durham@fb.com> parents: 27642 diff changeset	136 if linkrevs:
62ce86fcfd06 verify: move checkentry() to be a class function Durham Goode <durham@fb.com> parents: 27642 diff changeset	137 if f and len(linkrevs) > 1:
62ce86fcfd06 verify: move checkentry() to be a class function Durham Goode <durham@fb.com> parents: 27642 diff changeset	138 try:
62ce86fcfd06 verify: move checkentry() to be a class function Durham Goode <durham@fb.com> parents: 27642 diff changeset	139 # attempt to filter down to real linkrevs
62ce86fcfd06 verify: move checkentry() to be a class function Durham Goode <durham@fb.com> parents: 27642 diff changeset	140 linkrevs = [l for l in linkrevs
62ce86fcfd06 verify: move checkentry() to be a class function Durham Goode <durham@fb.com> parents: 27642 diff changeset	141 if self.lrugetctx(l)[f].filenode() == node]
62ce86fcfd06 verify: move checkentry() to be a class function Durham Goode <durham@fb.com> parents: 27642 diff changeset	142 except Exception:
62ce86fcfd06 verify: move checkentry() to be a class function Durham Goode <durham@fb.com> parents: 27642 diff changeset	143 pass
41867 c66037fb1bc5 verify: make the `warn` method private Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41866 diff changeset	144 self._warn(_(" (expected %s)") % " ".join
c66037fb1bc5 verify: make the `warn` method private Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41866 diff changeset	145 (map(pycompat.bytestr, linkrevs)))
27643 62ce86fcfd06 verify: move checkentry() to be a class function Durham Goode <durham@fb.com> parents: 27642 diff changeset	146 lr = None # can't be trusted
62ce86fcfd06 verify: move checkentry() to be a class function Durham Goode <durham@fb.com> parents: 27642 diff changeset	147
62ce86fcfd06 verify: move checkentry() to be a class function Durham Goode <durham@fb.com> parents: 27642 diff changeset	148 try:
62ce86fcfd06 verify: move checkentry() to be a class function Durham Goode <durham@fb.com> parents: 27642 diff changeset	149 p1, p2 = obj.parents(node)
62ce86fcfd06 verify: move checkentry() to be a class function Durham Goode <durham@fb.com> parents: 27642 diff changeset	150 if p1 not in seen and p1 != nullid:
41869 7eaf4b1ac2a3 verify: make `err` a private method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41868 diff changeset	151 self._err(lr, _("unknown parent 1 %s of %s") %
27643 62ce86fcfd06 verify: move checkentry() to be a class function Durham Goode <durham@fb.com> parents: 27642 diff changeset	152 (short(p1), short(node)), f)
62ce86fcfd06 verify: move checkentry() to be a class function Durham Goode <durham@fb.com> parents: 27642 diff changeset	153 if p2 not in seen and p2 != nullid:
41869 7eaf4b1ac2a3 verify: make `err` a private method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41868 diff changeset	154 self._err(lr, _("unknown parent 2 %s of %s") %
27643 62ce86fcfd06 verify: move checkentry() to be a class function Durham Goode <durham@fb.com> parents: 27642 diff changeset	155 (short(p2), short(node)), f)
62ce86fcfd06 verify: move checkentry() to be a class function Durham Goode <durham@fb.com> parents: 27642 diff changeset	156 except Exception as inst:
41871 cfe08588d711 verify: make the `exc` method private Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41870 diff changeset	157 self._exc(lr, _("checking parents of %s") % short(node), inst, f)
27643 62ce86fcfd06 verify: move checkentry() to be a class function Durham Goode <durham@fb.com> parents: 27642 diff changeset	158
62ce86fcfd06 verify: move checkentry() to be a class function Durham Goode <durham@fb.com> parents: 27642 diff changeset	159 if node in seen:
41869 7eaf4b1ac2a3 verify: make `err` a private method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41868 diff changeset	160 self._err(lr, _("duplicate revision %d (%d)") % (i, seen[node]), f)
27643 62ce86fcfd06 verify: move checkentry() to be a class function Durham Goode <durham@fb.com> parents: 27642 diff changeset	161 seen[node] = i
62ce86fcfd06 verify: move checkentry() to be a class function Durham Goode <durham@fb.com> parents: 27642 diff changeset	162 return lr
62ce86fcfd06 verify: move checkentry() to be a class function Durham Goode <durham@fb.com> parents: 27642 diff changeset	163
27444 6647401858ab verify: move widely used variables into class members Durham Goode <durham@fb.com> parents: 27443 diff changeset	164 def verify(self):
41872 e8c4a9f5b986 verify: minimal documentation for `verifier.verify` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41871 diff changeset	165 """verify the content of the Mercurial repository
e8c4a9f5b986 verify: minimal documentation for `verifier.verify` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41871 diff changeset	166
e8c4a9f5b986 verify: minimal documentation for `verifier.verify` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41871 diff changeset	167 This method run all verifications, displaying issues as they are found.
e8c4a9f5b986 verify: minimal documentation for `verifier.verify` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41871 diff changeset	168
41873 567892b4306c verify: explicitly return 0 if no error are encountered Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41872 diff changeset	169 return 1 if any error have been encountered, 0 otherwise."""
41874 4da2261e949b verify: add some inline documentation to the top level `verify` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41873 diff changeset	170 # initial validation and generic report
27444 6647401858ab verify: move widely used variables into class members Durham Goode <durham@fb.com> parents: 27443 diff changeset	171 repo = self.repo
27443 937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	172 ui = repo.ui
937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	173 if not repo.url().startswith('file:'):
937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	174 raise error.Abort(_("cannot verify bundle or remote repos"))
6752 e79a8f36c2a5 verify: lots of refactoring Matt Mackall <mpm@selenic.com> parents: 6751 diff changeset	175
27443 937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	176 if os.path.exists(repo.sjoin("journal")):
937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	177 ui.warn(_("abandoned transaction found - run hg recover\n"))
937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	178
27648 e72e669dd51f verify: get rid of some unnecessary local variables Durham Goode <durham@fb.com> parents: 27647 diff changeset	179 if ui.verbose or not self.revlogv1:
27443 937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	180 ui.status(_("repository uses revlog format %d\n") %
27648 e72e669dd51f verify: get rid of some unnecessary local variables Durham Goode <durham@fb.com> parents: 27647 diff changeset	181 (self.revlogv1 and 1 or 0))
27443 937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	182
41874 4da2261e949b verify: add some inline documentation to the top level `verify` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41873 diff changeset	183 # data verification
27695 fb0cc863d172 verify: replace "output parameters" by return values Martin von Zweigbergk <martinvonz@google.com> parents: 27648 diff changeset	184 mflinkrevs, filelinkrevs = self._verifychangelog()
fb0cc863d172 verify: replace "output parameters" by return values Martin von Zweigbergk <martinvonz@google.com> parents: 27648 diff changeset	185 filenodes = self._verifymanifest(mflinkrevs)
28111 06205989264b verify: move cross-checking of changeset/manifest out of _crosscheckfiles() Martin von Zweigbergk <martinvonz@google.com> parents: 28007 diff changeset	186 del mflinkrevs
06205989264b verify: move cross-checking of changeset/manifest out of _crosscheckfiles() Martin von Zweigbergk <martinvonz@google.com> parents: 28007 diff changeset	187 self._crosscheckfiles(filelinkrevs, filenodes)
27647 2c2858f3c1bb verify: move changelog verificaiton to its own function Durham Goode <durham@fb.com> parents: 27646 diff changeset	188 totalfiles, filerevisions = self._verifyfiles(filenodes, filelinkrevs)
2c2858f3c1bb verify: move changelog verificaiton to its own function Durham Goode <durham@fb.com> parents: 27646 diff changeset	189
41874 4da2261e949b verify: add some inline documentation to the top level `verify` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41873 diff changeset	190 # final report
39506 f1186c292d03 verify: make output less confusing (issue5924) Meirambek Omyrzak <meirambek77@gmail.com> parents: 39272 diff changeset	191 ui.status(_("checked %d changesets with %d changes to %d files\n") %
f1186c292d03 verify: make output less confusing (issue5924) Meirambek Omyrzak <meirambek77@gmail.com> parents: 39272 diff changeset	192 (len(repo.changelog), filerevisions, totalfiles))
27647 2c2858f3c1bb verify: move changelog verificaiton to its own function Durham Goode <durham@fb.com> parents: 27646 diff changeset	193 if self.warnings:
2c2858f3c1bb verify: move changelog verificaiton to its own function Durham Goode <durham@fb.com> parents: 27646 diff changeset	194 ui.warn(_("%d warnings encountered!\n") % self.warnings)
2c2858f3c1bb verify: move changelog verificaiton to its own function Durham Goode <durham@fb.com> parents: 27646 diff changeset	195 if self.fncachewarned:
2c2858f3c1bb verify: move changelog verificaiton to its own function Durham Goode <durham@fb.com> parents: 27646 diff changeset	196 ui.warn(_('hint: run "hg debugrebuildfncache" to recover from '
2c2858f3c1bb verify: move changelog verificaiton to its own function Durham Goode <durham@fb.com> parents: 27646 diff changeset	197 'corrupt fncache\n'))
2c2858f3c1bb verify: move changelog verificaiton to its own function Durham Goode <durham@fb.com> parents: 27646 diff changeset	198 if self.errors:
2c2858f3c1bb verify: move changelog verificaiton to its own function Durham Goode <durham@fb.com> parents: 27646 diff changeset	199 ui.warn(_("%d integrity errors encountered!\n") % self.errors)
27648 e72e669dd51f verify: get rid of some unnecessary local variables Durham Goode <durham@fb.com> parents: 27647 diff changeset	200 if self.badrevs:
27647 2c2858f3c1bb verify: move changelog verificaiton to its own function Durham Goode <durham@fb.com> parents: 27646 diff changeset	201 ui.warn(_("(first damaged changeset appears to be %d)\n")
27648 e72e669dd51f verify: get rid of some unnecessary local variables Durham Goode <durham@fb.com> parents: 27647 diff changeset	202 % min(self.badrevs))
27647 2c2858f3c1bb verify: move changelog verificaiton to its own function Durham Goode <durham@fb.com> parents: 27646 diff changeset	203 return 1
41873 567892b4306c verify: explicitly return 0 if no error are encountered Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41872 diff changeset	204 return 0
27647 2c2858f3c1bb verify: move changelog verificaiton to its own function Durham Goode <durham@fb.com> parents: 27646 diff changeset	205
27695 fb0cc863d172 verify: replace "output parameters" by return values Martin von Zweigbergk <martinvonz@google.com> parents: 27648 diff changeset	206 def _verifychangelog(self):
41880 2f1f475e9646 verify: document `_verifychangelog` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41879 diff changeset	207 """verify the changelog of a repository
2f1f475e9646 verify: document `_verifychangelog` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41879 diff changeset	208
2f1f475e9646 verify: document `_verifychangelog` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41879 diff changeset	209 The following checks are performed:
2f1f475e9646 verify: document `_verifychangelog` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41879 diff changeset	210 - all of `_checkrevlog` checks,
2f1f475e9646 verify: document `_verifychangelog` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41879 diff changeset	211 - all of `_checkentry` checks (for each revisions),
2f1f475e9646 verify: document `_verifychangelog` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41879 diff changeset	212 - each revision can be read.
2f1f475e9646 verify: document `_verifychangelog` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41879 diff changeset	213
2f1f475e9646 verify: document `_verifychangelog` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41879 diff changeset	214 The function returns some of the data observed in the changesets as a
2f1f475e9646 verify: document `_verifychangelog` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41879 diff changeset	215 (mflinkrevs, filelinkrevs) tuples:
2f1f475e9646 verify: document `_verifychangelog` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41879 diff changeset	216 - mflinkrevs: is a { manifest-node -> [changelog-rev] } mapping
2f1f475e9646 verify: document `_verifychangelog` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41879 diff changeset	217 - filelinkrevs: is a { file-path -> [changelog-rev] } mapping
2f1f475e9646 verify: document `_verifychangelog` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41879 diff changeset	218
2f1f475e9646 verify: document `_verifychangelog` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41879 diff changeset	219 If a matcher was specified, filelinkrevs will only contains matched
2f1f475e9646 verify: document `_verifychangelog` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41879 diff changeset	220 files.
2f1f475e9646 verify: document `_verifychangelog` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41879 diff changeset	221 """
27647 2c2858f3c1bb verify: move changelog verificaiton to its own function Durham Goode <durham@fb.com> parents: 27646 diff changeset	222 ui = self.ui
2c2858f3c1bb verify: move changelog verificaiton to its own function Durham Goode <durham@fb.com> parents: 27646 diff changeset	223 repo = self.repo
30900 5249b6470de9 verify: replace _validpath() by matcher Martin von Zweigbergk <martinvonz@google.com> parents: 30403 diff changeset	224 match = self.match
27647 2c2858f3c1bb verify: move changelog verificaiton to its own function Durham Goode <durham@fb.com> parents: 27646 diff changeset	225 cl = repo.changelog
2c2858f3c1bb verify: move changelog verificaiton to its own function Durham Goode <durham@fb.com> parents: 27646 diff changeset	226
27443 937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	227 ui.status(_("checking changesets\n"))
27695 fb0cc863d172 verify: replace "output parameters" by return values Martin von Zweigbergk <martinvonz@google.com> parents: 27648 diff changeset	228 mflinkrevs = {}
fb0cc863d172 verify: replace "output parameters" by return values Martin von Zweigbergk <martinvonz@google.com> parents: 27648 diff changeset	229 filelinkrevs = {}
27443 937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	230 seen = {}
41879 1f412223f5bb verify: rename the `checklog` to `_checkrevlog` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41878 diff changeset	231 self._checkrevlog(cl, "changelog", 0)
38403 1249475f0bd6 verify: use progress helper Martin von Zweigbergk <martinvonz@google.com> parents: 38402 diff changeset	232 progress = ui.makeprogress(_('checking'), unit=_('changesets'),
1249475f0bd6 verify: use progress helper Martin von Zweigbergk <martinvonz@google.com> parents: 38402 diff changeset	233 total=len(repo))
27443 937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	234 for i in repo:
38403 1249475f0bd6 verify: use progress helper Martin von Zweigbergk <martinvonz@google.com> parents: 38402 diff changeset	235 progress.update(i)
27443 937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	236 n = cl.node(i)
41876 9e737ca539f6 verify: make `checkentry` a private method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41875 diff changeset	237 self._checkentry(cl, i, n, seen, [i], "changelog")
2778 fdc232d8a193 Move repo.verify Matt Mackall <mpm@selenic.com> parents: diff changeset	238
27443 937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	239 try:
937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	240 changes = cl.read(n)
937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	241 if changes[0] != nullid:
937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	242 mflinkrevs.setdefault(changes[0], []).append(i)
27444 6647401858ab verify: move widely used variables into class members Durham Goode <durham@fb.com> parents: 27443 diff changeset	243 self.refersmf = True
27443 937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	244 for f in changes[3]:
30900 5249b6470de9 verify: replace _validpath() by matcher Martin von Zweigbergk <martinvonz@google.com> parents: 30403 diff changeset	245 if match(f):
27443 937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	246 filelinkrevs.setdefault(_normpath(f), []).append(i)
937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	247 except Exception as inst:
27444 6647401858ab verify: move widely used variables into class members Durham Goode <durham@fb.com> parents: 27443 diff changeset	248 self.refersmf = True
41871 cfe08588d711 verify: make the `exc` method private Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41870 diff changeset	249 self._exc(i, _("unpacking changeset %s") % short(n), inst)
38403 1249475f0bd6 verify: use progress helper Martin von Zweigbergk <martinvonz@google.com> parents: 38402 diff changeset	250 progress.complete()
27695 fb0cc863d172 verify: replace "output parameters" by return values Martin von Zweigbergk <martinvonz@google.com> parents: 27648 diff changeset	251 return mflinkrevs, filelinkrevs
2778 fdc232d8a193 Move repo.verify Matt Mackall <mpm@selenic.com> parents: diff changeset	252
28205 53f42c8d5f71 verify: show progress while verifying dirlogs Martin von Zweigbergk <martinvonz@google.com> parents: 28204 diff changeset	253 def _verifymanifest(self, mflinkrevs, dir="", storefiles=None,
38402 0ddbe03c5aaa verify: use progress helper for subdirectory progress Martin von Zweigbergk <martinvonz@google.com> parents: 37417 diff changeset	254 subdirprogress=None):
41881 5ad5a70df2f7 verify: document the `_verifymanifest` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41880 diff changeset	255 """verify the manifestlog content
5ad5a70df2f7 verify: document the `_verifymanifest` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41880 diff changeset	256
5ad5a70df2f7 verify: document the `_verifymanifest` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41880 diff changeset	257 Inputs:
5ad5a70df2f7 verify: document the `_verifymanifest` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41880 diff changeset	258 - mflinkrevs: a {manifest-node -> [changelog-revisions]} mapping
5ad5a70df2f7 verify: document the `_verifymanifest` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41880 diff changeset	259 - dir: a subdirectory to check (for tree manifest repo)
5ad5a70df2f7 verify: document the `_verifymanifest` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41880 diff changeset	260 - storefiles: set of currently "orphan" files.
5ad5a70df2f7 verify: document the `_verifymanifest` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41880 diff changeset	261 - subdirprogress: a progress object
5ad5a70df2f7 verify: document the `_verifymanifest` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41880 diff changeset	262
5ad5a70df2f7 verify: document the `_verifymanifest` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41880 diff changeset	263 This function checks:
5ad5a70df2f7 verify: document the `_verifymanifest` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41880 diff changeset	264 * all of `_checkrevlog` checks (for all manifest related revlogs)
5ad5a70df2f7 verify: document the `_verifymanifest` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41880 diff changeset	265 * all of `_checkentry` checks (for all manifest related revisions)
5ad5a70df2f7 verify: document the `_verifymanifest` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41880 diff changeset	266 * nodes for subdirectory exists in the sub-directory manifest
5ad5a70df2f7 verify: document the `_verifymanifest` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41880 diff changeset	267 * each manifest entries have a file path
5ad5a70df2f7 verify: document the `_verifymanifest` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41880 diff changeset	268 * each manifest node refered in mflinkrevs exist in the manifest log
5ad5a70df2f7 verify: document the `_verifymanifest` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41880 diff changeset	269
5ad5a70df2f7 verify: document the `_verifymanifest` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41880 diff changeset	270 If tree manifest is in use and a matchers is specified, only the
5ad5a70df2f7 verify: document the `_verifymanifest` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41880 diff changeset	271 sub-directories matching it will be verified.
5ad5a70df2f7 verify: document the `_verifymanifest` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41880 diff changeset	272
5ad5a70df2f7 verify: document the `_verifymanifest` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41880 diff changeset	273 return a two level mapping:
5ad5a70df2f7 verify: document the `_verifymanifest` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41880 diff changeset	274 {"path" -> { filenode -> changelog-revision}}
5ad5a70df2f7 verify: document the `_verifymanifest` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41880 diff changeset	275
5ad5a70df2f7 verify: document the `_verifymanifest` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41880 diff changeset	276 This mapping primarily contains entries for every files in the
5ad5a70df2f7 verify: document the `_verifymanifest` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41880 diff changeset	277 repository. In addition, when tree-manifest is used, it also contains
5ad5a70df2f7 verify: document the `_verifymanifest` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41880 diff changeset	278 sub-directory entries.
5ad5a70df2f7 verify: document the `_verifymanifest` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41880 diff changeset	279
5ad5a70df2f7 verify: document the `_verifymanifest` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41880 diff changeset	280 If a matcher is provided, only matching paths will be included.
5ad5a70df2f7 verify: document the `_verifymanifest` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41880 diff changeset	281 """
27646 8f43793382c6 verify: move manifest verification to its own function Durham Goode <durham@fb.com> parents: 27645 diff changeset	282 repo = self.repo
8f43793382c6 verify: move manifest verification to its own function Durham Goode <durham@fb.com> parents: 27645 diff changeset	283 ui = self.ui
30900 5249b6470de9 verify: replace _validpath() by matcher Martin von Zweigbergk <martinvonz@google.com> parents: 30403 diff changeset	284 match = self.match
30309 f65faa4422c8 manifest: remove manifest.readshallowdelta Durham Goode <durham@fb.com> parents: 28467 diff changeset	285 mfl = self.repo.manifestlog
39272 73cf21b2e8a6 manifest: add getstorage() to manifestlog and use it globally Gregory Szorc <gregory.szorc@gmail.com> parents: 38403 diff changeset	286 mf = mfl.getstorage(dir)
27646 8f43793382c6 verify: move manifest verification to its own function Durham Goode <durham@fb.com> parents: 27645 diff changeset	287
28203 7297e9e13a8a verify: check directory manifests Martin von Zweigbergk <martinvonz@google.com> parents: 28115 diff changeset	288 if not dir:
7297e9e13a8a verify: check directory manifests Martin von Zweigbergk <martinvonz@google.com> parents: 28115 diff changeset	289 self.ui.status(_("checking manifests\n"))
7297e9e13a8a verify: check directory manifests Martin von Zweigbergk <martinvonz@google.com> parents: 28115 diff changeset	290
27695 fb0cc863d172 verify: replace "output parameters" by return values Martin von Zweigbergk <martinvonz@google.com> parents: 27648 diff changeset	291 filenodes = {}
28203 7297e9e13a8a verify: check directory manifests Martin von Zweigbergk <martinvonz@google.com> parents: 28115 diff changeset	292 subdirnodes = {}
27443 937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	293 seen = {}
28115 bd279da57e4b verify: extract "manifest" constant into variable Martin von Zweigbergk <martinvonz@google.com> parents: 28114 diff changeset	294 label = "manifest"
28203 7297e9e13a8a verify: check directory manifests Martin von Zweigbergk <martinvonz@google.com> parents: 28115 diff changeset	295 if dir:
7297e9e13a8a verify: check directory manifests Martin von Zweigbergk <martinvonz@google.com> parents: 28115 diff changeset	296 label = dir
28204 962921c330b0 verify: check for orphaned dirlogs Martin von Zweigbergk <martinvonz@google.com> parents: 28203 diff changeset	297 revlogfiles = mf.files()
962921c330b0 verify: check for orphaned dirlogs Martin von Zweigbergk <martinvonz@google.com> parents: 28203 diff changeset	298 storefiles.difference_update(revlogfiles)
38402 0ddbe03c5aaa verify: use progress helper for subdirectory progress Martin von Zweigbergk <martinvonz@google.com> parents: 37417 diff changeset	299 if subdirprogress: # should be true since we're in a subdirectory
0ddbe03c5aaa verify: use progress helper for subdirectory progress Martin von Zweigbergk <martinvonz@google.com> parents: 37417 diff changeset	300 subdirprogress.increment()
27444 6647401858ab verify: move widely used variables into class members Durham Goode <durham@fb.com> parents: 27443 diff changeset	301 if self.refersmf:
27443 937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	302 # Do not check manifest if there are only changelog entries with
937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	303 # null manifests.
41879 1f412223f5bb verify: rename the `checklog` to `_checkrevlog` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41878 diff changeset	304 self._checkrevlog(mf, label, 0)
38403 1249475f0bd6 verify: use progress helper Martin von Zweigbergk <martinvonz@google.com> parents: 38402 diff changeset	305 progress = ui.makeprogress(_('checking'), unit=_('manifests'),
1249475f0bd6 verify: use progress helper Martin von Zweigbergk <martinvonz@google.com> parents: 38402 diff changeset	306 total=len(mf))
27443 937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	307 for i in mf:
28203 7297e9e13a8a verify: check directory manifests Martin von Zweigbergk <martinvonz@google.com> parents: 28115 diff changeset	308 if not dir:
38403 1249475f0bd6 verify: use progress helper Martin von Zweigbergk <martinvonz@google.com> parents: 38402 diff changeset	309 progress.update(i)
27443 937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	310 n = mf.node(i)
41876 9e737ca539f6 verify: make `checkentry` a private method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41875 diff changeset	311 lr = self._checkentry(mf, i, n, seen, mflinkrevs.get(n, []), label)
27443 937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	312 if n in mflinkrevs:
937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	313 del mflinkrevs[n]
28203 7297e9e13a8a verify: check directory manifests Martin von Zweigbergk <martinvonz@google.com> parents: 28115 diff changeset	314 elif dir:
41869 7eaf4b1ac2a3 verify: make `err` a private method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41868 diff changeset	315 self._err(lr, _("%s not in parent-directory manifest") %
28203 7297e9e13a8a verify: check directory manifests Martin von Zweigbergk <martinvonz@google.com> parents: 28115 diff changeset	316 short(n), label)
27443 937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	317 else:
41869 7eaf4b1ac2a3 verify: make `err` a private method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41868 diff changeset	318 self._err(lr, _("%s not in changesets") % short(n), label)
2778 fdc232d8a193 Move repo.verify Matt Mackall <mpm@selenic.com> parents: diff changeset	319
27443 937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	320 try:
30309 f65faa4422c8 manifest: remove manifest.readshallowdelta Durham Goode <durham@fb.com> parents: 28467 diff changeset	321 mfdelta = mfl.get(dir, n).readdelta(shallow=True)
f65faa4422c8 manifest: remove manifest.readshallowdelta Durham Goode <durham@fb.com> parents: 28467 diff changeset	322 for f, fn, fl in mfdelta.iterentries():
27443 937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	323 if not f:
41869 7eaf4b1ac2a3 verify: make `err` a private method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41868 diff changeset	324 self._err(lr, _("entry without name in manifest"))
28203 7297e9e13a8a verify: check directory manifests Martin von Zweigbergk <martinvonz@google.com> parents: 28115 diff changeset	325 elif f == "/dev/null": # ignore this in very old repos
7297e9e13a8a verify: check directory manifests Martin von Zweigbergk <martinvonz@google.com> parents: 28115 diff changeset	326 continue
7297e9e13a8a verify: check directory manifests Martin von Zweigbergk <martinvonz@google.com> parents: 28115 diff changeset	327 fullpath = dir + _normpath(f)
7297e9e13a8a verify: check directory manifests Martin von Zweigbergk <martinvonz@google.com> parents: 28115 diff changeset	328 if fl == 't':
30900 5249b6470de9 verify: replace _validpath() by matcher Martin von Zweigbergk <martinvonz@google.com> parents: 30403 diff changeset	329 if not match.visitdir(fullpath):
5249b6470de9 verify: replace _validpath() by matcher Martin von Zweigbergk <martinvonz@google.com> parents: 30403 diff changeset	330 continue
28203 7297e9e13a8a verify: check directory manifests Martin von Zweigbergk <martinvonz@google.com> parents: 28115 diff changeset	331 subdirnodes.setdefault(fullpath + '/', {}).setdefault(
7297e9e13a8a verify: check directory manifests Martin von Zweigbergk <martinvonz@google.com> parents: 28115 diff changeset	332 fn, []).append(lr)
7297e9e13a8a verify: check directory manifests Martin von Zweigbergk <martinvonz@google.com> parents: 28115 diff changeset	333 else:
30900 5249b6470de9 verify: replace _validpath() by matcher Martin von Zweigbergk <martinvonz@google.com> parents: 30403 diff changeset	334 if not match(fullpath):
5249b6470de9 verify: replace _validpath() by matcher Martin von Zweigbergk <martinvonz@google.com> parents: 30403 diff changeset	335 continue
28203 7297e9e13a8a verify: check directory manifests Martin von Zweigbergk <martinvonz@google.com> parents: 28115 diff changeset	336 filenodes.setdefault(fullpath, {}).setdefault(fn, lr)
27443 937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	337 except Exception as inst:
41871 cfe08588d711 verify: make the `exc` method private Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41870 diff changeset	338 self._exc(lr, _("reading delta %s") % short(n), inst, label)
28203 7297e9e13a8a verify: check directory manifests Martin von Zweigbergk <martinvonz@google.com> parents: 28115 diff changeset	339 if not dir:
38403 1249475f0bd6 verify: use progress helper Martin von Zweigbergk <martinvonz@google.com> parents: 38402 diff changeset	340 progress.complete()
27443 937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	341
28111 06205989264b verify: move cross-checking of changeset/manifest out of _crosscheckfiles() Martin von Zweigbergk <martinvonz@google.com> parents: 28007 diff changeset	342 if self.havemf:
41882 9c5a6af74afa verify: small refactoring and documentation in `_verifymanifest` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41881 diff changeset	343 # since we delete entry in `mflinkrevs` during iteration, any
9c5a6af74afa verify: small refactoring and documentation in `_verifymanifest` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41881 diff changeset	344 # remaining entries are "missing". We need to issue errors for them.
9c5a6af74afa verify: small refactoring and documentation in `_verifymanifest` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41881 diff changeset	345 changesetpairs = [(c, m) for m in mflinkrevs for c in mflinkrevs[m]]
9c5a6af74afa verify: small refactoring and documentation in `_verifymanifest` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41881 diff changeset	346 for c, m in sorted(changesetpairs):
28203 7297e9e13a8a verify: check directory manifests Martin von Zweigbergk <martinvonz@google.com> parents: 28115 diff changeset	347 if dir:
41869 7eaf4b1ac2a3 verify: make `err` a private method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41868 diff changeset	348 self._err(c, _("parent-directory manifest refers to unknown"
7eaf4b1ac2a3 verify: make `err` a private method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41868 diff changeset	349 " revision %s") % short(m), label)
28203 7297e9e13a8a verify: check directory manifests Martin von Zweigbergk <martinvonz@google.com> parents: 28115 diff changeset	350 else:
41869 7eaf4b1ac2a3 verify: make `err` a private method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41868 diff changeset	351 self._err(c, _("changeset refers to unknown revision %s") %
7eaf4b1ac2a3 verify: make `err` a private method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41868 diff changeset	352 short(m), label)
28203 7297e9e13a8a verify: check directory manifests Martin von Zweigbergk <martinvonz@google.com> parents: 28115 diff changeset	353
7297e9e13a8a verify: check directory manifests Martin von Zweigbergk <martinvonz@google.com> parents: 28115 diff changeset	354 if not dir and subdirnodes:
7297e9e13a8a verify: check directory manifests Martin von Zweigbergk <martinvonz@google.com> parents: 28115 diff changeset	355 self.ui.status(_("checking directory manifests\n"))
28204 962921c330b0 verify: check for orphaned dirlogs Martin von Zweigbergk <martinvonz@google.com> parents: 28203 diff changeset	356 storefiles = set()
28205 53f42c8d5f71 verify: show progress while verifying dirlogs Martin von Zweigbergk <martinvonz@google.com> parents: 28204 diff changeset	357 subdirs = set()
28204 962921c330b0 verify: check for orphaned dirlogs Martin von Zweigbergk <martinvonz@google.com> parents: 28203 diff changeset	358 revlogv1 = self.revlogv1
962921c330b0 verify: check for orphaned dirlogs Martin von Zweigbergk <martinvonz@google.com> parents: 28203 diff changeset	359 for f, f2, size in repo.store.datafiles():
962921c330b0 verify: check for orphaned dirlogs Martin von Zweigbergk <martinvonz@google.com> parents: 28203 diff changeset	360 if not f:
41869 7eaf4b1ac2a3 verify: make `err` a private method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41868 diff changeset	361 self._err(None, _("cannot decode filename '%s'") % f2)
28204 962921c330b0 verify: check for orphaned dirlogs Martin von Zweigbergk <martinvonz@google.com> parents: 28203 diff changeset	362 elif (size > 0 or not revlogv1) and f.startswith('meta/'):
962921c330b0 verify: check for orphaned dirlogs Martin von Zweigbergk <martinvonz@google.com> parents: 28203 diff changeset	363 storefiles.add(_normpath(f))
28205 53f42c8d5f71 verify: show progress while verifying dirlogs Martin von Zweigbergk <martinvonz@google.com> parents: 28204 diff changeset	364 subdirs.add(os.path.dirname(f))
38402 0ddbe03c5aaa verify: use progress helper for subdirectory progress Martin von Zweigbergk <martinvonz@google.com> parents: 37417 diff changeset	365 subdirprogress = ui.makeprogress(_('checking'), unit=_('manifests'),
0ddbe03c5aaa verify: use progress helper for subdirectory progress Martin von Zweigbergk <martinvonz@google.com> parents: 37417 diff changeset	366 total=len(subdirs))
28204 962921c330b0 verify: check for orphaned dirlogs Martin von Zweigbergk <martinvonz@google.com> parents: 28203 diff changeset	367
28203 7297e9e13a8a verify: check directory manifests Martin von Zweigbergk <martinvonz@google.com> parents: 28115 diff changeset	368 for subdir, linkrevs in subdirnodes.iteritems():
28205 53f42c8d5f71 verify: show progress while verifying dirlogs Martin von Zweigbergk <martinvonz@google.com> parents: 28204 diff changeset	369 subdirfilenodes = self._verifymanifest(linkrevs, subdir, storefiles,
38402 0ddbe03c5aaa verify: use progress helper for subdirectory progress Martin von Zweigbergk <martinvonz@google.com> parents: 37417 diff changeset	370 subdirprogress)
28203 7297e9e13a8a verify: check directory manifests Martin von Zweigbergk <martinvonz@google.com> parents: 28115 diff changeset	371 for f, onefilenodes in subdirfilenodes.iteritems():
7297e9e13a8a verify: check directory manifests Martin von Zweigbergk <martinvonz@google.com> parents: 28115 diff changeset	372 filenodes.setdefault(f, {}).update(onefilenodes)
28111 06205989264b verify: move cross-checking of changeset/manifest out of _crosscheckfiles() Martin von Zweigbergk <martinvonz@google.com> parents: 28007 diff changeset	373
28204 962921c330b0 verify: check for orphaned dirlogs Martin von Zweigbergk <martinvonz@google.com> parents: 28203 diff changeset	374 if not dir and subdirnodes:
38402 0ddbe03c5aaa verify: use progress helper for subdirectory progress Martin von Zweigbergk <martinvonz@google.com> parents: 37417 diff changeset	375 subdirprogress.complete()
37417 76d2115cb817 verify: allow suppressing warnings about extra files Gregory Szorc <gregory.szorc@gmail.com> parents: 37410 diff changeset	376 if self.warnorphanstorefiles:
76d2115cb817 verify: allow suppressing warnings about extra files Gregory Szorc <gregory.szorc@gmail.com> parents: 37410 diff changeset	377 for f in sorted(storefiles):
41867 c66037fb1bc5 verify: make the `warn` method private Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41866 diff changeset	378 self._warn(_("warning: orphan data file '%s'") % f)
28204 962921c330b0 verify: check for orphaned dirlogs Martin von Zweigbergk <martinvonz@google.com> parents: 28203 diff changeset	379
27695 fb0cc863d172 verify: replace "output parameters" by return values Martin von Zweigbergk <martinvonz@google.com> parents: 27648 diff changeset	380 return filenodes
27645 df8973e1fb45 verify: move file cross checking to its own function Durham Goode <durham@fb.com> parents: 27644 diff changeset	381
28111 06205989264b verify: move cross-checking of changeset/manifest out of _crosscheckfiles() Martin von Zweigbergk <martinvonz@google.com> parents: 28007 diff changeset	382 def _crosscheckfiles(self, filelinkrevs, filenodes):
27645 df8973e1fb45 verify: move file cross checking to its own function Durham Goode <durham@fb.com> parents: 27644 diff changeset	383 repo = self.repo
df8973e1fb45 verify: move file cross checking to its own function Durham Goode <durham@fb.com> parents: 27644 diff changeset	384 ui = self.ui
27443 937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	385 ui.status(_("crosschecking files in changesets and manifests\n"))
2778 fdc232d8a193 Move repo.verify Matt Mackall <mpm@selenic.com> parents: diff changeset	386
28111 06205989264b verify: move cross-checking of changeset/manifest out of _crosscheckfiles() Martin von Zweigbergk <martinvonz@google.com> parents: 28007 diff changeset	387 total = len(filelinkrevs) + len(filenodes)
40437 d2ff0af6e959 verify: provide unit to ui.makeprogress() Anton Shestakov <av6@dwimlabs.net> parents: 39943 diff changeset	388 progress = ui.makeprogress(_('crosschecking'), unit=_('files'),
d2ff0af6e959 verify: provide unit to ui.makeprogress() Anton Shestakov <av6@dwimlabs.net> parents: 39943 diff changeset	389 total=total)
27645 df8973e1fb45 verify: move file cross checking to its own function Durham Goode <durham@fb.com> parents: 27644 diff changeset	390 if self.havemf:
27443 937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	391 for f in sorted(filelinkrevs):
38403 1249475f0bd6 verify: use progress helper Martin von Zweigbergk <martinvonz@google.com> parents: 38402 diff changeset	392 progress.increment()
27443 937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	393 if f not in filenodes:
937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	394 lr = filelinkrevs[f][0]
41869 7eaf4b1ac2a3 verify: make `err` a private method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41868 diff changeset	395 self._err(lr, _("in changeset but not in manifest"), f)
6892 dab95717058d verify: check repo.store Adrian Buehlmann <adrian@cadifra.com> parents: 6889 diff changeset	396
27645 df8973e1fb45 verify: move file cross checking to its own function Durham Goode <durham@fb.com> parents: 27644 diff changeset	397 if self.havecl:
27443 937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	398 for f in sorted(filenodes):
38403 1249475f0bd6 verify: use progress helper Martin von Zweigbergk <martinvonz@google.com> parents: 38402 diff changeset	399 progress.increment()
27443 937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	400 if f not in filelinkrevs:
937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	401 try:
937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	402 fl = repo.file(f)
937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	403 lr = min([fl.linkrev(fl.rev(n)) for n in filenodes[f]])
937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	404 except Exception:
937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	405 lr = None
41869 7eaf4b1ac2a3 verify: make `err` a private method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41868 diff changeset	406 self._err(lr, _("in manifest but not in changeset"), f)
27443 937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	407
38403 1249475f0bd6 verify: use progress helper Martin von Zweigbergk <martinvonz@google.com> parents: 38402 diff changeset	408 progress.complete()
8291 f5c1a9094e41 verify: avoid exception on missing file revlog Henrik Stuart <hg@hstuart.dk> parents: 8225 diff changeset	409
27644 331e5c28f5f0 verify: move filelog verification to its own function Durham Goode <durham@fb.com> parents: 27643 diff changeset	410 def _verifyfiles(self, filenodes, filelinkrevs):
331e5c28f5f0 verify: move filelog verification to its own function Durham Goode <durham@fb.com> parents: 27643 diff changeset	411 repo = self.repo
331e5c28f5f0 verify: move filelog verification to its own function Durham Goode <durham@fb.com> parents: 27643 diff changeset	412 ui = self.ui
331e5c28f5f0 verify: move filelog verification to its own function Durham Goode <durham@fb.com> parents: 27643 diff changeset	413 lrugetctx = self.lrugetctx
331e5c28f5f0 verify: move filelog verification to its own function Durham Goode <durham@fb.com> parents: 27643 diff changeset	414 revlogv1 = self.revlogv1
331e5c28f5f0 verify: move filelog verification to its own function Durham Goode <durham@fb.com> parents: 27643 diff changeset	415 havemf = self.havemf
27443 937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	416 ui.status(_("checking files\n"))
8291 f5c1a9094e41 verify: avoid exception on missing file revlog Henrik Stuart <hg@hstuart.dk> parents: 8225 diff changeset	417
27443 937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	418 storefiles = set()
937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	419 for f, f2, size in repo.store.datafiles():
937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	420 if not f:
41869 7eaf4b1ac2a3 verify: make `err` a private method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41868 diff changeset	421 self._err(None, _("cannot decode filename '%s'") % f2)
28007 fb92927f9775 treemanifests: fix streaming clone Martin von Zweigbergk <martinvonz@google.com> parents: 27964 diff changeset	422 elif (size > 0 or not revlogv1) and f.startswith('data/'):
27443 937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	423 storefiles.add(_normpath(f))
6892 dab95717058d verify: check repo.store Adrian Buehlmann <adrian@cadifra.com> parents: 6889 diff changeset	424
39847 97986c9c69d3 verify: start to abstract file verification Gregory Szorc <gregory.szorc@gmail.com> parents: 39793 diff changeset	425 state = {
39850 e6d3d39cc1c7 revlog: use proper version comparison during verify Gregory Szorc <gregory.szorc@gmail.com> parents: 39847 diff changeset	426 # TODO this assumes revlog storage for changelog.
39877 733db72f0f54 revlog: move revision verification out of verify Gregory Szorc <gregory.szorc@gmail.com> parents: 39850 diff changeset	427 'expectedversion': self.repo.changelog.version & 0xFFFF,
733db72f0f54 revlog: move revision verification out of verify Gregory Szorc <gregory.szorc@gmail.com> parents: 39850 diff changeset	428 'skipflags': self.skipflags,
733db72f0f54 revlog: move revision verification out of verify Gregory Szorc <gregory.szorc@gmail.com> parents: 39850 diff changeset	429 # experimental config: censor.policy
733db72f0f54 revlog: move revision verification out of verify Gregory Szorc <gregory.szorc@gmail.com> parents: 39850 diff changeset	430 'erroroncensored': ui.config('censor', 'policy') == 'abort',
39847 97986c9c69d3 verify: start to abstract file verification Gregory Szorc <gregory.szorc@gmail.com> parents: 39793 diff changeset	431 }
97986c9c69d3 verify: start to abstract file verification Gregory Szorc <gregory.szorc@gmail.com> parents: 39793 diff changeset	432
27443 937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	433 files = sorted(set(filenodes) \| set(filelinkrevs))
27644 331e5c28f5f0 verify: move filelog verification to its own function Durham Goode <durham@fb.com> parents: 27643 diff changeset	434 revisions = 0
38403 1249475f0bd6 verify: use progress helper Martin von Zweigbergk <martinvonz@google.com> parents: 38402 diff changeset	435 progress = ui.makeprogress(_('checking'), unit=_('files'),
1249475f0bd6 verify: use progress helper Martin von Zweigbergk <martinvonz@google.com> parents: 38402 diff changeset	436 total=len(files))
27443 937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	437 for i, f in enumerate(files):
38403 1249475f0bd6 verify: use progress helper Martin von Zweigbergk <martinvonz@google.com> parents: 38402 diff changeset	438 progress.update(i, item=f)
6892 dab95717058d verify: check repo.store Adrian Buehlmann <adrian@cadifra.com> parents: 6889 diff changeset	439 try:
27443 937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	440 linkrevs = filelinkrevs[f]
6892 dab95717058d verify: check repo.store Adrian Buehlmann <adrian@cadifra.com> parents: 6889 diff changeset	441 except KeyError:
27443 937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	442 # in manifest but not in changelog
937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	443 linkrevs = []
2778 fdc232d8a193 Move repo.verify Matt Mackall <mpm@selenic.com> parents: diff changeset	444
27443 937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	445 if linkrevs:
937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	446 lr = linkrevs[0]
937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	447 else:
937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	448 lr = None
2778 fdc232d8a193 Move repo.verify Matt Mackall <mpm@selenic.com> parents: diff changeset	449
3744 d626fc9e3985 verify: add rename link checking Matt Mackall <mpm@selenic.com> parents: 3473 diff changeset	450 try:
27443 937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	451 fl = repo.file(f)
39793 b63dee7bd0d9 global: replace most uses of RevlogError with StorageError (API) Gregory Szorc <gregory.szorc@gmail.com> parents: 39506 diff changeset	452 except error.StorageError as e:
41869 7eaf4b1ac2a3 verify: make `err` a private method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41868 diff changeset	453 self._err(lr, _("broken revlog! (%s)") % e, f)
27443 937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	454 continue
937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	455
937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	456 for ff in fl.files():
937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	457 try:
937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	458 storefiles.remove(ff)
937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	459 except KeyError:
37417 76d2115cb817 verify: allow suppressing warnings about extra files Gregory Szorc <gregory.szorc@gmail.com> parents: 37410 diff changeset	460 if self.warnorphanstorefiles:
41867 c66037fb1bc5 verify: make the `warn` method private Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41866 diff changeset	461 self._warn(_(" warning: revlog '%s' not in fncache!") %
37417 76d2115cb817 verify: allow suppressing warnings about extra files Gregory Szorc <gregory.szorc@gmail.com> parents: 37410 diff changeset	462 ff)
76d2115cb817 verify: allow suppressing warnings about extra files Gregory Szorc <gregory.szorc@gmail.com> parents: 37410 diff changeset	463 self.fncachewarned = True
27443 937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	464
39847 97986c9c69d3 verify: start to abstract file verification Gregory Szorc <gregory.szorc@gmail.com> parents: 39793 diff changeset	465 if not len(fl) and (self.havecl or self.havemf):
41869 7eaf4b1ac2a3 verify: make `err` a private method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41868 diff changeset	466 self._err(lr, _("empty or missing %s") % f)
39847 97986c9c69d3 verify: start to abstract file verification Gregory Szorc <gregory.szorc@gmail.com> parents: 39793 diff changeset	467 else:
39877 733db72f0f54 revlog: move revision verification out of verify Gregory Szorc <gregory.szorc@gmail.com> parents: 39850 diff changeset	468 # Guard against implementations not setting this.
733db72f0f54 revlog: move revision verification out of verify Gregory Szorc <gregory.szorc@gmail.com> parents: 39850 diff changeset	469 state['skipread'] = set()
39847 97986c9c69d3 verify: start to abstract file verification Gregory Szorc <gregory.szorc@gmail.com> parents: 39793 diff changeset	470 for problem in fl.verifyintegrity(state):
39877 733db72f0f54 revlog: move revision verification out of verify Gregory Szorc <gregory.szorc@gmail.com> parents: 39850 diff changeset	471 if problem.node is not None:
733db72f0f54 revlog: move revision verification out of verify Gregory Szorc <gregory.szorc@gmail.com> parents: 39850 diff changeset	472 linkrev = fl.linkrev(fl.rev(problem.node))
733db72f0f54 revlog: move revision verification out of verify Gregory Szorc <gregory.szorc@gmail.com> parents: 39850 diff changeset	473 else:
733db72f0f54 revlog: move revision verification out of verify Gregory Szorc <gregory.szorc@gmail.com> parents: 39850 diff changeset	474 linkrev = None
733db72f0f54 revlog: move revision verification out of verify Gregory Szorc <gregory.szorc@gmail.com> parents: 39850 diff changeset	475
39847 97986c9c69d3 verify: start to abstract file verification Gregory Szorc <gregory.szorc@gmail.com> parents: 39793 diff changeset	476 if problem.warning:
41867 c66037fb1bc5 verify: make the `warn` method private Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41866 diff changeset	477 self._warn(problem.warning)
39847 97986c9c69d3 verify: start to abstract file verification Gregory Szorc <gregory.szorc@gmail.com> parents: 39793 diff changeset	478 elif problem.error:
41869 7eaf4b1ac2a3 verify: make `err` a private method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41868 diff changeset	479 self._err(linkrev if linkrev is not None else lr,
7eaf4b1ac2a3 verify: make `err` a private method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41868 diff changeset	480 problem.error, f)
39847 97986c9c69d3 verify: start to abstract file verification Gregory Szorc <gregory.szorc@gmail.com> parents: 39793 diff changeset	481 else:
97986c9c69d3 verify: start to abstract file verification Gregory Szorc <gregory.szorc@gmail.com> parents: 39793 diff changeset	482 raise error.ProgrammingError(
97986c9c69d3 verify: start to abstract file verification Gregory Szorc <gregory.szorc@gmail.com> parents: 39793 diff changeset	483 'problem instance does not set warning or error '
97986c9c69d3 verify: start to abstract file verification Gregory Szorc <gregory.szorc@gmail.com> parents: 39793 diff changeset	484 'attribute: %s' % problem.msg)
97986c9c69d3 verify: start to abstract file verification Gregory Szorc <gregory.szorc@gmail.com> parents: 39793 diff changeset	485
27443 937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	486 seen = {}
937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	487 for i in fl:
937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	488 revisions += 1
937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	489 n = fl.node(i)
41876 9e737ca539f6 verify: make `checkentry` a private method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41875 diff changeset	490 lr = self._checkentry(fl, i, n, seen, linkrevs, f)
27443 937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	491 if f in filenodes:
937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	492 if havemf and n not in filenodes[f]:
41869 7eaf4b1ac2a3 verify: make `err` a private method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41868 diff changeset	493 self._err(lr, _("%s not in manifests") % (short(n)), f)
6534 9b35a9f34675 verify: check copy source revlog and nodeid Patrick Mezard <pmezard@gmail.com> parents: 6211 diff changeset	494 else:
27443 937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	495 del filenodes[f][n]
937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	496
39877 733db72f0f54 revlog: move revision verification out of verify Gregory Szorc <gregory.szorc@gmail.com> parents: 39850 diff changeset	497 if n in state['skipread']:
733db72f0f54 revlog: move revision verification out of verify Gregory Szorc <gregory.szorc@gmail.com> parents: 39850 diff changeset	498 continue
3744 d626fc9e3985 verify: add rename link checking Matt Mackall <mpm@selenic.com> parents: 3473 diff changeset	499
27443 937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	500 # check renames
937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	501 try:
39877 733db72f0f54 revlog: move revision verification out of verify Gregory Szorc <gregory.szorc@gmail.com> parents: 39850 diff changeset	502 # This requires resolving fulltext (at least on revlogs). We
733db72f0f54 revlog: move revision verification out of verify Gregory Szorc <gregory.szorc@gmail.com> parents: 39850 diff changeset	503 # may want ``verifyintegrity()`` to pass a set of nodes with
733db72f0f54 revlog: move revision verification out of verify Gregory Szorc <gregory.szorc@gmail.com> parents: 39850 diff changeset	504 # rename metadata as an optimization.
733db72f0f54 revlog: move revision verification out of verify Gregory Szorc <gregory.szorc@gmail.com> parents: 39850 diff changeset	505 rp = fl.renamed(n)
27443 937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	506 if rp:
937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	507 if lr is not None and ui.verbose:
937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	508 ctx = lrugetctx(lr)
36372 a4d41ba4ad23 verify: don't reimplement any() Martin von Zweigbergk <martinvonz@google.com> parents: 36223 diff changeset	509 if not any(rp[0] in pctx for pctx in ctx.parents()):
41867 c66037fb1bc5 verify: make the `warn` method private Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41866 diff changeset	510 self._warn(_("warning: copy source of '%s' not"
27446 6b2c1a1871a6 verify: move warn() to a class level function Durham Goode <durham@fb.com> parents: 27445 diff changeset	511 " in parents of %s") % (f, ctx))
27443 937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	512 fl2 = repo.file(rp[0])
937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	513 if not len(fl2):
41869 7eaf4b1ac2a3 verify: make `err` a private method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41868 diff changeset	514 self._err(lr,
7eaf4b1ac2a3 verify: make `err` a private method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41868 diff changeset	515 _("empty or missing copy source revlog "
7eaf4b1ac2a3 verify: make `err` a private method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41868 diff changeset	516 "%s:%s") % (rp[0],
7eaf4b1ac2a3 verify: make `err` a private method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41868 diff changeset	517 short(rp[1])),
7eaf4b1ac2a3 verify: make `err` a private method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41868 diff changeset	518 f)
27443 937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	519 elif rp[1] == nullid:
937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	520 ui.note(_("warning: %s@%s: copy source"
937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	521 " revision is nullid %s:%s\n")
937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	522 % (f, lr, rp[0], short(rp[1])))
937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	523 else:
937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	524 fl2.rev(rp[1])
937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	525 except Exception as inst:
41871 cfe08588d711 verify: make the `exc` method private Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41870 diff changeset	526 self._exc(lr, _("checking rename of %s") % short(n),
cfe08588d711 verify: make the `exc` method private Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41870 diff changeset	527 inst, f)
6892 dab95717058d verify: check repo.store Adrian Buehlmann <adrian@cadifra.com> parents: 6889 diff changeset	528
27443 937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	529 # cross-check
937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	530 if f in filenodes:
30403 b667b78099eb verify: avoid shadowing two variables with a list comprehension Augie Fackler <augie@google.com> parents: 30385 diff changeset	531 fns = [(v, k) for k, v in filenodes[f].iteritems()]
27443 937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	532 for lr, node in sorted(fns):
41869 7eaf4b1ac2a3 verify: make `err` a private method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41868 diff changeset	533 self._err(lr, _("manifest refers to unknown revision %s") %
7eaf4b1ac2a3 verify: make `err` a private method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41868 diff changeset	534 short(node), f)
38403 1249475f0bd6 verify: use progress helper Martin von Zweigbergk <martinvonz@google.com> parents: 38402 diff changeset	535 progress.complete()
27443 937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	536
37417 76d2115cb817 verify: allow suppressing warnings about extra files Gregory Szorc <gregory.szorc@gmail.com> parents: 37410 diff changeset	537 if self.warnorphanstorefiles:
76d2115cb817 verify: allow suppressing warnings about extra files Gregory Szorc <gregory.szorc@gmail.com> parents: 37410 diff changeset	538 for f in sorted(storefiles):
41867 c66037fb1bc5 verify: make the `warn` method private Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41866 diff changeset	539 self._warn(_("warning: orphan data file '%s'") % f)
27443 937e73a6e4ff verify: move verify logic into a class Durham Goode <durham@fb.com> parents: 26900 diff changeset	540
27644 331e5c28f5f0 verify: move filelog verification to its own function Durham Goode <durham@fb.com> parents: 27643 diff changeset	541 return len(files), revisions

Mercurial > public > mercurial-scm > hg-stable

annotate mercurial/verify.py @ 42197:57539e5ea2e0