mercurial-scm/hg-stable: mercurial/revlogutils/deltas.py annotate

annotate mercurial/revlogutils/deltas.py @ 49885:d57b966cdeb1

delta-find: properly report full snapshot used from cache as such The number of tries and the delta base is reported differently so we missed there detection initially.

author	Pierre-Yves David <pierre-yves.david@octobus.net>
date	Wed, 07 Dec 2022 20:05:19 +0100
parents	bcae90c53def
children	acdb9a15137c

rev	line source
39357 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	1 # revlogdeltas.py - Logic around delta computation for revlog
8226 8b2cd04a6e97 put license and copyright info into comment blocks Martin Geisler <mg@lazybytes.net> parents: 8225 diff changeset	2 #
46819 d4ba4d51f85f contributor: change mentions of mpm to olivia Rapha?l Gom?s <rgomes@octobus.net> parents: 43089 diff changeset	3 # Copyright 2005-2007 Olivia Mackall <olivia@selenic.com>
39357 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	4 # Copyright 2018 Octobus <contact@octobus.net>
8226 8b2cd04a6e97 put license and copyright info into comment blocks Martin Geisler <mg@lazybytes.net> parents: 8225 diff changeset	5 #
8b2cd04a6e97 put license and copyright info into comment blocks Martin Geisler <mg@lazybytes.net> parents: 8225 diff changeset	6 # This software may be used and distributed according to the terms of the
10263 25e572394f5c Update license to GPLv2+ Matt Mackall <mpm@selenic.com> parents: 10047 diff changeset	7 # GNU General Public License version 2 or any later version.
39357 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	8 """Helper class to compute deltas stored inside revlogs"""
8227 0a9542703300 turn some comments back into module docstrings Martin Geisler <mg@lazybytes.net> parents: 8226 diff changeset	9
27361 29f50344fa83 revlog: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27251 diff changeset	10
39510 3ca144f1c8dd snapshot: search for unrelated but reusable full-snapshot Boris Feld <boris.feld@octobus.net> parents: 39509 diff changeset	11 import collections
27361 29f50344fa83 revlog: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27251 diff changeset	12 import struct
29f50344fa83 revlog: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27251 diff changeset	13
29f50344fa83 revlog: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27251 diff changeset	14 # import stuff from node for others to import from revlog
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	15 from ..node import nullrev
39357 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	16 from ..i18n import _
43089 c59eb1560c44 py3: manually import getattr where it is needed Gregory Szorc <gregory.szorc@gmail.com> parents: 43077 diff changeset	17 from ..pycompat import getattr
39357 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	18
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	19 from .constants import (
47458 c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47407 diff changeset	20 COMP_MODE_DEFAULT,
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47407 diff changeset	21 COMP_MODE_INLINE,
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47407 diff changeset	22 COMP_MODE_PLAIN,
49879 bcae90c53def delta-find: add a delta-reuse policy that blindly accepts incoming deltas Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49795 diff changeset	23 DELTA_BASE_REUSE_FORCE,
49791 05db41701ece find-delta: pass the cache-delta usage policy alongside the cache-delta Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49789 diff changeset	24 DELTA_BASE_REUSE_NO,
49247 e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	25 KIND_CHANGELOG,
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	26 KIND_FILELOG,
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	27 KIND_MANIFESTLOG,
39356 729082bb9938 revlog: split constants into a new `revlogutils.constants` module Boris Feld <boris.feld@octobus.net> parents: 39260 diff changeset	28 REVIDX_ISCENSORED,
729082bb9938 revlog: split constants into a new `revlogutils.constants` module Boris Feld <boris.feld@octobus.net> parents: 39260 diff changeset	29 REVIDX_RAWTEXT_CHANGING_FLAGS,
729082bb9938 revlog: split constants into a new `revlogutils.constants` module Boris Feld <boris.feld@octobus.net> parents: 39260 diff changeset	30 )
39357 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	31
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	32 from ..thirdparty import attr
39357 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	33
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	34 from .. import (
27361 29f50344fa83 revlog: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27251 diff changeset	35 error,
29f50344fa83 revlog: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27251 diff changeset	36 mdiff,
41111 38e88450138c delta: have a native implementation of _findsnapshot Boris Feld <boris.feld@octobus.net> parents: 41082 diff changeset	37 util,
27361 29f50344fa83 revlog: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27251 diff changeset	38 )
10913 f2ecc5733c89 revlog: factor out _maxinline global. Greg Ward <greg-hg@gerg.ca> parents: 10404 diff changeset	39
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	40 from . import flagutil
42994 dff95420480f flagprocessors: make `processflagsraw` a module level function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42884 diff changeset	41
39357 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	42 # maximum <delta-chain-data>/<revision-text-length> ratio
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	43 LIMIT_DELTA2TEXT = 2
1091 d62130f99a73 Move hash function back to revlog from node mpm@selenic.com parents: 1089 diff changeset	44
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	45
49037 642e31cb55f0 py3: use class X: instead of class X(object): Gregory Szorc <gregory.szorc@gmail.com> parents: 48966 diff changeset	46 class _testrevlog:
38637 e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	47 """minimalist fake revlog to use in doctests"""
e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	48
40684 85b14f0dc334 doctest: add a `issnapshot` method to _testrevlog Boris Feld <boris.feld@octobus.net> parents: 40655 diff changeset	49 def __init__(self, data, density=0.5, mingap=0, snapshot=()):
38637 e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	50 """data is an list of revision payload boundaries"""
e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	51 self._data = data
e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	52 self._srdensitythreshold = density
e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	53 self._srmingapsize = mingap
40684 85b14f0dc334 doctest: add a `issnapshot` method to _testrevlog Boris Feld <boris.feld@octobus.net> parents: 40655 diff changeset	54 self._snapshot = set(snapshot)
40748 39d29542fe40 sparse-revlog: put the native implementation of slicechunktodensity to use Boris Feld <boris.feld@octobus.net> parents: 40698 diff changeset	55 self.index = None
38637 e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	56
e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	57 def start(self, rev):
41082 88d813cd9acd revlog: fix pure python slicing test when chain contains nullrev Boris Feld <boris.feld@octobus.net> parents: 41036 diff changeset	58 if rev == nullrev:
88d813cd9acd revlog: fix pure python slicing test when chain contains nullrev Boris Feld <boris.feld@octobus.net> parents: 41036 diff changeset	59 return 0
38637 e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	60 if rev == 0:
e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	61 return 0
e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	62 return self._data[rev - 1]
e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	63
e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	64 def end(self, rev):
41082 88d813cd9acd revlog: fix pure python slicing test when chain contains nullrev Boris Feld <boris.feld@octobus.net> parents: 41036 diff changeset	65 if rev == nullrev:
88d813cd9acd revlog: fix pure python slicing test when chain contains nullrev Boris Feld <boris.feld@octobus.net> parents: 41036 diff changeset	66 return 0
38637 e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	67 return self._data[rev]
e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	68
e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	69 def length(self, rev):
e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	70 return self.end(rev) - self.start(rev)
e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	71
38718 f8762ea73e0d sparse-revlog: implement algorithm to write sparse delta chains (issue5480) Paul Morelle <paul.morelle@octobus.net> parents: 38717 diff changeset	72 def __len__(self):
f8762ea73e0d sparse-revlog: implement algorithm to write sparse delta chains (issue5480) Paul Morelle <paul.morelle@octobus.net> parents: 38717 diff changeset	73 return len(self._data)
f8762ea73e0d sparse-revlog: implement algorithm to write sparse delta chains (issue5480) Paul Morelle <paul.morelle@octobus.net> parents: 38717 diff changeset	74
40684 85b14f0dc334 doctest: add a `issnapshot` method to _testrevlog Boris Feld <boris.feld@octobus.net> parents: 40655 diff changeset	75 def issnapshot(self, rev):
41082 88d813cd9acd revlog: fix pure python slicing test when chain contains nullrev Boris Feld <boris.feld@octobus.net> parents: 41036 diff changeset	76 if rev == nullrev:
88d813cd9acd revlog: fix pure python slicing test when chain contains nullrev Boris Feld <boris.feld@octobus.net> parents: 41036 diff changeset	77 return True
40684 85b14f0dc334 doctest: add a `issnapshot` method to _testrevlog Boris Feld <boris.feld@octobus.net> parents: 40655 diff changeset	78 return rev in self._snapshot
85b14f0dc334 doctest: add a `issnapshot` method to _testrevlog Boris Feld <boris.feld@octobus.net> parents: 40655 diff changeset	79
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	80
40651 3ac23dad6364 sparse-revlog: drop unused deltainfo parameter from _slicechunktodensity Boris Feld <boris.feld@octobus.net> parents: 40650 diff changeset	81 def slicechunk(revlog, revs, targetsize=None):
34824 e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	82 """slice revs to reduce the amount of unrelated data to be read from disk.
e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	83
e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	84 ``revs`` is sliced into groups that should be read in one time.
e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	85 Assume that revs are sorted.
38640 f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	86
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	87 The initial chunk is sliced until the overall density (payload/chunks-span
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	88 ratio) is above `revlog._srdensitythreshold`. No gap smaller than
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	89 `revlog._srmingapsize` is skipped.
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	90
38643 967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size Boris Feld <boris.feld@octobus.net> parents: 38642 diff changeset	91 If `targetsize` is set, no chunk larger than `targetsize` will be yield.
967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size Boris Feld <boris.feld@octobus.net> parents: 38642 diff changeset	92 For consistency with other slicing choice, this limit won't go lower than
967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size Boris Feld <boris.feld@octobus.net> parents: 38642 diff changeset	93 `revlog._srmingapsize`.
967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size Boris Feld <boris.feld@octobus.net> parents: 38642 diff changeset	94
967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size Boris Feld <boris.feld@octobus.net> parents: 38642 diff changeset	95 If individual revisions chunk are larger than this limit, they will still
967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size Boris Feld <boris.feld@octobus.net> parents: 38642 diff changeset	96 be raised individually.
967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size Boris Feld <boris.feld@octobus.net> parents: 38642 diff changeset	97
40685 9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	98 >>> data = [
38640 f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	99 ... 5, #00 (5)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	100 ... 10, #01 (5)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	101 ... 12, #02 (2)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	102 ... 12, #03 (empty)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	103 ... 27, #04 (15)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	104 ... 31, #05 (4)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	105 ... 31, #06 (empty)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	106 ... 42, #07 (11)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	107 ... 47, #08 (5)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	108 ... 47, #09 (empty)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	109 ... 48, #10 (1)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	110 ... 51, #11 (3)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	111 ... 74, #12 (23)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	112 ... 85, #13 (11)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	113 ... 86, #14 (1)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	114 ... 91, #15 (5)
40685 9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	115 ... ]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	116 >>> revlog = _testrevlog(data, snapshot=range(16))
38640 f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	117
39357 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	118 >>> list(slicechunk(revlog, list(range(16))))
38640 f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	119 [[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]]
39357 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	120 >>> list(slicechunk(revlog, [0, 15]))
38640 f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	121 [[0], [15]]
39357 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	122 >>> list(slicechunk(revlog, [0, 11, 15]))
38640 f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	123 [[0], [11], [15]]
39357 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	124 >>> list(slicechunk(revlog, [0, 11, 13, 15]))
38640 f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	125 [[0], [11, 13, 15]]
39357 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	126 >>> list(slicechunk(revlog, [1, 2, 3, 5, 8, 10, 11, 14]))
38640 f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	127 [[1, 2], [5, 8, 10, 11], [14]]
38643 967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size Boris Feld <boris.feld@octobus.net> parents: 38642 diff changeset	128
967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size Boris Feld <boris.feld@octobus.net> parents: 38642 diff changeset	129 Slicing with a maximum chunk size
39357 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	130 >>> list(slicechunk(revlog, [0, 11, 13, 15], targetsize=15))
38643 967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size Boris Feld <boris.feld@octobus.net> parents: 38642 diff changeset	131 [[0], [11], [13], [15]]
39357 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	132 >>> list(slicechunk(revlog, [0, 11, 13, 15], targetsize=20))
38643 967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size Boris Feld <boris.feld@octobus.net> parents: 38642 diff changeset	133 [[0], [11], [13, 15]]
41082 88d813cd9acd revlog: fix pure python slicing test when chain contains nullrev Boris Feld <boris.feld@octobus.net> parents: 41036 diff changeset	134
88d813cd9acd revlog: fix pure python slicing test when chain contains nullrev Boris Feld <boris.feld@octobus.net> parents: 41036 diff changeset	135 Slicing involving nullrev
88d813cd9acd revlog: fix pure python slicing test when chain contains nullrev Boris Feld <boris.feld@octobus.net> parents: 41036 diff changeset	136 >>> list(slicechunk(revlog, [-1, 0, 11, 13, 15], targetsize=20))
88d813cd9acd revlog: fix pure python slicing test when chain contains nullrev Boris Feld <boris.feld@octobus.net> parents: 41036 diff changeset	137 [[-1, 0], [11], [13, 15]]
88d813cd9acd revlog: fix pure python slicing test when chain contains nullrev Boris Feld <boris.feld@octobus.net> parents: 41036 diff changeset	138 >>> list(slicechunk(revlog, [-1, 13, 15], targetsize=5))
88d813cd9acd revlog: fix pure python slicing test when chain contains nullrev Boris Feld <boris.feld@octobus.net> parents: 41036 diff changeset	139 [[-1], [13], [15]]
34824 e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	140 """
38643 967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size Boris Feld <boris.feld@octobus.net> parents: 38642 diff changeset	141 if targetsize is not None:
967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size Boris Feld <boris.feld@octobus.net> parents: 38642 diff changeset	142 targetsize = max(targetsize, revlog._srmingapsize)
38718 f8762ea73e0d sparse-revlog: implement algorithm to write sparse delta chains (issue5480) Paul Morelle <paul.morelle@octobus.net> parents: 38717 diff changeset	143 # targetsize should not be specified when evaluating delta candidates:
f8762ea73e0d sparse-revlog: implement algorithm to write sparse delta chains (issue5480) Paul Morelle <paul.morelle@octobus.net> parents: 38717 diff changeset	144 # * targetsize is used to ensure we stay within specification when reading,
40748 39d29542fe40 sparse-revlog: put the native implementation of slicechunktodensity to use Boris Feld <boris.feld@octobus.net> parents: 40698 diff changeset	145 densityslicing = getattr(revlog.index, 'slicechunktodensity', None)
39d29542fe40 sparse-revlog: put the native implementation of slicechunktodensity to use Boris Feld <boris.feld@octobus.net> parents: 40698 diff changeset	146 if densityslicing is None:
39d29542fe40 sparse-revlog: put the native implementation of slicechunktodensity to use Boris Feld <boris.feld@octobus.net> parents: 40698 diff changeset	147 densityslicing = lambda x, y, z: _slicechunktodensity(revlog, x, y, z)
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	148 for chunk in densityslicing(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	149 revs, revlog._srdensitythreshold, revlog._srmingapsize
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	150 ):
38643 967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size Boris Feld <boris.feld@octobus.net> parents: 38642 diff changeset	151 for subchunk in _slicechunktosize(revlog, chunk, targetsize):
967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size Boris Feld <boris.feld@octobus.net> parents: 38642 diff changeset	152 yield subchunk
38641 feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	153
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	154
38718 f8762ea73e0d sparse-revlog: implement algorithm to write sparse delta chains (issue5480) Paul Morelle <paul.morelle@octobus.net> parents: 38717 diff changeset	155 def _slicechunktosize(revlog, revs, targetsize=None):
38642 e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	156 """slice revs to match the target size
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	157
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	158 This is intended to be used on chunk that density slicing selected by that
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	159 are still too large compared to the read garantee of revlog. This might
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	160 happens when "minimal gap size" interrupted the slicing or when chain are
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	161 built in a way that create large blocks next to each other.
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	162
40685 9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	163 >>> data = [
38642 e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	164 ... 3, #0 (3)
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	165 ... 5, #1 (2)
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	166 ... 6, #2 (1)
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	167 ... 8, #3 (2)
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	168 ... 8, #4 (empty)
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	169 ... 11, #5 (3)
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	170 ... 12, #6 (1)
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	171 ... 13, #7 (1)
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	172 ... 14, #8 (1)
40685 9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	173 ... ]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	174
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	175 == All snapshots cases ==
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	176 >>> revlog = _testrevlog(data, snapshot=range(9))
38642 e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	177
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	178 Cases where chunk is already small enough
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	179 >>> list(_slicechunktosize(revlog, [0], 3))
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	180 [[0]]
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	181 >>> list(_slicechunktosize(revlog, [6, 7], 3))
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	182 [[6, 7]]
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	183 >>> list(_slicechunktosize(revlog, [0], None))
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	184 [[0]]
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	185 >>> list(_slicechunktosize(revlog, [6, 7], None))
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	186 [[6, 7]]
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	187
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	188 cases where we need actual slicing
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	189 >>> list(_slicechunktosize(revlog, [0, 1], 3))
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	190 [[0], [1]]
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	191 >>> list(_slicechunktosize(revlog, [1, 3], 3))
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	192 [[1], [3]]
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	193 >>> list(_slicechunktosize(revlog, [1, 2, 3], 3))
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	194 [[1, 2], [3]]
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	195 >>> list(_slicechunktosize(revlog, [3, 5], 3))
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	196 [[3], [5]]
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	197 >>> list(_slicechunktosize(revlog, [3, 4, 5], 3))
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	198 [[3], [5]]
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	199 >>> list(_slicechunktosize(revlog, [5, 6, 7, 8], 3))
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	200 [[5], [6, 7, 8]]
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	201 >>> list(_slicechunktosize(revlog, [0, 1, 2, 3, 4, 5, 6, 7, 8], 3))
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	202 [[0], [1, 2], [3], [5], [6, 7, 8]]
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	203
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	204 Case with too large individual chunk (must return valid chunk)
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	205 >>> list(_slicechunktosize(revlog, [0, 1], 2))
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	206 [[0], [1]]
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	207 >>> list(_slicechunktosize(revlog, [1, 3], 1))
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	208 [[1], [3]]
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	209 >>> list(_slicechunktosize(revlog, [3, 4, 5], 2))
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	210 [[3], [5]]
40685 9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	211
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	212 == No Snapshot cases ==
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	213 >>> revlog = _testrevlog(data)
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	214
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	215 Cases where chunk is already small enough
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	216 >>> list(_slicechunktosize(revlog, [0], 3))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	217 [[0]]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	218 >>> list(_slicechunktosize(revlog, [6, 7], 3))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	219 [[6, 7]]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	220 >>> list(_slicechunktosize(revlog, [0], None))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	221 [[0]]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	222 >>> list(_slicechunktosize(revlog, [6, 7], None))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	223 [[6, 7]]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	224
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	225 cases where we need actual slicing
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	226 >>> list(_slicechunktosize(revlog, [0, 1], 3))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	227 [[0], [1]]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	228 >>> list(_slicechunktosize(revlog, [1, 3], 3))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	229 [[1], [3]]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	230 >>> list(_slicechunktosize(revlog, [1, 2, 3], 3))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	231 [[1], [2, 3]]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	232 >>> list(_slicechunktosize(revlog, [3, 5], 3))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	233 [[3], [5]]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	234 >>> list(_slicechunktosize(revlog, [3, 4, 5], 3))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	235 [[3], [4, 5]]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	236 >>> list(_slicechunktosize(revlog, [5, 6, 7, 8], 3))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	237 [[5], [6, 7, 8]]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	238 >>> list(_slicechunktosize(revlog, [0, 1, 2, 3, 4, 5, 6, 7, 8], 3))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	239 [[0], [1, 2], [3], [5], [6, 7, 8]]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	240
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	241 Case with too large individual chunk (must return valid chunk)
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	242 >>> list(_slicechunktosize(revlog, [0, 1], 2))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	243 [[0], [1]]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	244 >>> list(_slicechunktosize(revlog, [1, 3], 1))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	245 [[1], [3]]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	246 >>> list(_slicechunktosize(revlog, [3, 4, 5], 2))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	247 [[3], [5]]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	248
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	249 == mixed case ==
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	250 >>> revlog = _testrevlog(data, snapshot=[0, 1, 2])
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	251 >>> list(_slicechunktosize(revlog, list(range(9)), 5))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	252 [[0, 1], [2], [3, 4, 5], [6, 7, 8]]
38642 e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	253 """
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	254 assert targetsize is None or 0 <= targetsize
40685 9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	255 startdata = revlog.start(revs[0])
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	256 enddata = revlog.end(revs[-1])
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	257 fullspan = enddata - startdata
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	258 if targetsize is None or fullspan <= targetsize:
38642 e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	259 yield revs
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	260 return
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	261
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	262 startrevidx = 0
40698 2eb48aa0acce sparse-revlog: align endrevidx usages in the _slicechunktosize Boris Feld <boris.feld@octobus.net> parents: 40697 diff changeset	263 endrevidx = 1
38642 e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	264 iterrevs = enumerate(revs)
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	265 next(iterrevs) # skip first rev.
40685 9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	266 # first step: get snapshots out of the way
38642 e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	267 for idx, r in iterrevs:
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	268 span = revlog.end(r) - startdata
40685 9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	269 snapshot = revlog.issnapshot(r)
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	270 if span <= targetsize and snapshot:
40698 2eb48aa0acce sparse-revlog: align endrevidx usages in the _slicechunktosize Boris Feld <boris.feld@octobus.net> parents: 40697 diff changeset	271 endrevidx = idx + 1
38642 e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	272 else:
40698 2eb48aa0acce sparse-revlog: align endrevidx usages in the _slicechunktosize Boris Feld <boris.feld@octobus.net> parents: 40697 diff changeset	273 chunk = _trimchunk(revlog, revs, startrevidx, endrevidx)
38642 e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	274 if chunk:
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	275 yield chunk
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	276 startrevidx = idx
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	277 startdata = revlog.start(r)
40698 2eb48aa0acce sparse-revlog: align endrevidx usages in the _slicechunktosize Boris Feld <boris.feld@octobus.net> parents: 40697 diff changeset	278 endrevidx = idx + 1
40685 9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	279 if not snapshot:
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	280 break
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	281
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	282 # for the others, we use binary slicing to quickly converge toward valid
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	283 # chunks (otherwise, we might end up looking for start/end of many
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	284 # revisions). This logic is not looking for the perfect slicing point, it
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	285 # focuses on quickly converging toward valid chunks.
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	286 nbitem = len(revs)
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	287 while (enddata - startdata) > targetsize:
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	288 endrevidx = nbitem
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	289 if nbitem - startrevidx <= 1:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	290 break # protect against individual chunk larger than limit
40685 9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	291 localenddata = revlog.end(revs[endrevidx - 1])
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	292 span = localenddata - startdata
40697 fd1d41ccbe38 sparse-revlog: use `span` variable as intended Boris Feld <boris.feld@octobus.net> parents: 40685 diff changeset	293 while span > targetsize:
40685 9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	294 if endrevidx - startrevidx <= 1:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	295 break # protect against individual chunk larger than limit
40685 9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	296 endrevidx -= (endrevidx - startrevidx) // 2
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	297 localenddata = revlog.end(revs[endrevidx - 1])
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	298 span = localenddata - startdata
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	299 chunk = _trimchunk(revlog, revs, startrevidx, endrevidx)
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	300 if chunk:
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	301 yield chunk
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	302 startrevidx = endrevidx
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	303 startdata = revlog.start(revs[startrevidx])
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	304
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	305 chunk = _trimchunk(revlog, revs, startrevidx)
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	306 if chunk:
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	307 yield chunk
38642 e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	308
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	309
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	310 def _slicechunktodensity(revlog, revs, targetdensity=0.5, mingapsize=0):
38641 feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	311 """slice revs to reduce the amount of unrelated data to be read from disk.
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	312
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	313 ``revs`` is sliced into groups that should be read in one time.
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	314 Assume that revs are sorted.
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	315
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	316 The initial chunk is sliced until the overall density (payload/chunks-span
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	317 ratio) is above `targetdensity`. No gap smaller than `mingapsize` is
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	318 skipped.
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	319
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	320 >>> revlog = _testrevlog([
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	321 ... 5, #00 (5)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	322 ... 10, #01 (5)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	323 ... 12, #02 (2)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	324 ... 12, #03 (empty)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	325 ... 27, #04 (15)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	326 ... 31, #05 (4)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	327 ... 31, #06 (empty)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	328 ... 42, #07 (11)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	329 ... 47, #08 (5)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	330 ... 47, #09 (empty)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	331 ... 48, #10 (1)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	332 ... 51, #11 (3)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	333 ... 74, #12 (23)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	334 ... 85, #13 (11)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	335 ... 86, #14 (1)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	336 ... 91, #15 (5)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	337 ... ])
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	338
38655 cd1c484e31e8 revlog: adjust doctest examples to be portable to Python 3 Augie Fackler <augie@google.com> parents: 38644 diff changeset	339 >>> list(_slicechunktodensity(revlog, list(range(16))))
38641 feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	340 [[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]]
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	341 >>> list(_slicechunktodensity(revlog, [0, 15]))
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	342 [[0], [15]]
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	343 >>> list(_slicechunktodensity(revlog, [0, 11, 15]))
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	344 [[0], [11], [15]]
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	345 >>> list(_slicechunktodensity(revlog, [0, 11, 13, 15]))
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	346 [[0], [11, 13, 15]]
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	347 >>> list(_slicechunktodensity(revlog, [1, 2, 3, 5, 8, 10, 11, 14]))
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	348 [[1, 2], [5, 8, 10, 11], [14]]
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	349 >>> list(_slicechunktodensity(revlog, [1, 2, 3, 5, 8, 10, 11, 14],
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	350 ... mingapsize=20))
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	351 [[1, 2, 3, 5, 8, 10, 11], [14]]
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	352 >>> list(_slicechunktodensity(revlog, [1, 2, 3, 5, 8, 10, 11, 14],
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	353 ... targetdensity=0.95))
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	354 [[1, 2], [5], [8, 10, 11], [14]]
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	355 >>> list(_slicechunktodensity(revlog, [1, 2, 3, 5, 8, 10, 11, 14],
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	356 ... targetdensity=0.95, mingapsize=12))
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	357 [[1, 2], [5, 8, 10, 11], [14]]
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	358 """
34824 e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	359 start = revlog.start
e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	360 length = revlog.length
e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	361
34880 9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	362 if len(revs) <= 1:
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	363 yield revs
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	364 return
34824 e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	365
40651 3ac23dad6364 sparse-revlog: drop unused deltainfo parameter from _slicechunktodensity Boris Feld <boris.feld@octobus.net> parents: 40650 diff changeset	366 deltachainspan = segmentspan(revlog, revs)
34880 9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	367
38641 feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	368 if deltachainspan < mingapsize:
38635 d083ae26c325 revlog: early return in _slicechunk when span is already small enough Boris Feld <boris.feld@octobus.net> parents: 38634 diff changeset	369 yield revs
d083ae26c325 revlog: early return in _slicechunk when span is already small enough Boris Feld <boris.feld@octobus.net> parents: 38634 diff changeset	370 return
d083ae26c325 revlog: early return in _slicechunk when span is already small enough Boris Feld <boris.feld@octobus.net> parents: 38634 diff changeset	371
38718 f8762ea73e0d sparse-revlog: implement algorithm to write sparse delta chains (issue5480) Paul Morelle <paul.morelle@octobus.net> parents: 38717 diff changeset	372 readdata = deltachainspan
40653 bfbfd15d65bd sparse-revlog: fast-path before computing payload size Boris Feld <boris.feld@octobus.net> parents: 40652 diff changeset	373 chainpayload = sum(length(r) for r in revs)
34824 e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	374
34880 9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	375 if deltachainspan:
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	376 density = chainpayload / float(deltachainspan)
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	377 else:
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	378 density = 1.0
34824 e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	379
38641 feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	380 if density >= targetdensity:
38634 f0ea8b847831 revlog: early return in _slicechunk when density is already good Paul Morelle <paul.morelle@octobus.net> parents: 38632 diff changeset	381 yield revs
f0ea8b847831 revlog: early return in _slicechunk when density is already good Paul Morelle <paul.morelle@octobus.net> parents: 38632 diff changeset	382 return
f0ea8b847831 revlog: early return in _slicechunk when density is already good Paul Morelle <paul.morelle@octobus.net> parents: 38632 diff changeset	383
34880 9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	384 # Store the gaps in a heap to have them sorted by decreasing size
40654 54de23400b2a sparse-revlog: stop using a heap to track gaps Boris Feld <boris.feld@octobus.net> parents: 40653 diff changeset	385 gaps = []
34880 9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	386 prevend = None
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	387 for i, rev in enumerate(revs):
40651 3ac23dad6364 sparse-revlog: drop unused deltainfo parameter from _slicechunktodensity Boris Feld <boris.feld@octobus.net> parents: 40650 diff changeset	388 revstart = start(rev)
3ac23dad6364 sparse-revlog: drop unused deltainfo parameter from _slicechunktodensity Boris Feld <boris.feld@octobus.net> parents: 40650 diff changeset	389 revlen = length(rev)
34824 e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	390
34898 1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk Paul Morelle <paul.morelle@octobus.net> parents: 34881 diff changeset	391 # Skip empty revisions to form larger holes
1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk Paul Morelle <paul.morelle@octobus.net> parents: 34881 diff changeset	392 if revlen == 0:
1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk Paul Morelle <paul.morelle@octobus.net> parents: 34881 diff changeset	393 continue
1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk Paul Morelle <paul.morelle@octobus.net> parents: 34881 diff changeset	394
34880 9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	395 if prevend is not None:
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	396 gapsize = revstart - prevend
34881 8c9b08a0c48c sparse-read: skip gaps too small to be worth splitting Paul Morelle <paul.morelle@octobus.net> parents: 34880 diff changeset	397 # only consider holes that are large enough
38641 feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	398 if gapsize > mingapsize:
40654 54de23400b2a sparse-revlog: stop using a heap to track gaps Boris Feld <boris.feld@octobus.net> parents: 40653 diff changeset	399 gaps.append((gapsize, i))
34880 9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	400
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	401 prevend = revstart + revlen
40654 54de23400b2a sparse-revlog: stop using a heap to track gaps Boris Feld <boris.feld@octobus.net> parents: 40653 diff changeset	402 # sort the gaps to pop them from largest to small
54de23400b2a sparse-revlog: stop using a heap to track gaps Boris Feld <boris.feld@octobus.net> parents: 40653 diff changeset	403 gaps.sort()
34880 9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	404
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	405 # Collect the indices of the largest holes until the density is acceptable
40655 526ee887c4d5 sparse-revlog: stop using a heap to track selected gap Boris Feld <boris.feld@octobus.net> parents: 40654 diff changeset	406 selected = []
40654 54de23400b2a sparse-revlog: stop using a heap to track gaps Boris Feld <boris.feld@octobus.net> parents: 40653 diff changeset	407 while gaps and density < targetdensity:
54de23400b2a sparse-revlog: stop using a heap to track gaps Boris Feld <boris.feld@octobus.net> parents: 40653 diff changeset	408 gapsize, gapidx = gaps.pop()
34880 9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	409
40655 526ee887c4d5 sparse-revlog: stop using a heap to track selected gap Boris Feld <boris.feld@octobus.net> parents: 40654 diff changeset	410 selected.append(gapidx)
34824 e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	411
34880 9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	412 # the gap sizes are stored as negatives to be sorted decreasingly
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	413 # by the heap
40654 54de23400b2a sparse-revlog: stop using a heap to track gaps Boris Feld <boris.feld@octobus.net> parents: 40653 diff changeset	414 readdata -= gapsize
34880 9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	415 if readdata > 0:
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	416 density = chainpayload / float(readdata)
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	417 else:
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	418 density = 1.0
40655 526ee887c4d5 sparse-revlog: stop using a heap to track selected gap Boris Feld <boris.feld@octobus.net> parents: 40654 diff changeset	419 selected.sort()
34824 e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	420
34880 9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	421 # Cut the revs at collected indices
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	422 previdx = 0
40655 526ee887c4d5 sparse-revlog: stop using a heap to track selected gap Boris Feld <boris.feld@octobus.net> parents: 40654 diff changeset	423 for idx in selected:
34898 1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk Paul Morelle <paul.morelle@octobus.net> parents: 34881 diff changeset	424
1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk Paul Morelle <paul.morelle@octobus.net> parents: 34881 diff changeset	425 chunk = _trimchunk(revlog, revs, previdx, idx)
1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk Paul Morelle <paul.morelle@octobus.net> parents: 34881 diff changeset	426 if chunk:
1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk Paul Morelle <paul.morelle@octobus.net> parents: 34881 diff changeset	427 yield chunk
1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk Paul Morelle <paul.morelle@octobus.net> parents: 34881 diff changeset	428
34880 9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	429 previdx = idx
34898 1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk Paul Morelle <paul.morelle@octobus.net> parents: 34881 diff changeset	430
1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk Paul Morelle <paul.morelle@octobus.net> parents: 34881 diff changeset	431 chunk = _trimchunk(revlog, revs, previdx)
1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk Paul Morelle <paul.morelle@octobus.net> parents: 34881 diff changeset	432 if chunk:
1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk Paul Morelle <paul.morelle@octobus.net> parents: 34881 diff changeset	433 yield chunk
34824 e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	434
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	435
39357 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	436 def _trimchunk(revlog, revs, startidx, endidx=None):
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	437 """returns revs[startidx:endidx] without empty trailing revs
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	438
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	439 Doctest Setup
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	440 >>> revlog = _testrevlog([
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	441 ... 5, #0
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	442 ... 10, #1
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	443 ... 12, #2
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	444 ... 12, #3 (empty)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	445 ... 17, #4
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	446 ... 21, #5
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	447 ... 21, #6 (empty)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	448 ... ])
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	449
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	450 Contiguous cases:
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	451 >>> _trimchunk(revlog, [0, 1, 2, 3, 4, 5, 6], 0)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	452 [0, 1, 2, 3, 4, 5]
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	453 >>> _trimchunk(revlog, [0, 1, 2, 3, 4, 5, 6], 0, 5)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	454 [0, 1, 2, 3, 4]
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	455 >>> _trimchunk(revlog, [0, 1, 2, 3, 4, 5, 6], 0, 4)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	456 [0, 1, 2]
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	457 >>> _trimchunk(revlog, [0, 1, 2, 3, 4, 5, 6], 2, 4)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	458 [2]
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	459 >>> _trimchunk(revlog, [0, 1, 2, 3, 4, 5, 6], 3)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	460 [3, 4, 5]
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	461 >>> _trimchunk(revlog, [0, 1, 2, 3, 4, 5, 6], 3, 5)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	462 [3, 4]
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	463
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	464 Discontiguous cases:
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	465 >>> _trimchunk(revlog, [1, 3, 5, 6], 0)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	466 [1, 3, 5]
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	467 >>> _trimchunk(revlog, [1, 3, 5, 6], 0, 2)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	468 [1]
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	469 >>> _trimchunk(revlog, [1, 3, 5, 6], 1, 3)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	470 [3, 5]
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	471 >>> _trimchunk(revlog, [1, 3, 5, 6], 1)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	472 [3, 5]
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	473 """
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	474 length = revlog.length
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	475
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	476 if endidx is None:
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	477 endidx = len(revs)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	478
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	479 # If we have a non-emtpy delta candidate, there are nothing to trim
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	480 if revs[endidx - 1] < len(revlog):
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	481 # Trim empty revs at the end, except the very first revision of a chain
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	482 while (
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	483 endidx > 1 and endidx > startidx and length(revs[endidx - 1]) == 0
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	484 ):
39357 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	485 endidx -= 1
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	486
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	487 return revs[startidx:endidx]
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	488
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	489
40652 a32ccd32982b sparse-revlog: drop unused deltainfo parameter from segmentspan Boris Feld <boris.feld@octobus.net> parents: 40651 diff changeset	490 def segmentspan(revlog, revs):
39357 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	491 """Get the byte span of a segment of revisions
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	492
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	493 revs is a sorted array of revision numbers
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	494
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	495 >>> revlog = _testrevlog([
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	496 ... 5, #0
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	497 ... 10, #1
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	498 ... 12, #2
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	499 ... 12, #3 (empty)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	500 ... 17, #4
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	501 ... ])
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	502
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	503 >>> segmentspan(revlog, [0, 1, 2, 3, 4])
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	504 17
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	505 >>> segmentspan(revlog, [0, 4])
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	506 17
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	507 >>> segmentspan(revlog, [3, 4])
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	508 5
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	509 >>> segmentspan(revlog, [1, 2, 3,])
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	510 7
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	511 >>> segmentspan(revlog, [1, 3])
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	512 7
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	513 """
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	514 if not revs:
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	515 return 0
40652 a32ccd32982b sparse-revlog: drop unused deltainfo parameter from segmentspan Boris Feld <boris.feld@octobus.net> parents: 40651 diff changeset	516 end = revlog.end(revs[-1])
39357 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	517 return end - revlog.start(revs[0])
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	518
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	519
39358 fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39357 diff changeset	520 def _textfromdelta(fh, revlog, baserev, delta, p1, p2, flags, expectednode):
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39357 diff changeset	521 """build full text from a (base, delta) pair and other metadata"""
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39357 diff changeset	522 # special case deltas which replace entire base; no need to decode
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39357 diff changeset	523 # base revision. this neatly avoids censored bases, which throw when
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39357 diff changeset	524 # they're decoded.
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	525 hlen = struct.calcsize(b">lll")
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	526 if delta[:hlen] == mdiff.replacediffheader(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	527 revlog.rawsize(baserev), len(delta) - hlen
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	528 ):
39358 fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39357 diff changeset	529 fulltext = delta[hlen:]
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39357 diff changeset	530 else:
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39357 diff changeset	531 # deltabase is rawtext before changed by flag processors, which is
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39357 diff changeset	532 # equivalent to non-raw text
48564 c514936d92b4 revlog: remove deprecated APIs Rapha?l Gom?s <rgomes@octobus.net> parents: 47463 diff changeset	533 basetext = revlog.revision(baserev, _df=fh)
39358 fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39357 diff changeset	534 fulltext = mdiff.patch(basetext, delta)
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39357 diff changeset	535
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39357 diff changeset	536 try:
42994 dff95420480f flagprocessors: make `processflagsraw` a module level function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42884 diff changeset	537 validatehash = flagutil.processflagsraw(revlog, fulltext, flags)
39358 fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39357 diff changeset	538 if validatehash:
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39357 diff changeset	539 revlog.checkhash(fulltext, expectednode, p1=p1, p2=p2)
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39357 diff changeset	540 if flags & REVIDX_ISCENSORED:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	541 raise error.StorageError(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	542 _(b'node %s is not censored') % expectednode
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	543 )
39790 4a2466b2a434 revlog: drop some more error aliases (API) Gregory Szorc <gregory.szorc@gmail.com> parents: 39789 diff changeset	544 except error.CensoredNodeError:
39358 fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39357 diff changeset	545 # must pass the censored index flag to add censored revisions
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39357 diff changeset	546 if not flags & REVIDX_ISCENSORED:
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39357 diff changeset	547 raise
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39357 diff changeset	548 return fulltext
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39357 diff changeset	549
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	550
35638 edc9330acac1 revlog: introduce 'deltainfo' to distinguish from 'delta' Paul Morelle <paul.morelle@octobus.net> parents: 35637 diff changeset	551 @attr.s(slots=True, frozen=True)
49037 642e31cb55f0 py3: use class X: instead of class X(object): Gregory Szorc <gregory.szorc@gmail.com> parents: 48966 diff changeset	552 class _deltainfo:
35638 edc9330acac1 revlog: introduce 'deltainfo' to distinguish from 'delta' Paul Morelle <paul.morelle@octobus.net> parents: 35637 diff changeset	553 distance = attr.ib()
edc9330acac1 revlog: introduce 'deltainfo' to distinguish from 'delta' Paul Morelle <paul.morelle@octobus.net> parents: 35637 diff changeset	554 deltalen = attr.ib()
edc9330acac1 revlog: introduce 'deltainfo' to distinguish from 'delta' Paul Morelle <paul.morelle@octobus.net> parents: 35637 diff changeset	555 data = attr.ib()
edc9330acac1 revlog: introduce 'deltainfo' to distinguish from 'delta' Paul Morelle <paul.morelle@octobus.net> parents: 35637 diff changeset	556 base = attr.ib()
edc9330acac1 revlog: introduce 'deltainfo' to distinguish from 'delta' Paul Morelle <paul.morelle@octobus.net> parents: 35637 diff changeset	557 chainbase = attr.ib()
edc9330acac1 revlog: introduce 'deltainfo' to distinguish from 'delta' Paul Morelle <paul.morelle@octobus.net> parents: 35637 diff changeset	558 chainlen = attr.ib()
edc9330acac1 revlog: introduce 'deltainfo' to distinguish from 'delta' Paul Morelle <paul.morelle@octobus.net> parents: 35637 diff changeset	559 compresseddeltalen = attr.ib()
39187 e0da43e2f71f revlog: compute snapshot depth on delta info Boris Feld <boris.feld@octobus.net> parents: 39185 diff changeset	560 snapshotdepth = attr.ib()
35638 edc9330acac1 revlog: introduce 'deltainfo' to distinguish from 'delta' Paul Morelle <paul.morelle@octobus.net> parents: 35637 diff changeset	561
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	562
47264 b876f0bf7366 revlog: introduce a plain compression mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46819 diff changeset	563 def drop_u_compression(delta):
b876f0bf7366 revlog: introduce a plain compression mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46819 diff changeset	564 """turn into a "u" (no-compression) into no-compression without header
b876f0bf7366 revlog: introduce a plain compression mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46819 diff changeset	565
b876f0bf7366 revlog: introduce a plain compression mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46819 diff changeset	566 This is useful for revlog format that has better compression method.
b876f0bf7366 revlog: introduce a plain compression mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46819 diff changeset	567 """
b876f0bf7366 revlog: introduce a plain compression mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46819 diff changeset	568 assert delta.data[0] == b'u', delta.data[0]
b876f0bf7366 revlog: introduce a plain compression mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46819 diff changeset	569 return _deltainfo(
b876f0bf7366 revlog: introduce a plain compression mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46819 diff changeset	570 delta.distance,
b876f0bf7366 revlog: introduce a plain compression mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46819 diff changeset	571 delta.deltalen - 1,
b876f0bf7366 revlog: introduce a plain compression mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46819 diff changeset	572 (b'', delta.data[1]),
b876f0bf7366 revlog: introduce a plain compression mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46819 diff changeset	573 delta.base,
b876f0bf7366 revlog: introduce a plain compression mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46819 diff changeset	574 delta.chainbase,
b876f0bf7366 revlog: introduce a plain compression mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46819 diff changeset	575 delta.chainlen,
b876f0bf7366 revlog: introduce a plain compression mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46819 diff changeset	576 delta.compresseddeltalen,
b876f0bf7366 revlog: introduce a plain compression mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46819 diff changeset	577 delta.snapshotdepth,
b876f0bf7366 revlog: introduce a plain compression mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46819 diff changeset	578 )
b876f0bf7366 revlog: introduce a plain compression mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46819 diff changeset	579
b876f0bf7366 revlog: introduce a plain compression mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46819 diff changeset	580
49788 5af4a0a73e4c find-delta: rename _isgooddeltainfo Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49772 diff changeset	581 def is_good_delta_info(revlog, deltainfo, revinfo):
39357 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	582 """Returns True if the given delta is good. Good means that it is within
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	583 the disk span, disk size, and chain length bounds that we know to be
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	584 performant."""
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	585 if deltainfo is None:
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	586 return False
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	587
49879 bcae90c53def delta-find: add a delta-reuse policy that blindly accepts incoming deltas Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49795 diff changeset	588 if (
bcae90c53def delta-find: add a delta-reuse policy that blindly accepts incoming deltas Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49795 diff changeset	589 revinfo.cachedelta is not None
bcae90c53def delta-find: add a delta-reuse policy that blindly accepts incoming deltas Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49795 diff changeset	590 and deltainfo.base == revinfo.cachedelta[0]
bcae90c53def delta-find: add a delta-reuse policy that blindly accepts incoming deltas Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49795 diff changeset	591 and revinfo.cachedelta[2] == DELTA_BASE_REUSE_FORCE
bcae90c53def delta-find: add a delta-reuse policy that blindly accepts incoming deltas Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49795 diff changeset	592 ):
bcae90c53def delta-find: add a delta-reuse policy that blindly accepts incoming deltas Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49795 diff changeset	593 return True
bcae90c53def delta-find: add a delta-reuse policy that blindly accepts incoming deltas Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49795 diff changeset	594
39357 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	595 # - 'deltainfo.distance' is the distance from the base revision --
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	596 # bounding it limits the amount of I/O we need to do.
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	597 # - 'deltainfo.compresseddeltalen' is the sum of the total size of
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	598 # deltas we need to apply -- bounding it limits the amount of CPU
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	599 # we consume.
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	600
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	601 textlen = revinfo.textlen
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	602 defaultmax = textlen * 4
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	603 maxdist = revlog._maxdeltachainspan
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	604 if not maxdist:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	605 maxdist = deltainfo.distance # ensure the conditional pass
39357 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	606 maxdist = max(maxdist, defaultmax)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	607
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	608 # Bad delta from read span:
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	609 #
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	610 # If the span of data read is larger than the maximum allowed.
40650 2f7e531ef3e7 sparse-revlog: skip the span check in the sparse-revlog case Boris Feld <boris.feld@octobus.net> parents: 40425 diff changeset	611 #
2f7e531ef3e7 sparse-revlog: skip the span check in the sparse-revlog case Boris Feld <boris.feld@octobus.net> parents: 40425 diff changeset	612 # In the sparse-revlog case, we rely on the associated "sparse reading"
2f7e531ef3e7 sparse-revlog: skip the span check in the sparse-revlog case Boris Feld <boris.feld@octobus.net> parents: 40425 diff changeset	613 # to avoid issue related to the span of data. In theory, it would be
2f7e531ef3e7 sparse-revlog: skip the span check in the sparse-revlog case Boris Feld <boris.feld@octobus.net> parents: 40425 diff changeset	614 # possible to build pathological revlog where delta pattern would lead
2f7e531ef3e7 sparse-revlog: skip the span check in the sparse-revlog case Boris Feld <boris.feld@octobus.net> parents: 40425 diff changeset	615 # to too many reads. However, they do not happen in practice at all. So
2f7e531ef3e7 sparse-revlog: skip the span check in the sparse-revlog case Boris Feld <boris.feld@octobus.net> parents: 40425 diff changeset	616 # we skip the span check entirely.
2f7e531ef3e7 sparse-revlog: skip the span check in the sparse-revlog case Boris Feld <boris.feld@octobus.net> parents: 40425 diff changeset	617 if not revlog._sparserevlog and maxdist < deltainfo.distance:
39357 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	618 return False
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	619
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	620 # Bad delta from new delta size:
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	621 #
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	622 # If the delta size is larger than the target text, storing the
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	623 # delta will be inefficient.
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	624 if textlen < deltainfo.deltalen:
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	625 return False
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	626
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	627 # Bad delta from cumulated payload size:
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	628 #
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	629 # If the sum of delta get larger than K * target text length.
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	630 if textlen * LIMIT_DELTA2TEXT < deltainfo.compresseddeltalen:
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	631 return False
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	632
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	633 # Bad delta from chain length:
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	634 #
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	635 # If the number of delta in the chain gets too high.
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	636 if revlog._maxchainlen and revlog._maxchainlen < deltainfo.chainlen:
39357 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	637 return False
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	638
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	639 # bad delta from intermediate snapshot size limit
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	640 #
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	641 # If an intermediate snapshot size is higher than the limit. The
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	642 # limit exist to prevent endless chain of intermediate delta to be
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	643 # created.
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	644 if (
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	645 deltainfo.snapshotdepth is not None
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	646 and (textlen >> deltainfo.snapshotdepth) < deltainfo.deltalen
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	647 ):
39357 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	648 return False
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	649
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	650 # bad delta if new intermediate snapshot is larger than the previous
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	651 # snapshot
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	652 if (
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	653 deltainfo.snapshotdepth
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	654 and revlog.length(deltainfo.base) < deltainfo.deltalen
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	655 ):
39357 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	656 return False
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	657
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	658 return True
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	659
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	660
40991 42f59d3f714d delta: exclude base candidate much smaller than the target Boris Feld <boris.feld@octobus.net> parents: 40970 diff changeset	661 # If a revision's full text is that much bigger than a base candidate full
42f59d3f714d delta: exclude base candidate much smaller than the target Boris Feld <boris.feld@octobus.net> parents: 40970 diff changeset	662 # text's, it is very unlikely that it will produce a valid delta. We no longer
42f59d3f714d delta: exclude base candidate much smaller than the target Boris Feld <boris.feld@octobus.net> parents: 40970 diff changeset	663 # consider these candidates.
41036 b373477948df revlog: limit base to rev size ratio to 500 instead of 50 Boris Feld <boris.feld@octobus.net> parents: 40992 diff changeset	664 LIMIT_BASE2TEXT = 500
40991 42f59d3f714d delta: exclude base candidate much smaller than the target Boris Feld <boris.feld@octobus.net> parents: 40970 diff changeset	665
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	666
49740 2afee2176775 delta-find: expand a function definition and call before extendin it Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49739 diff changeset	667 def _candidategroups(
2afee2176775 delta-find: expand a function definition and call before extendin it Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49739 diff changeset	668 revlog,
2afee2176775 delta-find: expand a function definition and call before extendin it Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49739 diff changeset	669 textlen,
2afee2176775 delta-find: expand a function definition and call before extendin it Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49739 diff changeset	670 p1,
2afee2176775 delta-find: expand a function definition and call before extendin it Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49739 diff changeset	671 p2,
2afee2176775 delta-find: expand a function definition and call before extendin it Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49739 diff changeset	672 cachedelta,
49741 e706bb41fdb3 delta-find: move pre-filtering with other pre-filtering logic Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49740 diff changeset	673 excluded_bases=None,
e706bb41fdb3 delta-find: move pre-filtering with other pre-filtering logic Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49740 diff changeset	674 target_rev=None,
49795 c261a628e525 delta-find: use a single snapshot cache when applying a group to an object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49794 diff changeset	675 snapshot_cache=None,
49740 2afee2176775 delta-find: expand a function definition and call before extendin it Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49739 diff changeset	676 ):
39363 1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39362 diff changeset	677 """Provides group of revision to be tested as delta base
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39362 diff changeset	678
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39362 diff changeset	679 This top level function focus on emitting groups with unique and worthwhile
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39362 diff changeset	680 content. See _raw_candidate_groups for details about the group order.
39361 507f5b1dd7c8 revlogdeltas: extract _getcandidaterevs in a function Boris Feld <boris.feld@octobus.net> parents: 39360 diff changeset	681 """
39363 1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39362 diff changeset	682 # should we try to build a delta?
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39362 diff changeset	683 if not (len(revlog) and revlog._storedeltachains):
39514 5b308a4e6d03 snapshot: use None as a stop value when looking for a good delta Boris Feld <boris.feld@octobus.net> parents: 39513 diff changeset	684 yield None
39363 1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39362 diff changeset	685 return
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39362 diff changeset	686
39364 37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups Boris Feld <boris.feld@octobus.net> parents: 39363 diff changeset	687 deltalength = revlog.length
37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups Boris Feld <boris.feld@octobus.net> parents: 39363 diff changeset	688 deltaparent = revlog.deltaparent
40991 42f59d3f714d delta: exclude base candidate much smaller than the target Boris Feld <boris.feld@octobus.net> parents: 40970 diff changeset	689 sparse = revlog._sparserevlog
39515 04b75f3a3f2a snapshot: add refining logic at the findeltainfo level Boris Feld <boris.feld@octobus.net> parents: 39514 diff changeset	690 good = None
39364 37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups Boris Feld <boris.feld@octobus.net> parents: 39363 diff changeset	691
37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups Boris Feld <boris.feld@octobus.net> parents: 39363 diff changeset	692 deltas_limit = textlen * LIMIT_DELTA2TEXT
49772 f5f113f1b011 delta-find: add a way to control the number of bases tested at the same time Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49744 diff changeset	693 group_chunk_size = revlog._candidate_group_chunk_size
39364 37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups Boris Feld <boris.feld@octobus.net> parents: 39363 diff changeset	694
42057 566daffc607d cleanup: use set literals where possible Martin von Zweigbergk <martinvonz@google.com> parents: 41824 diff changeset	695 tested = {nullrev}
49739 35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	696 candidates = _refinedgroups(
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	697 revlog,
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	698 p1,
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	699 p2,
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	700 cachedelta,
49795 c261a628e525 delta-find: use a single snapshot cache when applying a group to an object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49794 diff changeset	701 snapshot_cache=snapshot_cache,
49739 35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	702 )
39516 51cec7fb672e snapshot: also use None as a stop value for `_refinegroup` Boris Feld <boris.feld@octobus.net> parents: 39515 diff changeset	703 while True:
39517 cc85ebb68ff9 snapshot: turn _refinedgroups into a coroutine Boris Feld <boris.feld@octobus.net> parents: 39516 diff changeset	704 temptative = candidates.send(good)
39516 51cec7fb672e snapshot: also use None as a stop value for `_refinegroup` Boris Feld <boris.feld@octobus.net> parents: 39515 diff changeset	705 if temptative is None:
51cec7fb672e snapshot: also use None as a stop value for `_refinegroup` Boris Feld <boris.feld@octobus.net> parents: 39515 diff changeset	706 break
39364 37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups Boris Feld <boris.feld@octobus.net> parents: 39363 diff changeset	707 group = []
37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups Boris Feld <boris.feld@octobus.net> parents: 39363 diff changeset	708 for rev in temptative:
37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups Boris Feld <boris.feld@octobus.net> parents: 39363 diff changeset	709 # skip over empty delta (no need to include them in a chain)
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	710 while revlog._generaldelta and not (
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	711 rev == nullrev or rev in tested or deltalength(rev)
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	712 ):
39610 bdb41eaa8b59 snapshot: fix line order when skipping over empty deltas Boris Feld <boris.feld@octobus.net> parents: 39522 diff changeset	713 tested.add(rev)
39364 37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups Boris Feld <boris.feld@octobus.net> parents: 39363 diff changeset	714 rev = deltaparent(rev)
40970 f960c51eebf3 delta: filter nullrev out first Boris Feld <boris.feld@octobus.net> parents: 40748 diff changeset	715 # no need to try a delta against nullrev, this will be done as a
f960c51eebf3 delta: filter nullrev out first Boris Feld <boris.feld@octobus.net> parents: 40748 diff changeset	716 # last resort.
f960c51eebf3 delta: filter nullrev out first Boris Feld <boris.feld@octobus.net> parents: 40748 diff changeset	717 if rev == nullrev:
f960c51eebf3 delta: filter nullrev out first Boris Feld <boris.feld@octobus.net> parents: 40748 diff changeset	718 continue
39364 37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups Boris Feld <boris.feld@octobus.net> parents: 39363 diff changeset	719 # filter out revision we tested already
37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups Boris Feld <boris.feld@octobus.net> parents: 39363 diff changeset	720 if rev in tested:
37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups Boris Feld <boris.feld@octobus.net> parents: 39363 diff changeset	721 continue
49879 bcae90c53def delta-find: add a delta-reuse policy that blindly accepts incoming deltas Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49795 diff changeset	722
bcae90c53def delta-find: add a delta-reuse policy that blindly accepts incoming deltas Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49795 diff changeset	723 if (
bcae90c53def delta-find: add a delta-reuse policy that blindly accepts incoming deltas Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49795 diff changeset	724 cachedelta is not None
bcae90c53def delta-find: add a delta-reuse policy that blindly accepts incoming deltas Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49795 diff changeset	725 and rev == cachedelta[0]
bcae90c53def delta-find: add a delta-reuse policy that blindly accepts incoming deltas Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49795 diff changeset	726 and cachedelta[2] == DELTA_BASE_REUSE_FORCE
bcae90c53def delta-find: add a delta-reuse policy that blindly accepts incoming deltas Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49795 diff changeset	727 ):
bcae90c53def delta-find: add a delta-reuse policy that blindly accepts incoming deltas Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49795 diff changeset	728 # instructions are to forcibly consider/use this delta base
bcae90c53def delta-find: add a delta-reuse policy that blindly accepts incoming deltas Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49795 diff changeset	729 group.append(rev)
bcae90c53def delta-find: add a delta-reuse policy that blindly accepts incoming deltas Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49795 diff changeset	730 continue
bcae90c53def delta-find: add a delta-reuse policy that blindly accepts incoming deltas Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49795 diff changeset	731
49741 e706bb41fdb3 delta-find: move pre-filtering with other pre-filtering logic Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49740 diff changeset	732 # an higher authority deamed the base unworthy (e.g. censored)
e706bb41fdb3 delta-find: move pre-filtering with other pre-filtering logic Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49740 diff changeset	733 if excluded_bases is not None and rev in excluded_bases:
49744 4956942c0416 delta-find: adjust the moment when we mark something as "tested" Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49743 diff changeset	734 tested.add(rev)
49741 e706bb41fdb3 delta-find: move pre-filtering with other pre-filtering logic Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49740 diff changeset	735 continue
e706bb41fdb3 delta-find: move pre-filtering with other pre-filtering logic Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49740 diff changeset	736 # We are in some recomputation cases and that rev is too high in
e706bb41fdb3 delta-find: move pre-filtering with other pre-filtering logic Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49740 diff changeset	737 # the revlog
e706bb41fdb3 delta-find: move pre-filtering with other pre-filtering logic Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49740 diff changeset	738 if target_rev is not None and rev >= target_rev:
49744 4956942c0416 delta-find: adjust the moment when we mark something as "tested" Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49743 diff changeset	739 tested.add(rev)
49741 e706bb41fdb3 delta-find: move pre-filtering with other pre-filtering logic Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49740 diff changeset	740 continue
39364 37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups Boris Feld <boris.feld@octobus.net> parents: 39363 diff changeset	741 # filter out delta base that will never produce good delta
37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups Boris Feld <boris.feld@octobus.net> parents: 39363 diff changeset	742 if deltas_limit < revlog.length(rev):
49744 4956942c0416 delta-find: adjust the moment when we mark something as "tested" Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49743 diff changeset	743 tested.add(rev)
39364 37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups Boris Feld <boris.feld@octobus.net> parents: 39363 diff changeset	744 continue
40991 42f59d3f714d delta: exclude base candidate much smaller than the target Boris Feld <boris.feld@octobus.net> parents: 40970 diff changeset	745 if sparse and revlog.rawsize(rev) < (textlen // LIMIT_BASE2TEXT):
49744 4956942c0416 delta-find: adjust the moment when we mark something as "tested" Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49743 diff changeset	746 tested.add(rev)
40991 42f59d3f714d delta: exclude base candidate much smaller than the target Boris Feld <boris.feld@octobus.net> parents: 40970 diff changeset	747 continue
39364 37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups Boris Feld <boris.feld@octobus.net> parents: 39363 diff changeset	748 # no delta for rawtext-changing revs (see "candelta" for why)
37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups Boris Feld <boris.feld@octobus.net> parents: 39363 diff changeset	749 if revlog.flags(rev) & REVIDX_RAWTEXT_CHANGING_FLAGS:
49744 4956942c0416 delta-find: adjust the moment when we mark something as "tested" Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49743 diff changeset	750 tested.add(rev)
39364 37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups Boris Feld <boris.feld@octobus.net> parents: 39363 diff changeset	751 continue
49739 35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	752
40992 ba09db267cb6 delta: ignore base whose chains already don't match expectations Boris Feld <boris.feld@octobus.net> parents: 40991 diff changeset	753 # If we reach here, we are about to build and test a delta.
ba09db267cb6 delta: ignore base whose chains already don't match expectations Boris Feld <boris.feld@octobus.net> parents: 40991 diff changeset	754 # The delta building process will compute the chaininfo in all
ba09db267cb6 delta: ignore base whose chains already don't match expectations Boris Feld <boris.feld@octobus.net> parents: 40991 diff changeset	755 # case, since that computation is cached, it is fine to access it
ba09db267cb6 delta: ignore base whose chains already don't match expectations Boris Feld <boris.feld@octobus.net> parents: 40991 diff changeset	756 # here too.
ba09db267cb6 delta: ignore base whose chains already don't match expectations Boris Feld <boris.feld@octobus.net> parents: 40991 diff changeset	757 chainlen, chainsize = revlog._chaininfo(rev)
ba09db267cb6 delta: ignore base whose chains already don't match expectations Boris Feld <boris.feld@octobus.net> parents: 40991 diff changeset	758 # if chain will be too long, skip base
ba09db267cb6 delta: ignore base whose chains already don't match expectations Boris Feld <boris.feld@octobus.net> parents: 40991 diff changeset	759 if revlog._maxchainlen and chainlen >= revlog._maxchainlen:
49744 4956942c0416 delta-find: adjust the moment when we mark something as "tested" Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49743 diff changeset	760 tested.add(rev)
40992 ba09db267cb6 delta: ignore base whose chains already don't match expectations Boris Feld <boris.feld@octobus.net> parents: 40991 diff changeset	761 continue
ba09db267cb6 delta: ignore base whose chains already don't match expectations Boris Feld <boris.feld@octobus.net> parents: 40991 diff changeset	762 # if chain already have too much data, skip base
ba09db267cb6 delta: ignore base whose chains already don't match expectations Boris Feld <boris.feld@octobus.net> parents: 40991 diff changeset	763 if deltas_limit < chainsize:
49744 4956942c0416 delta-find: adjust the moment when we mark something as "tested" Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49743 diff changeset	764 tested.add(rev)
40992 ba09db267cb6 delta: ignore base whose chains already don't match expectations Boris Feld <boris.feld@octobus.net> parents: 40991 diff changeset	765 continue
42479 a0b26fc8fbba deltas: skip if projected delta size does not match text size constraint Valentin Gatien-Baron <vgatien-baron@janestreet.com>, Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42057 diff changeset	766 if sparse and revlog.upperboundcomp is not None:
a0b26fc8fbba deltas: skip if projected delta size does not match text size constraint Valentin Gatien-Baron <vgatien-baron@janestreet.com>, Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42057 diff changeset	767 maxcomp = revlog.upperboundcomp
a0b26fc8fbba deltas: skip if projected delta size does not match text size constraint Valentin Gatien-Baron <vgatien-baron@janestreet.com>, Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42057 diff changeset	768 basenotsnap = (p1, p2, nullrev)
a0b26fc8fbba deltas: skip if projected delta size does not match text size constraint Valentin Gatien-Baron <vgatien-baron@janestreet.com>, Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42057 diff changeset	769 if rev not in basenotsnap and revlog.issnapshot(rev):
a0b26fc8fbba deltas: skip if projected delta size does not match text size constraint Valentin Gatien-Baron <vgatien-baron@janestreet.com>, Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42057 diff changeset	770 snapshotdepth = revlog.snapshotdepth(rev)
a0b26fc8fbba deltas: skip if projected delta size does not match text size constraint Valentin Gatien-Baron <vgatien-baron@janestreet.com>, Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42057 diff changeset	771 # If text is significantly larger than the base, we can
a0b26fc8fbba deltas: skip if projected delta size does not match text size constraint Valentin Gatien-Baron <vgatien-baron@janestreet.com>, Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42057 diff changeset	772 # expect the resulting delta to be proportional to the size
a0b26fc8fbba deltas: skip if projected delta size does not match text size constraint Valentin Gatien-Baron <vgatien-baron@janestreet.com>, Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42057 diff changeset	773 # difference
a0b26fc8fbba deltas: skip if projected delta size does not match text size constraint Valentin Gatien-Baron <vgatien-baron@janestreet.com>, Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42057 diff changeset	774 revsize = revlog.rawsize(rev)
a0b26fc8fbba deltas: skip if projected delta size does not match text size constraint Valentin Gatien-Baron <vgatien-baron@janestreet.com>, Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42057 diff changeset	775 rawsizedistance = max(textlen - revsize, 0)
a0b26fc8fbba deltas: skip if projected delta size does not match text size constraint Valentin Gatien-Baron <vgatien-baron@janestreet.com>, Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42057 diff changeset	776 # use an estimate of the compression upper bound.
a0b26fc8fbba deltas: skip if projected delta size does not match text size constraint Valentin Gatien-Baron <vgatien-baron@janestreet.com>, Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42057 diff changeset	777 lowestrealisticdeltalen = rawsizedistance // maxcomp
a0b26fc8fbba deltas: skip if projected delta size does not match text size constraint Valentin Gatien-Baron <vgatien-baron@janestreet.com>, Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42057 diff changeset	778
a0b26fc8fbba deltas: skip if projected delta size does not match text size constraint Valentin Gatien-Baron <vgatien-baron@janestreet.com>, Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42057 diff changeset	779 # check the absolute constraint on the delta size
a0b26fc8fbba deltas: skip if projected delta size does not match text size constraint Valentin Gatien-Baron <vgatien-baron@janestreet.com>, Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42057 diff changeset	780 snapshotlimit = textlen >> snapshotdepth
a0b26fc8fbba deltas: skip if projected delta size does not match text size constraint Valentin Gatien-Baron <vgatien-baron@janestreet.com>, Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42057 diff changeset	781 if snapshotlimit < lowestrealisticdeltalen:
a0b26fc8fbba deltas: skip if projected delta size does not match text size constraint Valentin Gatien-Baron <vgatien-baron@janestreet.com>, Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42057 diff changeset	782 # delta lower bound is larger than accepted upper bound
49744 4956942c0416 delta-find: adjust the moment when we mark something as "tested" Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49743 diff changeset	783 tested.add(rev)
42479 a0b26fc8fbba deltas: skip if projected delta size does not match text size constraint Valentin Gatien-Baron <vgatien-baron@janestreet.com>, Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42057 diff changeset	784 continue
a0b26fc8fbba deltas: skip if projected delta size does not match text size constraint Valentin Gatien-Baron <vgatien-baron@janestreet.com>, Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42057 diff changeset	785
42480 66c27df1be84 deltas: skip if projected delta size is bigger than previous snapshot Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42479 diff changeset	786 # check the relative constraint on the delta size
66c27df1be84 deltas: skip if projected delta size is bigger than previous snapshot Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42479 diff changeset	787 revlength = revlog.length(rev)
66c27df1be84 deltas: skip if projected delta size is bigger than previous snapshot Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42479 diff changeset	788 if revlength < lowestrealisticdeltalen:
66c27df1be84 deltas: skip if projected delta size is bigger than previous snapshot Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42479 diff changeset	789 # delta probable lower bound is larger than target base
49744 4956942c0416 delta-find: adjust the moment when we mark something as "tested" Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49743 diff changeset	790 tested.add(rev)
42480 66c27df1be84 deltas: skip if projected delta size is bigger than previous snapshot Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42479 diff changeset	791 continue
66c27df1be84 deltas: skip if projected delta size is bigger than previous snapshot Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42479 diff changeset	792
39364 37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups Boris Feld <boris.feld@octobus.net> parents: 39363 diff changeset	793 group.append(rev)
39363 1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39362 diff changeset	794 if group:
49772 f5f113f1b011 delta-find: add a way to control the number of bases tested at the same time Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49744 diff changeset	795 # When the size of the candidate group is big, it can result in a
f5f113f1b011 delta-find: add a way to control the number of bases tested at the same time Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49744 diff changeset	796 # quite significant performance impact. To reduce this, we can send
f5f113f1b011 delta-find: add a way to control the number of bases tested at the same time Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49744 diff changeset	797 # them in smaller batches until the new batch does not provide any
f5f113f1b011 delta-find: add a way to control the number of bases tested at the same time Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49744 diff changeset	798 # improvements.
f5f113f1b011 delta-find: add a way to control the number of bases tested at the same time Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49744 diff changeset	799 #
f5f113f1b011 delta-find: add a way to control the number of bases tested at the same time Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49744 diff changeset	800 # This might reduce the overall efficiency of the compression in
f5f113f1b011 delta-find: add a way to control the number of bases tested at the same time Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49744 diff changeset	801 # some corner cases, but that should also prevent very pathological
f5f113f1b011 delta-find: add a way to control the number of bases tested at the same time Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49744 diff changeset	802 # cases from being an issue. (eg. 20 000 candidates).
f5f113f1b011 delta-find: add a way to control the number of bases tested at the same time Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49744 diff changeset	803 #
f5f113f1b011 delta-find: add a way to control the number of bases tested at the same time Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49744 diff changeset	804 # XXX note that the ordering of the group becomes important as it
f5f113f1b011 delta-find: add a way to control the number of bases tested at the same time Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49744 diff changeset	805 # now impacts the final result. The current order is unprocessed
f5f113f1b011 delta-find: add a way to control the number of bases tested at the same time Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49744 diff changeset	806 # and can be improved.
f5f113f1b011 delta-find: add a way to control the number of bases tested at the same time Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49744 diff changeset	807 if group_chunk_size == 0:
f5f113f1b011 delta-find: add a way to control the number of bases tested at the same time Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49744 diff changeset	808 tested.update(group)
f5f113f1b011 delta-find: add a way to control the number of bases tested at the same time Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49744 diff changeset	809 good = yield tuple(group)
f5f113f1b011 delta-find: add a way to control the number of bases tested at the same time Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49744 diff changeset	810 else:
f5f113f1b011 delta-find: add a way to control the number of bases tested at the same time Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49744 diff changeset	811 prev_good = good
f5f113f1b011 delta-find: add a way to control the number of bases tested at the same time Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49744 diff changeset	812 for start in range(0, len(group), group_chunk_size):
f5f113f1b011 delta-find: add a way to control the number of bases tested at the same time Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49744 diff changeset	813 sub_group = group[start : start + group_chunk_size]
f5f113f1b011 delta-find: add a way to control the number of bases tested at the same time Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49744 diff changeset	814 tested.update(sub_group)
f5f113f1b011 delta-find: add a way to control the number of bases tested at the same time Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49744 diff changeset	815 good = yield tuple(sub_group)
f5f113f1b011 delta-find: add a way to control the number of bases tested at the same time Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49744 diff changeset	816 if prev_good == good:
f5f113f1b011 delta-find: add a way to control the number of bases tested at the same time Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49744 diff changeset	817 break
f5f113f1b011 delta-find: add a way to control the number of bases tested at the same time Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49744 diff changeset	818
39514 5b308a4e6d03 snapshot: use None as a stop value when looking for a good delta Boris Feld <boris.feld@octobus.net> parents: 39513 diff changeset	819 yield None
39363 1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39362 diff changeset	820
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	821
49795 c261a628e525 delta-find: use a single snapshot cache when applying a group to an object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49794 diff changeset	822 def _refinedgroups(revlog, p1, p2, cachedelta, snapshot_cache=None):
39513 2f9f7889549b snapshot: introduce an intermediate `_refinedgroups` generator Boris Feld <boris.feld@octobus.net> parents: 39512 diff changeset	823 good = None
39518 993d7e2c8b79 snapshot: make sure we'll never refine delta base from a reused source Boris Feld <boris.feld@octobus.net> parents: 39517 diff changeset	824 # First we try to reuse a the delta contained in the bundle.
993d7e2c8b79 snapshot: make sure we'll never refine delta base from a reused source Boris Feld <boris.feld@octobus.net> parents: 39517 diff changeset	825 # (or from the source revlog)
993d7e2c8b79 snapshot: make sure we'll never refine delta base from a reused source Boris Feld <boris.feld@octobus.net> parents: 39517 diff changeset	826 #
993d7e2c8b79 snapshot: make sure we'll never refine delta base from a reused source Boris Feld <boris.feld@octobus.net> parents: 39517 diff changeset	827 # This logic only applies to general delta repositories and can be disabled
993d7e2c8b79 snapshot: make sure we'll never refine delta base from a reused source Boris Feld <boris.feld@octobus.net> parents: 39517 diff changeset	828 # through configuration. Disabling reuse source delta is useful when
993d7e2c8b79 snapshot: make sure we'll never refine delta base from a reused source Boris Feld <boris.feld@octobus.net> parents: 39517 diff changeset	829 # we want to make sure we recomputed "optimal" deltas.
49739 35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	830 debug_info = None
49791 05db41701ece find-delta: pass the cache-delta usage policy alongside the cache-delta Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49789 diff changeset	831 if cachedelta is not None and cachedelta[2] > DELTA_BASE_REUSE_NO:
39518 993d7e2c8b79 snapshot: make sure we'll never refine delta base from a reused source Boris Feld <boris.feld@octobus.net> parents: 39517 diff changeset	832 # Assume what we received from the server is a good choice
993d7e2c8b79 snapshot: make sure we'll never refine delta base from a reused source Boris Feld <boris.feld@octobus.net> parents: 39517 diff changeset	833 # build delta will reuse the cache
49739 35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	834 if debug_info is not None:
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	835 debug_info['cached-delta.tested'] += 1
39518 993d7e2c8b79 snapshot: make sure we'll never refine delta base from a reused source Boris Feld <boris.feld@octobus.net> parents: 39517 diff changeset	836 good = yield (cachedelta[0],)
993d7e2c8b79 snapshot: make sure we'll never refine delta base from a reused source Boris Feld <boris.feld@octobus.net> parents: 39517 diff changeset	837 if good is not None:
49739 35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	838 if debug_info is not None:
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	839 debug_info['cached-delta.accepted'] += 1
39518 993d7e2c8b79 snapshot: make sure we'll never refine delta base from a reused source Boris Feld <boris.feld@octobus.net> parents: 39517 diff changeset	840 yield None
993d7e2c8b79 snapshot: make sure we'll never refine delta base from a reused source Boris Feld <boris.feld@octobus.net> parents: 39517 diff changeset	841 return
49795 c261a628e525 delta-find: use a single snapshot cache when applying a group to an object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49794 diff changeset	842 if snapshot_cache is None:
c261a628e525 delta-find: use a single snapshot cache when applying a group to an object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49794 diff changeset	843 snapshot_cache = SnapshotCache()
49739 35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	844 groups = _rawgroups(
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	845 revlog,
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	846 p1,
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	847 p2,
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	848 cachedelta,
49792 efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	849 snapshot_cache,
49739 35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	850 )
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	851 for candidates in groups:
39513 2f9f7889549b snapshot: introduce an intermediate `_refinedgroups` generator Boris Feld <boris.feld@octobus.net> parents: 39512 diff changeset	852 good = yield candidates
2f9f7889549b snapshot: introduce an intermediate `_refinedgroups` generator Boris Feld <boris.feld@octobus.net> parents: 39512 diff changeset	853 if good is not None:
2f9f7889549b snapshot: introduce an intermediate `_refinedgroups` generator Boris Feld <boris.feld@octobus.net> parents: 39512 diff changeset	854 break
39519 e4d4361d0bcd snapshot: try to refine new snapshot base down the chain Boris Feld <boris.feld@octobus.net> parents: 39518 diff changeset	855
40415 bafa1c4bb7a8 sparse-revlog: only refine delta candidates in the sparse case (issue6006) Boris Feld <boris.feld@octobus.net> parents: 39793 diff changeset	856 # If sparse revlog is enabled, we can try to refine the available deltas
bafa1c4bb7a8 sparse-revlog: only refine delta candidates in the sparse case (issue6006) Boris Feld <boris.feld@octobus.net> parents: 39793 diff changeset	857 if not revlog._sparserevlog:
bafa1c4bb7a8 sparse-revlog: only refine delta candidates in the sparse case (issue6006) Boris Feld <boris.feld@octobus.net> parents: 39793 diff changeset	858 yield None
bafa1c4bb7a8 sparse-revlog: only refine delta candidates in the sparse case (issue6006) Boris Feld <boris.feld@octobus.net> parents: 39793 diff changeset	859 return
bafa1c4bb7a8 sparse-revlog: only refine delta candidates in the sparse case (issue6006) Boris Feld <boris.feld@octobus.net> parents: 39793 diff changeset	860
39519 e4d4361d0bcd snapshot: try to refine new snapshot base down the chain Boris Feld <boris.feld@octobus.net> parents: 39518 diff changeset	861 # if we have a refinable value, try to refine it
e4d4361d0bcd snapshot: try to refine new snapshot base down the chain Boris Feld <boris.feld@octobus.net> parents: 39518 diff changeset	862 if good is not None and good not in (p1, p2) and revlog.issnapshot(good):
e4d4361d0bcd snapshot: try to refine new snapshot base down the chain Boris Feld <boris.feld@octobus.net> parents: 39518 diff changeset	863 # refine snapshot down
e4d4361d0bcd snapshot: try to refine new snapshot base down the chain Boris Feld <boris.feld@octobus.net> parents: 39518 diff changeset	864 previous = None
e4d4361d0bcd snapshot: try to refine new snapshot base down the chain Boris Feld <boris.feld@octobus.net> parents: 39518 diff changeset	865 while previous != good:
e4d4361d0bcd snapshot: try to refine new snapshot base down the chain Boris Feld <boris.feld@octobus.net> parents: 39518 diff changeset	866 previous = good
e4d4361d0bcd snapshot: try to refine new snapshot base down the chain Boris Feld <boris.feld@octobus.net> parents: 39518 diff changeset	867 base = revlog.deltaparent(good)
e4d4361d0bcd snapshot: try to refine new snapshot base down the chain Boris Feld <boris.feld@octobus.net> parents: 39518 diff changeset	868 if base == nullrev:
e4d4361d0bcd snapshot: try to refine new snapshot base down the chain Boris Feld <boris.feld@octobus.net> parents: 39518 diff changeset	869 break
e4d4361d0bcd snapshot: try to refine new snapshot base down the chain Boris Feld <boris.feld@octobus.net> parents: 39518 diff changeset	870 good = yield (base,)
39520 5aef5afa8654 snapshot: refine candidate snapshot base upward Boris Feld <boris.feld@octobus.net> parents: 39519 diff changeset	871 # refine snapshot up
49792 efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	872 if not snapshot_cache.snapshots:
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	873 snapshot_cache.update(revlog, good + 1)
39520 5aef5afa8654 snapshot: refine candidate snapshot base upward Boris Feld <boris.feld@octobus.net> parents: 39519 diff changeset	874 previous = None
5aef5afa8654 snapshot: refine candidate snapshot base upward Boris Feld <boris.feld@octobus.net> parents: 39519 diff changeset	875 while good != previous:
5aef5afa8654 snapshot: refine candidate snapshot base upward Boris Feld <boris.feld@octobus.net> parents: 39519 diff changeset	876 previous = good
49792 efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	877 children = tuple(sorted(c for c in snapshot_cache.snapshots[good]))
39520 5aef5afa8654 snapshot: refine candidate snapshot base upward Boris Feld <boris.feld@octobus.net> parents: 39519 diff changeset	878 good = yield children
5aef5afa8654 snapshot: refine candidate snapshot base upward Boris Feld <boris.feld@octobus.net> parents: 39519 diff changeset	879
49739 35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	880 if debug_info is not None:
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	881 if good is None:
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	882 debug_info['no-solution'] += 1
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	883
39516 51cec7fb672e snapshot: also use None as a stop value for `_refinegroup` Boris Feld <boris.feld@octobus.net> parents: 39515 diff changeset	884 yield None
39513 2f9f7889549b snapshot: introduce an intermediate `_refinedgroups` generator Boris Feld <boris.feld@octobus.net> parents: 39512 diff changeset	885
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	886
49792 efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	887 def _rawgroups(revlog, p1, p2, cachedelta, snapshot_cache=None):
39363 1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39362 diff changeset	888 """Provides group of revision to be tested as delta base
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39362 diff changeset	889
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39362 diff changeset	890 This lower level function focus on emitting delta theorically interresting
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39362 diff changeset	891 without looking it any practical details.
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39362 diff changeset	892
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39362 diff changeset	893 The group order aims at providing fast or small candidates first.
39361 507f5b1dd7c8 revlogdeltas: extract _getcandidaterevs in a function Boris Feld <boris.feld@octobus.net> parents: 39360 diff changeset	894 """
507f5b1dd7c8 revlogdeltas: extract _getcandidaterevs in a function Boris Feld <boris.feld@octobus.net> parents: 39360 diff changeset	895 gdelta = revlog._generaldelta
41312 189e06b2d719 revlog: make sure we never use sparserevlog without general delta (issue6056) Boris Feld <boris.feld@octobus.net> parents: 41112 diff changeset	896 # gate sparse behind general-delta because of issue6056
189e06b2d719 revlog: make sure we never use sparserevlog without general delta (issue6056) Boris Feld <boris.feld@octobus.net> parents: 41112 diff changeset	897 sparse = gdelta and revlog._sparserevlog
39361 507f5b1dd7c8 revlogdeltas: extract _getcandidaterevs in a function Boris Feld <boris.feld@octobus.net> parents: 39360 diff changeset	898 curr = len(revlog)
507f5b1dd7c8 revlogdeltas: extract _getcandidaterevs in a function Boris Feld <boris.feld@octobus.net> parents: 39360 diff changeset	899 prev = curr - 1
39509 a33f394b2bfd snapshot: try intermediate snapshot against parents' base Boris Feld <boris.feld@octobus.net> parents: 39505 diff changeset	900 deltachain = lambda rev: revlog._deltachain(rev)[0]
39361 507f5b1dd7c8 revlogdeltas: extract _getcandidaterevs in a function Boris Feld <boris.feld@octobus.net> parents: 39360 diff changeset	901
39363 1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39362 diff changeset	902 if gdelta:
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39362 diff changeset	903 # exclude already lazy tested base if any
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39362 diff changeset	904 parents = [p for p in (p1, p2) if p != nullrev]
39361 507f5b1dd7c8 revlogdeltas: extract _getcandidaterevs in a function Boris Feld <boris.feld@octobus.net> parents: 39360 diff changeset	905
39363 1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39362 diff changeset	906 if not revlog._deltabothparents and len(parents) == 2:
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39362 diff changeset	907 parents.sort()
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39362 diff changeset	908 # To minimize the chance of having to build a fulltext,
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39362 diff changeset	909 # pick first whichever parent is closest to us (max rev)
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39362 diff changeset	910 yield (parents[1],)
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39362 diff changeset	911 # then the other one (min rev) if the first did not fit
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39362 diff changeset	912 yield (parents[0],)
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39362 diff changeset	913 elif len(parents) > 0:
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39362 diff changeset	914 # Test all parents (1 or 2), and keep the best candidate
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39362 diff changeset	915 yield parents
39361 507f5b1dd7c8 revlogdeltas: extract _getcandidaterevs in a function Boris Feld <boris.feld@octobus.net> parents: 39360 diff changeset	916
39509 a33f394b2bfd snapshot: try intermediate snapshot against parents' base Boris Feld <boris.feld@octobus.net> parents: 39505 diff changeset	917 if sparse and parents:
49792 efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	918 if snapshot_cache is None:
49742 5447c1507c86 delta-find: small documentation update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49741 diff changeset	919 # map: base-rev: [snapshot-revs]
49792 efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	920 snapshot_cache = SnapshotCache()
39509 a33f394b2bfd snapshot: try intermediate snapshot against parents' base Boris Feld <boris.feld@octobus.net> parents: 39505 diff changeset	921 # See if we can use an existing snapshot in the parent chains to use as
a33f394b2bfd snapshot: try intermediate snapshot against parents' base Boris Feld <boris.feld@octobus.net> parents: 39505 diff changeset	922 # a base for a new intermediate-snapshot
39511 e72130f58f5d snapshot: consider all snapshots in the parents' chains Boris Feld <boris.feld@octobus.net> parents: 39510 diff changeset	923 #
e72130f58f5d snapshot: consider all snapshots in the parents' chains Boris Feld <boris.feld@octobus.net> parents: 39510 diff changeset	924 # search for snapshot in parents delta chain
e72130f58f5d snapshot: consider all snapshots in the parents' chains Boris Feld <boris.feld@octobus.net> parents: 39510 diff changeset	925 # map: snapshot-level: snapshot-rev
e72130f58f5d snapshot: consider all snapshots in the parents' chains Boris Feld <boris.feld@octobus.net> parents: 39510 diff changeset	926 parents_snaps = collections.defaultdict(set)
39521 05a165dc4f55 snapshot: extract parent chain computation Boris Feld <boris.feld@octobus.net> parents: 39520 diff changeset	927 candidate_chains = [deltachain(p) for p in parents]
05a165dc4f55 snapshot: extract parent chain computation Boris Feld <boris.feld@octobus.net> parents: 39520 diff changeset	928 for chain in candidate_chains:
05a165dc4f55 snapshot: extract parent chain computation Boris Feld <boris.feld@octobus.net> parents: 39520 diff changeset	929 for idx, s in enumerate(chain):
39511 e72130f58f5d snapshot: consider all snapshots in the parents' chains Boris Feld <boris.feld@octobus.net> parents: 39510 diff changeset	930 if not revlog.issnapshot(s):
e72130f58f5d snapshot: consider all snapshots in the parents' chains Boris Feld <boris.feld@octobus.net> parents: 39510 diff changeset	931 break
e72130f58f5d snapshot: consider all snapshots in the parents' chains Boris Feld <boris.feld@octobus.net> parents: 39510 diff changeset	932 parents_snaps[idx].add(s)
39512 6a53842727c1 snapshot: consider unrelated snapshots at a similar level first Boris Feld <boris.feld@octobus.net> parents: 39511 diff changeset	933 snapfloor = min(parents_snaps[0]) + 1
49792 efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	934 snapshot_cache.update(revlog, snapfloor)
39522 c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39521 diff changeset	935 # search for the highest "unrelated" revision
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39521 diff changeset	936 #
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39521 diff changeset	937 # Adding snapshots used by "unrelated" revision increase the odd we
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39521 diff changeset	938 # reuse an independant, yet better snapshot chain.
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39521 diff changeset	939 #
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39521 diff changeset	940 # XXX instead of building a set of revisions, we could lazily enumerate
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39521 diff changeset	941 # over the chains. That would be more efficient, however we stick to
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39521 diff changeset	942 # simple code for now.
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39521 diff changeset	943 all_revs = set()
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39521 diff changeset	944 for chain in candidate_chains:
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39521 diff changeset	945 all_revs.update(chain)
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39521 diff changeset	946 other = None
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39521 diff changeset	947 for r in revlog.revs(prev, snapfloor):
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39521 diff changeset	948 if r not in all_revs:
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39521 diff changeset	949 other = r
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39521 diff changeset	950 break
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39521 diff changeset	951 if other is not None:
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39521 diff changeset	952 # To avoid unfair competition, we won't use unrelated intermediate
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39521 diff changeset	953 # snapshot that are deeper than the ones from the parent delta
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39521 diff changeset	954 # chain.
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39521 diff changeset	955 max_depth = max(parents_snaps.keys())
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39521 diff changeset	956 chain = deltachain(other)
49743 01ccb45b7393 delta-find: rename a variable for clarity Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49742 diff changeset	957 for depth, s in enumerate(chain):
39522 c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39521 diff changeset	958 if s < snapfloor:
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39521 diff changeset	959 continue
49743 01ccb45b7393 delta-find: rename a variable for clarity Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49742 diff changeset	960 if max_depth < depth:
39522 c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39521 diff changeset	961 break
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39521 diff changeset	962 if not revlog.issnapshot(s):
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39521 diff changeset	963 break
49743 01ccb45b7393 delta-find: rename a variable for clarity Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49742 diff changeset	964 parents_snaps[depth].add(s)
39511 e72130f58f5d snapshot: consider all snapshots in the parents' chains Boris Feld <boris.feld@octobus.net> parents: 39510 diff changeset	965 # Test them as possible intermediate snapshot base
e72130f58f5d snapshot: consider all snapshots in the parents' chains Boris Feld <boris.feld@octobus.net> parents: 39510 diff changeset	966 # We test them from highest to lowest level. High level one are more
e72130f58f5d snapshot: consider all snapshots in the parents' chains Boris Feld <boris.feld@octobus.net> parents: 39510 diff changeset	967 # likely to result in small delta
39512 6a53842727c1 snapshot: consider unrelated snapshots at a similar level first Boris Feld <boris.feld@octobus.net> parents: 39511 diff changeset	968 floor = None
39511 e72130f58f5d snapshot: consider all snapshots in the parents' chains Boris Feld <boris.feld@octobus.net> parents: 39510 diff changeset	969 for idx, snaps in sorted(parents_snaps.items(), reverse=True):
39512 6a53842727c1 snapshot: consider unrelated snapshots at a similar level first Boris Feld <boris.feld@octobus.net> parents: 39511 diff changeset	970 siblings = set()
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first Boris Feld <boris.feld@octobus.net> parents: 39511 diff changeset	971 for s in snaps:
49792 efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	972 siblings.update(snapshot_cache.snapshots[s])
39512 6a53842727c1 snapshot: consider unrelated snapshots at a similar level first Boris Feld <boris.feld@octobus.net> parents: 39511 diff changeset	973 # Before considering making a new intermediate snapshot, we check
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first Boris Feld <boris.feld@octobus.net> parents: 39511 diff changeset	974 # if an existing snapshot, children of base we consider, would be
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first Boris Feld <boris.feld@octobus.net> parents: 39511 diff changeset	975 # suitable.
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first Boris Feld <boris.feld@octobus.net> parents: 39511 diff changeset	976 #
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first Boris Feld <boris.feld@octobus.net> parents: 39511 diff changeset	977 # It give a change to reuse a delta chain "unrelated" to the
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first Boris Feld <boris.feld@octobus.net> parents: 39511 diff changeset	978 # current revision instead of starting our own. Without such
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first Boris Feld <boris.feld@octobus.net> parents: 39511 diff changeset	979 # re-use, topological branches would keep reopening new chains.
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first Boris Feld <boris.feld@octobus.net> parents: 39511 diff changeset	980 # Creating more and more snapshot as the repository grow.
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first Boris Feld <boris.feld@octobus.net> parents: 39511 diff changeset	981
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first Boris Feld <boris.feld@octobus.net> parents: 39511 diff changeset	982 if floor is not None:
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first Boris Feld <boris.feld@octobus.net> parents: 39511 diff changeset	983 # We only do this for siblings created after the one in our
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first Boris Feld <boris.feld@octobus.net> parents: 39511 diff changeset	984 # parent's delta chain. Those created before has less chances
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first Boris Feld <boris.feld@octobus.net> parents: 39511 diff changeset	985 # to be valid base since our ancestors had to create a new
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first Boris Feld <boris.feld@octobus.net> parents: 39511 diff changeset	986 # snapshot.
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first Boris Feld <boris.feld@octobus.net> parents: 39511 diff changeset	987 siblings = [r for r in siblings if floor < r]
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first Boris Feld <boris.feld@octobus.net> parents: 39511 diff changeset	988 yield tuple(sorted(siblings))
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first Boris Feld <boris.feld@octobus.net> parents: 39511 diff changeset	989 # then test the base from our parent's delta chain.
39511 e72130f58f5d snapshot: consider all snapshots in the parents' chains Boris Feld <boris.feld@octobus.net> parents: 39510 diff changeset	990 yield tuple(sorted(snaps))
39512 6a53842727c1 snapshot: consider unrelated snapshots at a similar level first Boris Feld <boris.feld@octobus.net> parents: 39511 diff changeset	991 floor = min(snaps)
39510 3ca144f1c8dd snapshot: search for unrelated but reusable full-snapshot Boris Feld <boris.feld@octobus.net> parents: 39509 diff changeset	992 # No suitable base found in the parent chain, search if any full
3ca144f1c8dd snapshot: search for unrelated but reusable full-snapshot Boris Feld <boris.feld@octobus.net> parents: 39509 diff changeset	993 # snapshots emitted since parent's base would be a suitable base for an
3ca144f1c8dd snapshot: search for unrelated but reusable full-snapshot Boris Feld <boris.feld@octobus.net> parents: 39509 diff changeset	994 # intermediate snapshot.
3ca144f1c8dd snapshot: search for unrelated but reusable full-snapshot Boris Feld <boris.feld@octobus.net> parents: 39509 diff changeset	995 #
3ca144f1c8dd snapshot: search for unrelated but reusable full-snapshot Boris Feld <boris.feld@octobus.net> parents: 39509 diff changeset	996 # It give a chance to reuse a delta chain unrelated to the current
3ca144f1c8dd snapshot: search for unrelated but reusable full-snapshot Boris Feld <boris.feld@octobus.net> parents: 39509 diff changeset	997 # revisions instead of starting our own. Without such re-use,
3ca144f1c8dd snapshot: search for unrelated but reusable full-snapshot Boris Feld <boris.feld@octobus.net> parents: 39509 diff changeset	998 # topological branches would keep reopening new full chains. Creating
3ca144f1c8dd snapshot: search for unrelated but reusable full-snapshot Boris Feld <boris.feld@octobus.net> parents: 39509 diff changeset	999 # more and more snapshot as the repository grow.
49794 40e24d82b513 delta-find: make sure we only use newer full snapshot as candidate Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49793 diff changeset	1000 full = [r for r in snapshot_cache.snapshots[nullrev] if snapfloor <= r]
40e24d82b513 delta-find: make sure we only use newer full snapshot as candidate Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49793 diff changeset	1001 yield tuple(sorted(full))
39509 a33f394b2bfd snapshot: try intermediate snapshot against parents' base Boris Feld <boris.feld@octobus.net> parents: 39505 diff changeset	1002
39522 c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39521 diff changeset	1003 if not sparse:
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39521 diff changeset	1004 # other approach failed try against prev to hopefully save us a
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39521 diff changeset	1005 # fulltext.
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39521 diff changeset	1006 yield (prev,)
39361 507f5b1dd7c8 revlogdeltas: extract _getcandidaterevs in a function Boris Feld <boris.feld@octobus.net> parents: 39360 diff changeset	1007
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1008
49792 efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1009 class SnapshotCache:
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1010 __slots__ = ('snapshots', '_start_rev', '_end_rev')
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1011
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1012 def __init__(self):
49793 b670eb3dd6c9 delta-find: use sets instead of list in the snapshot cache Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49792 diff changeset	1013 self.snapshots = collections.defaultdict(set)
49792 efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1014 self._start_rev = None
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1015 self._end_rev = None
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1016
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1017 def update(self, revlog, start_rev=0):
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1018 """find snapshots from start_rev to tip"""
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1019 nb_revs = len(revlog)
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1020 end_rev = nb_revs - 1
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1021 if start_rev > end_rev:
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1022 return # range is empty
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1023
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1024 if self._start_rev is None:
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1025 assert self._end_rev is None
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1026 self._update(revlog, start_rev, end_rev)
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1027 elif not (self._start_rev <= start_rev and end_rev <= self._end_rev):
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1028 if start_rev < self._start_rev:
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1029 self._update(revlog, start_rev, self._start_rev - 1)
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1030 if self._end_rev < end_rev:
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1031 self._update(revlog, self._end_rev + 1, end_rev)
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1032
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1033 if self._start_rev is None:
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1034 assert self._end_rev is None
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1035 self._end_rev = end_rev
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1036 self._start_rev = start_rev
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1037 else:
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1038 self._start_rev = min(self._start_rev, start_rev)
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1039 self._end_rev = max(self._end_rev, end_rev)
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1040 assert self._start_rev <= self._end_rev, (
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1041 self._start_rev,
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1042 self._end_rev,
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1043 )
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1044
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1045 def _update(self, revlog, start_rev, end_rev):
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1046 """internal method that actually do update content"""
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1047 assert self._start_rev is None or (
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1048 start_rev < self._start_rev or start_rev > self._end_rev
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1049 ), (self._start_rev, self._end_rev, start_rev, end_rev)
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1050 assert self._start_rev is None or (
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1051 end_rev < self._start_rev or end_rev > self._end_rev
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1052 ), (self._start_rev, self._end_rev, start_rev, end_rev)
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1053 cache = self.snapshots
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1054 if util.safehasattr(revlog.index, b'findsnapshots'):
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1055 revlog.index.findsnapshots(cache, start_rev, end_rev)
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1056 else:
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1057 deltaparent = revlog.deltaparent
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1058 issnapshot = revlog.issnapshot
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1059 for rev in revlog.revs(start_rev, end_rev):
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1060 if issnapshot(rev):
49793 b670eb3dd6c9 delta-find: use sets instead of list in the snapshot cache Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49792 diff changeset	1061 cache[deltaparent(rev)].add(rev)
49792 efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1062
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1063
49037 642e31cb55f0 py3: use class X: instead of class X(object): Gregory Szorc <gregory.szorc@gmail.com> parents: 48966 diff changeset	1064 class deltacomputer:
49739 35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	1065 def __init__(
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	1066 self,
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	1067 revlog,
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	1068 write_debug=None,
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	1069 debug_search=False,
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	1070 debug_info=None,
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	1071 ):
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1072 self.revlog = revlog
49247 e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	1073 self._write_debug = write_debug
49249 b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1074 self._debug_search = debug_search
49739 35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	1075 self._debug_info = debug_info
49795 c261a628e525 delta-find: use a single snapshot cache when applying a group to an object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49794 diff changeset	1076 self._snapshot_cache = SnapshotCache()
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1077
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1078 def buildtext(self, revinfo, fh):
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1079 """Builds a fulltext version of a revision
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1080
47405 34cc102c73f5 revlog: move `revisioninfo` in `revlogutils` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47264 diff changeset	1081 revinfo: revisioninfo instance that contains all needed info
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1082 fh: file handle to either the .i or the .d revlog file,
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1083 depending on whether it is inlined or not
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1084 """
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1085 btext = revinfo.btext
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1086 if btext[0] is not None:
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1087 return btext[0]
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1088
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1089 revlog = self.revlog
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1090 cachedelta = revinfo.cachedelta
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1091 baserev = cachedelta[0]
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1092 delta = cachedelta[1]
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1093
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1094 fulltext = btext[0] = _textfromdelta(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1095 fh,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1096 revlog,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1097 baserev,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1098 delta,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1099 revinfo.p1,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1100 revinfo.p2,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1101 revinfo.flags,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1102 revinfo.node,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1103 )
39358 fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39357 diff changeset	1104 return fulltext
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1105
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1106 def _builddeltadiff(self, base, revinfo, fh):
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1107 revlog = self.revlog
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1108 t = self.buildtext(revinfo, fh)
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1109 if revlog.iscensored(base):
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1110 # deltas based on a censored revision must replace the
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1111 # full content in one patch, so delta works everywhere
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1112 header = mdiff.replacediffheader(revlog.rawsize(base), len(t))
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1113 delta = header + t
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1114 else:
42796 7a89b044eea4 rawdata: update callers in delta utils Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42484 diff changeset	1115 ptext = revlog.rawdata(base, _df=fh)
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1116 delta = mdiff.textdiff(ptext, t)
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1117
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1118 return delta
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1119
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1120 def _builddeltainfo(self, revinfo, base, fh):
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1121 # can we use the cached delta?
42481 6e9ba867a946 delta: move some delta chain related computation earlier in deltainfo Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42480 diff changeset	1122 revlog = self.revlog
49249 b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1123 debug_search = self._write_debug is not None and self._debug_search
42481 6e9ba867a946 delta: move some delta chain related computation earlier in deltainfo Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42480 diff changeset	1124 chainbase = revlog.chainbase(base)
6e9ba867a946 delta: move some delta chain related computation earlier in deltainfo Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42480 diff changeset	1125 if revlog._generaldelta:
6e9ba867a946 delta: move some delta chain related computation earlier in deltainfo Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42480 diff changeset	1126 deltabase = base
6e9ba867a946 delta: move some delta chain related computation earlier in deltainfo Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42480 diff changeset	1127 else:
6e9ba867a946 delta: move some delta chain related computation earlier in deltainfo Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42480 diff changeset	1128 deltabase = chainbase
6e9ba867a946 delta: move some delta chain related computation earlier in deltainfo Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42480 diff changeset	1129 snapshotdepth = None
6e9ba867a946 delta: move some delta chain related computation earlier in deltainfo Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42480 diff changeset	1130 if revlog._sparserevlog and deltabase == nullrev:
6e9ba867a946 delta: move some delta chain related computation earlier in deltainfo Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42480 diff changeset	1131 snapshotdepth = 0
6e9ba867a946 delta: move some delta chain related computation earlier in deltainfo Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42480 diff changeset	1132 elif revlog._sparserevlog and revlog.issnapshot(deltabase):
6e9ba867a946 delta: move some delta chain related computation earlier in deltainfo Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42480 diff changeset	1133 # A delta chain should always be one full snapshot,
6e9ba867a946 delta: move some delta chain related computation earlier in deltainfo Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42480 diff changeset	1134 # zero or more semi-snapshots, and zero or more deltas
6e9ba867a946 delta: move some delta chain related computation earlier in deltainfo Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42480 diff changeset	1135 p1, p2 = revlog.rev(revinfo.p1), revlog.rev(revinfo.p2)
6e9ba867a946 delta: move some delta chain related computation earlier in deltainfo Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42480 diff changeset	1136 if deltabase not in (p1, p2) and revlog.issnapshot(deltabase):
6e9ba867a946 delta: move some delta chain related computation earlier in deltainfo Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42480 diff changeset	1137 snapshotdepth = len(revlog._deltachain(deltabase)[0])
39611 a911932d5003 revlog: reuse cached delta for identical base revision (issue5975) Boris Feld <boris.feld@octobus.net> parents: 39610 diff changeset	1138 delta = None
a911932d5003 revlog: reuse cached delta for identical base revision (issue5975) Boris Feld <boris.feld@octobus.net> parents: 39610 diff changeset	1139 if revinfo.cachedelta:
49789 0fca63953810 find-delta: minor preparatory change Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49788 diff changeset	1140 cachebase = revinfo.cachedelta[0]
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1141 # check if the diff still apply
39611 a911932d5003 revlog: reuse cached delta for identical base revision (issue5975) Boris Feld <boris.feld@octobus.net> parents: 39610 diff changeset	1142 currentbase = cachebase
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1143 while (
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1144 currentbase != nullrev
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1145 and currentbase != base
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1146 and self.revlog.length(currentbase) == 0
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1147 ):
39611 a911932d5003 revlog: reuse cached delta for identical base revision (issue5975) Boris Feld <boris.feld@octobus.net> parents: 39610 diff changeset	1148 currentbase = self.revlog.deltaparent(currentbase)
41824 688fc33e105d storage: introduce a `revlog.reuse-external-delta` config Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41312 diff changeset	1149 if self.revlog._lazydelta and currentbase == base:
39611 a911932d5003 revlog: reuse cached delta for identical base revision (issue5975) Boris Feld <boris.feld@octobus.net> parents: 39610 diff changeset	1150 delta = revinfo.cachedelta[1]
a911932d5003 revlog: reuse cached delta for identical base revision (issue5975) Boris Feld <boris.feld@octobus.net> parents: 39610 diff changeset	1151 if delta is None:
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1152 delta = self._builddeltadiff(base, revinfo, fh)
49249 b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1153 if debug_search:
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1154 msg = b"DBG-DELTAS-SEARCH: uncompressed-delta-size=%d\n"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1155 msg %= len(delta)
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1156 self._write_debug(msg)
42483 c1c1872d25d1 deltas: skip if projected compressed size does not match text size constraint Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42482 diff changeset	1157 # snapshotdept need to be neither None nor 0 level snapshot
c1c1872d25d1 deltas: skip if projected compressed size does not match text size constraint Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42482 diff changeset	1158 if revlog.upperboundcomp is not None and snapshotdepth:
c1c1872d25d1 deltas: skip if projected compressed size does not match text size constraint Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42482 diff changeset	1159 lowestrealisticdeltalen = len(delta) // revlog.upperboundcomp
c1c1872d25d1 deltas: skip if projected compressed size does not match text size constraint Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42482 diff changeset	1160 snapshotlimit = revinfo.textlen >> snapshotdepth
49249 b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1161 if debug_search:
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1162 msg = b"DBG-DELTAS-SEARCH: projected-lower-size=%d\n"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1163 msg %= lowestrealisticdeltalen
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1164 self._write_debug(msg)
42483 c1c1872d25d1 deltas: skip if projected compressed size does not match text size constraint Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42482 diff changeset	1165 if snapshotlimit < lowestrealisticdeltalen:
49249 b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1166 if debug_search:
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1167 msg = b"DBG-DELTAS-SEARCH: DISCARDED (snapshot limit)\n"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1168 self._write_debug(msg)
42483 c1c1872d25d1 deltas: skip if projected compressed size does not match text size constraint Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42482 diff changeset	1169 return None
42484 9b5fbe5ead89 deltas: skip if projected compressed size is bigger than previous snapshot Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42483 diff changeset	1170 if revlog.length(base) < lowestrealisticdeltalen:
49249 b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1171 if debug_search:
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1172 msg = b"DBG-DELTAS-SEARCH: DISCARDED (prev size)\n"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1173 self._write_debug(msg)
42484 9b5fbe5ead89 deltas: skip if projected compressed size is bigger than previous snapshot Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42483 diff changeset	1174 return None
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1175 header, data = revlog.compress(delta)
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1176 deltalen = len(header) + len(data)
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1177 offset = revlog.end(len(revlog) - 1)
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1178 dist = deltalen + offset - revlog.start(chainbase)
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1179 chainlen, compresseddeltalen = revlog._chaininfo(base)
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1180 chainlen += 1
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1181 compresseddeltalen += deltalen
39187 e0da43e2f71f revlog: compute snapshot depth on delta info Boris Feld <boris.feld@octobus.net> parents: 39185 diff changeset	1182
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1183 return _deltainfo(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1184 dist,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1185 deltalen,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1186 (header, data),
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1187 deltabase,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1188 chainbase,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1189 chainlen,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1190 compresseddeltalen,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1191 snapshotdepth,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1192 )
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1193
47462 93f4e183b3f5 deltas: at a `target_rev` parameter to finddeltainfo Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47458 diff changeset	1194 def _fullsnapshotinfo(self, fh, revinfo, curr):
39360 5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39359 diff changeset	1195 rawtext = self.buildtext(revinfo, fh)
5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39359 diff changeset	1196 data = self.revlog.compress(rawtext)
5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39359 diff changeset	1197 compresseddeltalen = deltalen = dist = len(data[1]) + len(data[0])
5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39359 diff changeset	1198 deltabase = chainbase = curr
5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39359 diff changeset	1199 snapshotdepth = 0
5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39359 diff changeset	1200 chainlen = 1
5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39359 diff changeset	1201
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1202 return _deltainfo(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1203 dist,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1204 deltalen,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1205 data,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1206 deltabase,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1207 chainbase,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1208 chainlen,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1209 compresseddeltalen,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1210 snapshotdepth,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1211 )
39360 5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39359 diff changeset	1212
47462 93f4e183b3f5 deltas: at a `target_rev` parameter to finddeltainfo Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47458 diff changeset	1213 def finddeltainfo(self, revinfo, fh, excluded_bases=None, target_rev=None):
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1214 """Find an acceptable delta against a candidate revision
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1215
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1216 revinfo: information about the revision (instance of _revisioninfo)
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1217 fh: file handle to either the .i or the .d revlog file,
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1218 depending on whether it is inlined or not
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1219
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1220 Returns the first acceptable candidate revision, as ordered by
39361 507f5b1dd7c8 revlogdeltas: extract _getcandidaterevs in a function Boris Feld <boris.feld@octobus.net> parents: 39360 diff changeset	1221 _candidategroups
39360 5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39359 diff changeset	1222
5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39359 diff changeset	1223 If no suitable deltabase is found, we return delta info for a full
5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39359 diff changeset	1224 snapshot.
47407 1efe3cdef53a revlog: add a ways to blacklist some revision when searching for a delta Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47405 diff changeset	1225
1efe3cdef53a revlog: add a ways to blacklist some revision when searching for a delta Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47405 diff changeset	1226 `excluded_bases` is an optional set of revision that cannot be used as
1efe3cdef53a revlog: add a ways to blacklist some revision when searching for a delta Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47405 diff changeset	1227 a delta base. Use this to recompute delta suitable in censor or strip
1efe3cdef53a revlog: add a ways to blacklist some revision when searching for a delta Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47405 diff changeset	1228 context.
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1229 """
47462 93f4e183b3f5 deltas: at a `target_rev` parameter to finddeltainfo Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47458 diff changeset	1230 if target_rev is None:
47463 f8330a3fc39f censor: implement censoring for revlogv2 Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47462 diff changeset	1231 target_rev = len(self.revlog)
47462 93f4e183b3f5 deltas: at a `target_rev` parameter to finddeltainfo Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47458 diff changeset	1232
39122 dbb3e9e44fce revlog: do not search for delta for empty content Boris Feld <boris.feld@octobus.net> parents: 39121 diff changeset	1233 if not revinfo.textlen:
47462 93f4e183b3f5 deltas: at a `target_rev` parameter to finddeltainfo Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47458 diff changeset	1234 return self._fullsnapshotinfo(fh, revinfo, target_rev)
39122 dbb3e9e44fce revlog: do not search for delta for empty content Boris Feld <boris.feld@octobus.net> parents: 39121 diff changeset	1235
47407 1efe3cdef53a revlog: add a ways to blacklist some revision when searching for a delta Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47405 diff changeset	1236 if excluded_bases is None:
1efe3cdef53a revlog: add a ways to blacklist some revision when searching for a delta Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47405 diff changeset	1237 excluded_bases = set()
1efe3cdef53a revlog: add a ways to blacklist some revision when searching for a delta Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47405 diff changeset	1238
39359 6f4b8f607a31 revlogdeltas: move special cases around raw revisions in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39358 diff changeset	1239 # no delta for flag processor revision (see "candelta" for why)
6f4b8f607a31 revlogdeltas: move special cases around raw revisions in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39358 diff changeset	1240 # not calling candelta since only one revision needs test, also to
6f4b8f607a31 revlogdeltas: move special cases around raw revisions in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39358 diff changeset	1241 # avoid overhead fetching flags again.
6f4b8f607a31 revlogdeltas: move special cases around raw revisions in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39358 diff changeset	1242 if revinfo.flags & REVIDX_RAWTEXT_CHANGING_FLAGS:
47462 93f4e183b3f5 deltas: at a `target_rev` parameter to finddeltainfo Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47458 diff changeset	1243 return self._fullsnapshotinfo(fh, revinfo, target_rev)
39359 6f4b8f607a31 revlogdeltas: move special cases around raw revisions in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39358 diff changeset	1244
49739 35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	1245 gather_debug = (
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	1246 self._write_debug is not None or self._debug_info is not None
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	1247 )
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	1248 debug_search = self._write_debug is not None and self._debug_search
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	1249
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	1250 if gather_debug:
49247 e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	1251 start = util.timer()
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	1252
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	1253 # count the number of different delta we tried (for debug purpose)
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	1254 dbg_try_count = 0
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	1255 # count the number of "search round" we did. (for debug purpose)
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	1256 dbg_try_rounds = 0
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	1257 dbg_type = b'unknown'
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	1258
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1259 cachedelta = revinfo.cachedelta
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1260 p1 = revinfo.p1
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1261 p2 = revinfo.p2
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1262 revlog = self.revlog
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1263
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1264 deltainfo = None
39362 1441eb38849f revlogdeltas: pass revision number to _candidatesgroups Boris Feld <boris.feld@octobus.net> parents: 39361 diff changeset	1265 p1r, p2r = revlog.rev(p1), revlog.rev(p2)
49247 e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	1266
49739 35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	1267 if gather_debug:
49247 e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	1268 if p1r != nullrev:
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	1269 p1_chain_len = revlog._chaininfo(p1r)[0]
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	1270 else:
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	1271 p1_chain_len = -1
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	1272 if p2r != nullrev:
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	1273 p2_chain_len = revlog._chaininfo(p2r)[0]
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	1274 else:
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	1275 p2_chain_len = -1
49249 b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1276 if debug_search:
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1277 msg = b"DBG-DELTAS-SEARCH: SEARCH rev=%d\n"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1278 msg %= target_rev
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1279 self._write_debug(msg)
49247 e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	1280
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1281 groups = _candidategroups(
49740 2afee2176775 delta-find: expand a function definition and call before extendin it Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49739 diff changeset	1282 self.revlog,
2afee2176775 delta-find: expand a function definition and call before extendin it Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49739 diff changeset	1283 revinfo.textlen,
2afee2176775 delta-find: expand a function definition and call before extendin it Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49739 diff changeset	1284 p1r,
2afee2176775 delta-find: expand a function definition and call before extendin it Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49739 diff changeset	1285 p2r,
2afee2176775 delta-find: expand a function definition and call before extendin it Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49739 diff changeset	1286 cachedelta,
49741 e706bb41fdb3 delta-find: move pre-filtering with other pre-filtering logic Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49740 diff changeset	1287 excluded_bases,
e706bb41fdb3 delta-find: move pre-filtering with other pre-filtering logic Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49740 diff changeset	1288 target_rev,
49795 c261a628e525 delta-find: use a single snapshot cache when applying a group to an object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49794 diff changeset	1289 snapshot_cache=self._snapshot_cache,
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1290 )
39514 5b308a4e6d03 snapshot: use None as a stop value when looking for a good delta Boris Feld <boris.feld@octobus.net> parents: 39513 diff changeset	1291 candidaterevs = next(groups)
5b308a4e6d03 snapshot: use None as a stop value when looking for a good delta Boris Feld <boris.feld@octobus.net> parents: 39513 diff changeset	1292 while candidaterevs is not None:
49247 e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	1293 dbg_try_rounds += 1
49249 b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1294 if debug_search:
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1295 prev = None
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1296 if deltainfo is not None:
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1297 prev = deltainfo.base
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1298
49737 78ba41878f2e delta-find: add debug information about reuse of cached data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49249 diff changeset	1299 if (
78ba41878f2e delta-find: add debug information about reuse of cached data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49249 diff changeset	1300 cachedelta is not None
78ba41878f2e delta-find: add debug information about reuse of cached data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49249 diff changeset	1301 and len(candidaterevs) == 1
78ba41878f2e delta-find: add debug information about reuse of cached data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49249 diff changeset	1302 and cachedelta[0] in candidaterevs
78ba41878f2e delta-find: add debug information about reuse of cached data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49249 diff changeset	1303 ):
78ba41878f2e delta-find: add debug information about reuse of cached data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49249 diff changeset	1304 round_type = b"cached-delta"
78ba41878f2e delta-find: add debug information about reuse of cached data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49249 diff changeset	1305 elif p1 in candidaterevs or p2 in candidaterevs:
49249 b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1306 round_type = b"parents"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1307 elif prev is not None and all(c < prev for c in candidaterevs):
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1308 round_type = b"refine-down"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1309 elif prev is not None and all(c > prev for c in candidaterevs):
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1310 round_type = b"refine-up"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1311 else:
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1312 round_type = b"search-down"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1313 msg = b"DBG-DELTAS-SEARCH: ROUND #%d - %d candidates - %s\n"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1314 msg %= (dbg_try_rounds, len(candidaterevs), round_type)
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1315 self._write_debug(msg)
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1316 nominateddeltas = []
39515 04b75f3a3f2a snapshot: add refining logic at the findeltainfo level Boris Feld <boris.feld@octobus.net> parents: 39514 diff changeset	1317 if deltainfo is not None:
49249 b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1318 if debug_search:
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1319 msg = (
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1320 b"DBG-DELTAS-SEARCH: CONTENDER: rev=%d - length=%d\n"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1321 )
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1322 msg %= (deltainfo.base, deltainfo.deltalen)
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1323 self._write_debug(msg)
39515 04b75f3a3f2a snapshot: add refining logic at the findeltainfo level Boris Feld <boris.feld@octobus.net> parents: 39514 diff changeset	1324 # if we already found a good delta,
04b75f3a3f2a snapshot: add refining logic at the findeltainfo level Boris Feld <boris.feld@octobus.net> parents: 39514 diff changeset	1325 # challenge it against refined candidates
04b75f3a3f2a snapshot: add refining logic at the findeltainfo level Boris Feld <boris.feld@octobus.net> parents: 39514 diff changeset	1326 nominateddeltas.append(deltainfo)
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1327 for candidaterev in candidaterevs:
49249 b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1328 if debug_search:
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1329 msg = b"DBG-DELTAS-SEARCH: CANDIDATE: rev=%d\n"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1330 msg %= candidaterev
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1331 self._write_debug(msg)
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1332 candidate_type = None
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1333 if candidaterev == p1:
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1334 candidate_type = b"p1"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1335 elif candidaterev == p2:
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1336 candidate_type = b"p2"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1337 elif self.revlog.issnapshot(candidaterev):
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1338 candidate_type = b"snapshot-%d"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1339 candidate_type %= self.revlog.snapshotdepth(
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1340 candidaterev
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1341 )
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1342
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1343 if candidate_type is not None:
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1344 msg = b"DBG-DELTAS-SEARCH: type=%s\n"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1345 msg %= candidate_type
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1346 self._write_debug(msg)
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1347 msg = b"DBG-DELTAS-SEARCH: size=%d\n"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1348 msg %= self.revlog.length(candidaterev)
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1349 self._write_debug(msg)
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1350 msg = b"DBG-DELTAS-SEARCH: base=%d\n"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1351 msg %= self.revlog.deltaparent(candidaterev)
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1352 self._write_debug(msg)
49741 e706bb41fdb3 delta-find: move pre-filtering with other pre-filtering logic Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49740 diff changeset	1353
49247 e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	1354 dbg_try_count += 1
49249 b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1355
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1356 if debug_search:
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1357 delta_start = util.timer()
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1358 candidatedelta = self._builddeltainfo(revinfo, candidaterev, fh)
49249 b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1359 if debug_search:
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1360 delta_end = util.timer()
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1361 msg = b"DBG-DELTAS-SEARCH: delta-search-time=%f\n"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1362 msg %= delta_end - delta_start
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1363 self._write_debug(msg)
42482 465f2d0df9ae deltas: accept and skip None return for delta info Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42481 diff changeset	1364 if candidatedelta is not None:
49788 5af4a0a73e4c find-delta: rename _isgooddeltainfo Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49772 diff changeset	1365 if is_good_delta_info(self.revlog, candidatedelta, revinfo):
49249 b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1366 if debug_search:
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1367 msg = b"DBG-DELTAS-SEARCH: DELTA: length=%d (GOOD)\n"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1368 msg %= candidatedelta.deltalen
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1369 self._write_debug(msg)
42482 465f2d0df9ae deltas: accept and skip None return for delta info Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42481 diff changeset	1370 nominateddeltas.append(candidatedelta)
49249 b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1371 elif debug_search:
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1372 msg = b"DBG-DELTAS-SEARCH: DELTA: length=%d (BAD)\n"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1373 msg %= candidatedelta.deltalen
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1374 self._write_debug(msg)
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1375 elif debug_search:
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1376 msg = b"DBG-DELTAS-SEARCH: NO-DELTA\n"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1377 self._write_debug(msg)
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1378 if nominateddeltas:
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1379 deltainfo = min(nominateddeltas, key=lambda x: x.deltalen)
39515 04b75f3a3f2a snapshot: add refining logic at the findeltainfo level Boris Feld <boris.feld@octobus.net> parents: 39514 diff changeset	1380 if deltainfo is not None:
04b75f3a3f2a snapshot: add refining logic at the findeltainfo level Boris Feld <boris.feld@octobus.net> parents: 39514 diff changeset	1381 candidaterevs = groups.send(deltainfo.base)
04b75f3a3f2a snapshot: add refining logic at the findeltainfo level Boris Feld <boris.feld@octobus.net> parents: 39514 diff changeset	1382 else:
04b75f3a3f2a snapshot: add refining logic at the findeltainfo level Boris Feld <boris.feld@octobus.net> parents: 39514 diff changeset	1383 candidaterevs = next(groups)
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1384
39360 5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39359 diff changeset	1385 if deltainfo is None:
49247 e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	1386 dbg_type = b"full"
47462 93f4e183b3f5 deltas: at a `target_rev` parameter to finddeltainfo Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47458 diff changeset	1387 deltainfo = self._fullsnapshotinfo(fh, revinfo, target_rev)
49249 b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1388 elif deltainfo.snapshotdepth: # pytype: disable=attribute-error
49247 e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	1389 dbg_type = b"snapshot"
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	1390 else:
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	1391 dbg_type = b"delta"
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	1392
49739 35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	1393 if gather_debug:
49247 e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	1394 end = util.timer()
49885 d57b966cdeb1 delta-find: properly report full snapshot used from cache as such Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49879 diff changeset	1395 if dbg_type == b'full':
d57b966cdeb1 delta-find: properly report full snapshot used from cache as such Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49879 diff changeset	1396 used_cached = (
d57b966cdeb1 delta-find: properly report full snapshot used from cache as such Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49879 diff changeset	1397 cachedelta is not None
d57b966cdeb1 delta-find: properly report full snapshot used from cache as such Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49879 diff changeset	1398 and dbg_try_rounds == 0
d57b966cdeb1 delta-find: properly report full snapshot used from cache as such Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49879 diff changeset	1399 and dbg_try_count == 0
d57b966cdeb1 delta-find: properly report full snapshot used from cache as such Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49879 diff changeset	1400 and cachedelta[0] == nullrev
d57b966cdeb1 delta-find: properly report full snapshot used from cache as such Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49879 diff changeset	1401 )
d57b966cdeb1 delta-find: properly report full snapshot used from cache as such Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49879 diff changeset	1402 else:
d57b966cdeb1 delta-find: properly report full snapshot used from cache as such Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49879 diff changeset	1403 used_cached = (
d57b966cdeb1 delta-find: properly report full snapshot used from cache as such Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49879 diff changeset	1404 cachedelta is not None
d57b966cdeb1 delta-find: properly report full snapshot used from cache as such Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49879 diff changeset	1405 and dbg_try_rounds == 1
d57b966cdeb1 delta-find: properly report full snapshot used from cache as such Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49879 diff changeset	1406 and dbg_try_count == 1
d57b966cdeb1 delta-find: properly report full snapshot used from cache as such Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49879 diff changeset	1407 and deltainfo.base == cachedelta[0]
d57b966cdeb1 delta-find: properly report full snapshot used from cache as such Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49879 diff changeset	1408 )
49247 e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	1409 dbg = {
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	1410 'duration': end - start,
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	1411 'revision': target_rev,
49739 35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	1412 'delta-base': deltainfo.base, # pytype: disable=attribute-error
49247 e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	1413 'search_round_count': dbg_try_rounds,
49737 78ba41878f2e delta-find: add debug information about reuse of cached data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49249 diff changeset	1414 'using-cached-base': used_cached,
49247 e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	1415 'delta_try_count': dbg_try_count,
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	1416 'type': dbg_type,
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	1417 'p1-chain-len': p1_chain_len,
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	1418 'p2-chain-len': p2_chain_len,
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	1419 }
49249 b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1420 if (
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1421 deltainfo.snapshotdepth # pytype: disable=attribute-error
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1422 is not None
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1423 ):
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1424 dbg[
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1425 'snapshot-depth'
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1426 ] = deltainfo.snapshotdepth # pytype: disable=attribute-error
49247 e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	1427 else:
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	1428 dbg['snapshot-depth'] = 0
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	1429 target_revlog = b"UNKNOWN"
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	1430 target_type = self.revlog.target[0]
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	1431 target_key = self.revlog.target[1]
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	1432 if target_type == KIND_CHANGELOG:
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	1433 target_revlog = b'CHANGELOG:'
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	1434 elif target_type == KIND_MANIFESTLOG:
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	1435 target_revlog = b'MANIFESTLOG:'
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	1436 if target_key:
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	1437 target_revlog += b'%s:' % target_key
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	1438 elif target_type == KIND_FILELOG:
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	1439 target_revlog = b'FILELOG:'
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	1440 if target_key:
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	1441 target_revlog += b'%s:' % target_key
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	1442 dbg['target-revlog'] = target_revlog
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	1443
49739 35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	1444 if self._debug_info is not None:
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	1445 self._debug_info.append(dbg)
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	1446
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	1447 if self._write_debug is not None:
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	1448 msg = (
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	1449 b"DBG-DELTAS:"
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	1450 b" %-12s"
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	1451 b" rev=%d:"
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	1452 b" delta-base=%d"
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	1453 b" is-cached=%d"
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	1454 b" - search-rounds=%d"
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	1455 b" try-count=%d"
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	1456 b" - delta-type=%-6s"
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	1457 b" snap-depth=%d"
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	1458 b" - p1-chain-length=%d"
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	1459 b" p2-chain-length=%d"
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	1460 b" - duration=%f"
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	1461 b"\n"
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	1462 )
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	1463 msg %= (
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	1464 dbg["target-revlog"],
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	1465 dbg["revision"],
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	1466 dbg["delta-base"],
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	1467 dbg["using-cached-base"],
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	1468 dbg["search_round_count"],
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	1469 dbg["delta_try_count"],
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	1470 dbg["type"],
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	1471 dbg["snapshot-depth"],
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	1472 dbg["p1-chain-len"],
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	1473 dbg["p2-chain-len"],
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	1474 dbg["duration"],
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	1475 )
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	1476 self._write_debug(msg)
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1477 return deltainfo
47458 c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47407 diff changeset	1478
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47407 diff changeset	1479
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47407 diff changeset	1480 def delta_compression(default_compression_header, deltainfo):
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47407 diff changeset	1481 """return (COMPRESSION_MODE, deltainfo)
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47407 diff changeset	1482
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47407 diff changeset	1483 used by revlog v2+ format to dispatch between PLAIN and DEFAULT
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47407 diff changeset	1484 compression.
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47407 diff changeset	1485 """
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47407 diff changeset	1486 h, d = deltainfo.data
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47407 diff changeset	1487 compression_mode = COMP_MODE_INLINE
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47407 diff changeset	1488 if not h and not d:
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47407 diff changeset	1489 # not data to store at all... declare them uncompressed
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47407 diff changeset	1490 compression_mode = COMP_MODE_PLAIN
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47407 diff changeset	1491 elif not h:
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47407 diff changeset	1492 t = d[0:1]
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47407 diff changeset	1493 if t == b'\0':
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47407 diff changeset	1494 compression_mode = COMP_MODE_PLAIN
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47407 diff changeset	1495 elif t == default_compression_header:
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47407 diff changeset	1496 compression_mode = COMP_MODE_DEFAULT
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47407 diff changeset	1497 elif h == b'u':
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47407 diff changeset	1498 # we have a more efficient way to declare uncompressed
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47407 diff changeset	1499 h = b''
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47407 diff changeset	1500 compression_mode = COMP_MODE_PLAIN
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47407 diff changeset	1501 deltainfo = drop_u_compression(deltainfo)
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47407 diff changeset	1502 return compression_mode, deltainfo

Mercurial > public > mercurial-scm > hg-stable

annotate mercurial/revlogutils/deltas.py @ 49885:d57b966cdeb1