mercurial-scm/hg-stable: mercurial/revlogutils/deltas.py annotate

annotate mercurial/revlogutils/deltas.py @ 52669:e627cc25b6f3

pyupgrade: rewrite `yield` statements in a loop to `yield from` This is the `legacy` fixer in `pyupgrade`, with the `yield` statement yielding loop commented back in. This seems to help pytype in some cases, and hurt it in others. But that can be manually fixed later. Note that it's possibly buggy in that it aggressively changed `import-checker.py` to `yield from 'fcntl', 'grp', 'pwd', 'select', 'termios': # Unix only`, which is invalid syntax. Possibly it needed help from the token fixer that I've disabled locally (because that wants to make a bunch of unrelated changes). Just change those few places to yield from a list, to avoid having to constantly revert that.

author	Matt Harbison <matt_harbison@yahoo.com>
date	Sun, 05 Jan 2025 22:26:16 -0500
parents	f4733654f144
children

rev	line source
39357 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	1 # revlogdeltas.py - Logic around delta computation for revlog
8226 8b2cd04a6e97 put license and copyright info into comment blocks Martin Geisler <mg@lazybytes.net> parents: 8225 diff changeset	2 #
46819 d4ba4d51f85f contributor: change mentions of mpm to olivia Rapha?l Gom?s <rgomes@octobus.net> parents: 43089 diff changeset	3 # Copyright 2005-2007 Olivia Mackall <olivia@selenic.com>
39357 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	4 # Copyright 2018 Octobus <contact@octobus.net>
8226 8b2cd04a6e97 put license and copyright info into comment blocks Martin Geisler <mg@lazybytes.net> parents: 8225 diff changeset	5 #
8b2cd04a6e97 put license and copyright info into comment blocks Martin Geisler <mg@lazybytes.net> parents: 8225 diff changeset	6 # This software may be used and distributed according to the terms of the
10263 25e572394f5c Update license to GPLv2+ Matt Mackall <mpm@selenic.com> parents: 10047 diff changeset	7 # GNU General Public License version 2 or any later version.
39357 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	8 """Helper class to compute deltas stored inside revlogs"""
8227 0a9542703300 turn some comments back into module docstrings Martin Geisler <mg@lazybytes.net> parents: 8226 diff changeset	9
51901 f4733654f144 typing: add `from __future__ import annotations` to most files Matt Harbison <matt_harbison@yahoo.com> parents: 51788 diff changeset	10 from __future__ import annotations
27361 29f50344fa83 revlog: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27251 diff changeset	11
51356 2a333d791ecf delta-find: introduce a base class for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51355 diff changeset	12 import abc
39510 3ca144f1c8dd snapshot: search for unrelated but reusable full-snapshot Boris Feld <boris.feld@octobus.net> parents: 39509 diff changeset	13 import collections
27361 29f50344fa83 revlog: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27251 diff changeset	14 import struct
51788 278af66e6595 typing: induce pytype to use the standard `attr` instead of the vendored copy Matt Harbison <matt_harbison@yahoo.com> parents: 51762 diff changeset	15 import typing
27361 29f50344fa83 revlog: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27251 diff changeset	16
29f50344fa83 revlog: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27251 diff changeset	17 # import stuff from node for others to import from revlog
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	18 from ..node import nullrev
39357 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	19 from ..i18n import _
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	20
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	21 from .constants import (
47458 c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47407 diff changeset	22 COMP_MODE_DEFAULT,
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47407 diff changeset	23 COMP_MODE_INLINE,
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47407 diff changeset	24 COMP_MODE_PLAIN,
49879 bcae90c53def delta-find: add a delta-reuse policy that blindly accepts incoming deltas Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49795 diff changeset	25 DELTA_BASE_REUSE_FORCE,
49791 05db41701ece find-delta: pass the cache-delta usage policy alongside the cache-delta Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49789 diff changeset	26 DELTA_BASE_REUSE_NO,
49247 e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	27 KIND_CHANGELOG,
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	28 KIND_FILELOG,
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	29 KIND_MANIFESTLOG,
39356 729082bb9938 revlog: split constants into a new `revlogutils.constants` module Boris Feld <boris.feld@octobus.net> parents: 39260 diff changeset	30 REVIDX_ISCENSORED,
729082bb9938 revlog: split constants into a new `revlogutils.constants` module Boris Feld <boris.feld@octobus.net> parents: 39260 diff changeset	31 REVIDX_RAWTEXT_CHANGING_FLAGS,
729082bb9938 revlog: split constants into a new `revlogutils.constants` module Boris Feld <boris.feld@octobus.net> parents: 39260 diff changeset	32 )
39357 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	33
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	34 from ..thirdparty import attr
39357 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	35
51788 278af66e6595 typing: induce pytype to use the standard `attr` instead of the vendored copy Matt Harbison <matt_harbison@yahoo.com> parents: 51762 diff changeset	36 # Force pytype to use the non-vendored package
278af66e6595 typing: induce pytype to use the standard `attr` instead of the vendored copy Matt Harbison <matt_harbison@yahoo.com> parents: 51762 diff changeset	37 if typing.TYPE_CHECKING:
278af66e6595 typing: induce pytype to use the standard `attr` instead of the vendored copy Matt Harbison <matt_harbison@yahoo.com> parents: 51762 diff changeset	38 # noinspection PyPackageRequirements
278af66e6595 typing: induce pytype to use the standard `attr` instead of the vendored copy Matt Harbison <matt_harbison@yahoo.com> parents: 51762 diff changeset	39 import attr
278af66e6595 typing: induce pytype to use the standard `attr` instead of the vendored copy Matt Harbison <matt_harbison@yahoo.com> parents: 51762 diff changeset	40
39357 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	41 from .. import (
27361 29f50344fa83 revlog: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27251 diff changeset	42 error,
29f50344fa83 revlog: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27251 diff changeset	43 mdiff,
41111 38e88450138c delta: have a native implementation of _findsnapshot Boris Feld <boris.feld@octobus.net> parents: 41082 diff changeset	44 util,
27361 29f50344fa83 revlog: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27251 diff changeset	45 )
10913 f2ecc5733c89 revlog: factor out _maxinline global. Greg Ward <greg-hg@gerg.ca> parents: 10404 diff changeset	46
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	47 from . import flagutil
42994 dff95420480f flagprocessors: make `processflagsraw` a module level function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42884 diff changeset	48
39357 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	49 # maximum <delta-chain-data>/<revision-text-length> ratio
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	50 LIMIT_DELTA2TEXT = 2
1091 d62130f99a73 Move hash function back to revlog from node mpm@selenic.com parents: 1089 diff changeset	51
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	52
49037 642e31cb55f0 py3: use class X: instead of class X(object): Gregory Szorc <gregory.szorc@gmail.com> parents: 48966 diff changeset	53 class _testrevlog:
38637 e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	54 """minimalist fake revlog to use in doctests"""
e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	55
40684 85b14f0dc334 doctest: add a `issnapshot` method to _testrevlog Boris Feld <boris.feld@octobus.net> parents: 40655 diff changeset	56 def __init__(self, data, density=0.5, mingap=0, snapshot=()):
38637 e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	57 """data is an list of revision payload boundaries"""
51050 127656e0b97b revlog: use the new Config classes in _testrevlog Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51025 diff changeset	58 from .. import revlog
127656e0b97b revlog: use the new Config classes in _testrevlog Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51025 diff changeset	59
38637 e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	60 self._data = data
51050 127656e0b97b revlog: use the new Config classes in _testrevlog Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51025 diff changeset	61 self.data_config = revlog.DataConfig()
127656e0b97b revlog: use the new Config classes in _testrevlog Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51025 diff changeset	62 self.data_config.sr_density_threshold = density
127656e0b97b revlog: use the new Config classes in _testrevlog Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51025 diff changeset	63 self.data_config.sr_min_gap_size = mingap
127656e0b97b revlog: use the new Config classes in _testrevlog Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51025 diff changeset	64 self.delta_config = revlog.DeltaConfig()
127656e0b97b revlog: use the new Config classes in _testrevlog Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51025 diff changeset	65 self.feature_config = revlog.FeatureConfig()
40684 85b14f0dc334 doctest: add a `issnapshot` method to _testrevlog Boris Feld <boris.feld@octobus.net> parents: 40655 diff changeset	66 self._snapshot = set(snapshot)
40748 39d29542fe40 sparse-revlog: put the native implementation of slicechunktodensity to use Boris Feld <boris.feld@octobus.net> parents: 40698 diff changeset	67 self.index = None
38637 e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	68
e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	69 def start(self, rev):
41082 88d813cd9acd revlog: fix pure python slicing test when chain contains nullrev Boris Feld <boris.feld@octobus.net> parents: 41036 diff changeset	70 if rev == nullrev:
88d813cd9acd revlog: fix pure python slicing test when chain contains nullrev Boris Feld <boris.feld@octobus.net> parents: 41036 diff changeset	71 return 0
38637 e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	72 if rev == 0:
e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	73 return 0
e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	74 return self._data[rev - 1]
e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	75
e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	76 def end(self, rev):
41082 88d813cd9acd revlog: fix pure python slicing test when chain contains nullrev Boris Feld <boris.feld@octobus.net> parents: 41036 diff changeset	77 if rev == nullrev:
88d813cd9acd revlog: fix pure python slicing test when chain contains nullrev Boris Feld <boris.feld@octobus.net> parents: 41036 diff changeset	78 return 0
38637 e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	79 return self._data[rev]
e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	80
e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	81 def length(self, rev):
e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	82 return self.end(rev) - self.start(rev)
e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	83
38718 f8762ea73e0d sparse-revlog: implement algorithm to write sparse delta chains (issue5480) Paul Morelle <paul.morelle@octobus.net> parents: 38717 diff changeset	84 def __len__(self):
f8762ea73e0d sparse-revlog: implement algorithm to write sparse delta chains (issue5480) Paul Morelle <paul.morelle@octobus.net> parents: 38717 diff changeset	85 return len(self._data)
f8762ea73e0d sparse-revlog: implement algorithm to write sparse delta chains (issue5480) Paul Morelle <paul.morelle@octobus.net> parents: 38717 diff changeset	86
40684 85b14f0dc334 doctest: add a `issnapshot` method to _testrevlog Boris Feld <boris.feld@octobus.net> parents: 40655 diff changeset	87 def issnapshot(self, rev):
41082 88d813cd9acd revlog: fix pure python slicing test when chain contains nullrev Boris Feld <boris.feld@octobus.net> parents: 41036 diff changeset	88 if rev == nullrev:
88d813cd9acd revlog: fix pure python slicing test when chain contains nullrev Boris Feld <boris.feld@octobus.net> parents: 41036 diff changeset	89 return True
40684 85b14f0dc334 doctest: add a `issnapshot` method to _testrevlog Boris Feld <boris.feld@octobus.net> parents: 40655 diff changeset	90 return rev in self._snapshot
85b14f0dc334 doctest: add a `issnapshot` method to _testrevlog Boris Feld <boris.feld@octobus.net> parents: 40655 diff changeset	91
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	92
40651 3ac23dad6364 sparse-revlog: drop unused deltainfo parameter from _slicechunktodensity Boris Feld <boris.feld@octobus.net> parents: 40650 diff changeset	93 def slicechunk(revlog, revs, targetsize=None):
34824 e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	94 """slice revs to reduce the amount of unrelated data to be read from disk.
e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	95
e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	96 ``revs`` is sliced into groups that should be read in one time.
e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	97 Assume that revs are sorted.
38640 f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	98
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	99 The initial chunk is sliced until the overall density (payload/chunks-span
51067 8c614fa16330 revlog: remove legacy usage of `_srdensitythreshold` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51065 diff changeset	100 ratio) is above `revlog.data_config.sr_density_threshold`. No gap smaller
51068 533d6943f6a3 revlog: remove legacy usage of `_srmingapsize` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51067 diff changeset	101 than `revlog.data_config.sr_min_gap_size` is skipped.
38640 f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	102
38643 967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size Boris Feld <boris.feld@octobus.net> parents: 38642 diff changeset	103 If `targetsize` is set, no chunk larger than `targetsize` will be yield.
967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size Boris Feld <boris.feld@octobus.net> parents: 38642 diff changeset	104 For consistency with other slicing choice, this limit won't go lower than
51068 533d6943f6a3 revlog: remove legacy usage of `_srmingapsize` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51067 diff changeset	105 `revlog.data_config.sr_min_gap_size`.
38643 967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size Boris Feld <boris.feld@octobus.net> parents: 38642 diff changeset	106
967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size Boris Feld <boris.feld@octobus.net> parents: 38642 diff changeset	107 If individual revisions chunk are larger than this limit, they will still
967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size Boris Feld <boris.feld@octobus.net> parents: 38642 diff changeset	108 be raised individually.
967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size Boris Feld <boris.feld@octobus.net> parents: 38642 diff changeset	109
40685 9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	110 >>> data = [
38640 f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	111 ... 5, #00 (5)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	112 ... 10, #01 (5)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	113 ... 12, #02 (2)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	114 ... 12, #03 (empty)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	115 ... 27, #04 (15)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	116 ... 31, #05 (4)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	117 ... 31, #06 (empty)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	118 ... 42, #07 (11)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	119 ... 47, #08 (5)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	120 ... 47, #09 (empty)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	121 ... 48, #10 (1)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	122 ... 51, #11 (3)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	123 ... 74, #12 (23)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	124 ... 85, #13 (11)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	125 ... 86, #14 (1)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	126 ... 91, #15 (5)
40685 9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	127 ... ]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	128 >>> revlog = _testrevlog(data, snapshot=range(16))
38640 f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	129
39357 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	130 >>> list(slicechunk(revlog, list(range(16))))
38640 f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	131 [[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]]
39357 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	132 >>> list(slicechunk(revlog, [0, 15]))
38640 f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	133 [[0], [15]]
39357 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	134 >>> list(slicechunk(revlog, [0, 11, 15]))
38640 f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	135 [[0], [11], [15]]
39357 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	136 >>> list(slicechunk(revlog, [0, 11, 13, 15]))
38640 f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	137 [[0], [11, 13, 15]]
39357 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	138 >>> list(slicechunk(revlog, [1, 2, 3, 5, 8, 10, 11, 14]))
38640 f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	139 [[1, 2], [5, 8, 10, 11], [14]]
38643 967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size Boris Feld <boris.feld@octobus.net> parents: 38642 diff changeset	140
967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size Boris Feld <boris.feld@octobus.net> parents: 38642 diff changeset	141 Slicing with a maximum chunk size
39357 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	142 >>> list(slicechunk(revlog, [0, 11, 13, 15], targetsize=15))
38643 967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size Boris Feld <boris.feld@octobus.net> parents: 38642 diff changeset	143 [[0], [11], [13], [15]]
39357 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	144 >>> list(slicechunk(revlog, [0, 11, 13, 15], targetsize=20))
38643 967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size Boris Feld <boris.feld@octobus.net> parents: 38642 diff changeset	145 [[0], [11], [13, 15]]
41082 88d813cd9acd revlog: fix pure python slicing test when chain contains nullrev Boris Feld <boris.feld@octobus.net> parents: 41036 diff changeset	146
88d813cd9acd revlog: fix pure python slicing test when chain contains nullrev Boris Feld <boris.feld@octobus.net> parents: 41036 diff changeset	147 Slicing involving nullrev
88d813cd9acd revlog: fix pure python slicing test when chain contains nullrev Boris Feld <boris.feld@octobus.net> parents: 41036 diff changeset	148 >>> list(slicechunk(revlog, [-1, 0, 11, 13, 15], targetsize=20))
88d813cd9acd revlog: fix pure python slicing test when chain contains nullrev Boris Feld <boris.feld@octobus.net> parents: 41036 diff changeset	149 [[-1, 0], [11], [13, 15]]
88d813cd9acd revlog: fix pure python slicing test when chain contains nullrev Boris Feld <boris.feld@octobus.net> parents: 41036 diff changeset	150 >>> list(slicechunk(revlog, [-1, 13, 15], targetsize=5))
88d813cd9acd revlog: fix pure python slicing test when chain contains nullrev Boris Feld <boris.feld@octobus.net> parents: 41036 diff changeset	151 [[-1], [13], [15]]
34824 e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	152 """
38643 967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size Boris Feld <boris.feld@octobus.net> parents: 38642 diff changeset	153 if targetsize is not None:
51068 533d6943f6a3 revlog: remove legacy usage of `_srmingapsize` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51067 diff changeset	154 targetsize = max(targetsize, revlog.data_config.sr_min_gap_size)
38718 f8762ea73e0d sparse-revlog: implement algorithm to write sparse delta chains (issue5480) Paul Morelle <paul.morelle@octobus.net> parents: 38717 diff changeset	155 # targetsize should not be specified when evaluating delta candidates:
f8762ea73e0d sparse-revlog: implement algorithm to write sparse delta chains (issue5480) Paul Morelle <paul.morelle@octobus.net> parents: 38717 diff changeset	156 # * targetsize is used to ensure we stay within specification when reading,
40748 39d29542fe40 sparse-revlog: put the native implementation of slicechunktodensity to use Boris Feld <boris.feld@octobus.net> parents: 40698 diff changeset	157 densityslicing = getattr(revlog.index, 'slicechunktodensity', None)
39d29542fe40 sparse-revlog: put the native implementation of slicechunktodensity to use Boris Feld <boris.feld@octobus.net> parents: 40698 diff changeset	158 if densityslicing is None:
39d29542fe40 sparse-revlog: put the native implementation of slicechunktodensity to use Boris Feld <boris.feld@octobus.net> parents: 40698 diff changeset	159 densityslicing = lambda x, y, z: _slicechunktodensity(revlog, x, y, z)
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	160 for chunk in densityslicing(
51068 533d6943f6a3 revlog: remove legacy usage of `_srmingapsize` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51067 diff changeset	161 revs,
533d6943f6a3 revlog: remove legacy usage of `_srmingapsize` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51067 diff changeset	162 revlog.data_config.sr_density_threshold,
533d6943f6a3 revlog: remove legacy usage of `_srmingapsize` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51067 diff changeset	163 revlog.data_config.sr_min_gap_size,
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	164 ):
52669 e627cc25b6f3 pyupgrade: rewrite `yield` statements in a loop to `yield from` Matt Harbison <matt_harbison@yahoo.com> parents: 51901 diff changeset	165 yield from _slicechunktosize(revlog, chunk, targetsize)
38641 feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	166
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	167
38718 f8762ea73e0d sparse-revlog: implement algorithm to write sparse delta chains (issue5480) Paul Morelle <paul.morelle@octobus.net> parents: 38717 diff changeset	168 def _slicechunktosize(revlog, revs, targetsize=None):
38642 e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	169 """slice revs to match the target size
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	170
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	171 This is intended to be used on chunk that density slicing selected by that
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	172 are still too large compared to the read garantee of revlog. This might
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	173 happens when "minimal gap size" interrupted the slicing or when chain are
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	174 built in a way that create large blocks next to each other.
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	175
40685 9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	176 >>> data = [
38642 e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	177 ... 3, #0 (3)
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	178 ... 5, #1 (2)
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	179 ... 6, #2 (1)
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	180 ... 8, #3 (2)
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	181 ... 8, #4 (empty)
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	182 ... 11, #5 (3)
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	183 ... 12, #6 (1)
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	184 ... 13, #7 (1)
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	185 ... 14, #8 (1)
40685 9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	186 ... ]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	187
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	188 == All snapshots cases ==
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	189 >>> revlog = _testrevlog(data, snapshot=range(9))
38642 e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	190
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	191 Cases where chunk is already small enough
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	192 >>> list(_slicechunktosize(revlog, [0], 3))
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	193 [[0]]
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	194 >>> list(_slicechunktosize(revlog, [6, 7], 3))
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	195 [[6, 7]]
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	196 >>> list(_slicechunktosize(revlog, [0], None))
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	197 [[0]]
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	198 >>> list(_slicechunktosize(revlog, [6, 7], None))
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	199 [[6, 7]]
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	200
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	201 cases where we need actual slicing
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	202 >>> list(_slicechunktosize(revlog, [0, 1], 3))
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	203 [[0], [1]]
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	204 >>> list(_slicechunktosize(revlog, [1, 3], 3))
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	205 [[1], [3]]
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	206 >>> list(_slicechunktosize(revlog, [1, 2, 3], 3))
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	207 [[1, 2], [3]]
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	208 >>> list(_slicechunktosize(revlog, [3, 5], 3))
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	209 [[3], [5]]
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	210 >>> list(_slicechunktosize(revlog, [3, 4, 5], 3))
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	211 [[3], [5]]
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	212 >>> list(_slicechunktosize(revlog, [5, 6, 7, 8], 3))
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	213 [[5], [6, 7, 8]]
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	214 >>> list(_slicechunktosize(revlog, [0, 1, 2, 3, 4, 5, 6, 7, 8], 3))
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	215 [[0], [1, 2], [3], [5], [6, 7, 8]]
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	216
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	217 Case with too large individual chunk (must return valid chunk)
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	218 >>> list(_slicechunktosize(revlog, [0, 1], 2))
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	219 [[0], [1]]
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	220 >>> list(_slicechunktosize(revlog, [1, 3], 1))
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	221 [[1], [3]]
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	222 >>> list(_slicechunktosize(revlog, [3, 4, 5], 2))
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	223 [[3], [5]]
40685 9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	224
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	225 == No Snapshot cases ==
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	226 >>> revlog = _testrevlog(data)
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	227
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	228 Cases where chunk is already small enough
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	229 >>> list(_slicechunktosize(revlog, [0], 3))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	230 [[0]]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	231 >>> list(_slicechunktosize(revlog, [6, 7], 3))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	232 [[6, 7]]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	233 >>> list(_slicechunktosize(revlog, [0], None))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	234 [[0]]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	235 >>> list(_slicechunktosize(revlog, [6, 7], None))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	236 [[6, 7]]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	237
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	238 cases where we need actual slicing
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	239 >>> list(_slicechunktosize(revlog, [0, 1], 3))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	240 [[0], [1]]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	241 >>> list(_slicechunktosize(revlog, [1, 3], 3))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	242 [[1], [3]]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	243 >>> list(_slicechunktosize(revlog, [1, 2, 3], 3))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	244 [[1], [2, 3]]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	245 >>> list(_slicechunktosize(revlog, [3, 5], 3))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	246 [[3], [5]]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	247 >>> list(_slicechunktosize(revlog, [3, 4, 5], 3))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	248 [[3], [4, 5]]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	249 >>> list(_slicechunktosize(revlog, [5, 6, 7, 8], 3))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	250 [[5], [6, 7, 8]]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	251 >>> list(_slicechunktosize(revlog, [0, 1, 2, 3, 4, 5, 6, 7, 8], 3))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	252 [[0], [1, 2], [3], [5], [6, 7, 8]]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	253
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	254 Case with too large individual chunk (must return valid chunk)
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	255 >>> list(_slicechunktosize(revlog, [0, 1], 2))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	256 [[0], [1]]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	257 >>> list(_slicechunktosize(revlog, [1, 3], 1))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	258 [[1], [3]]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	259 >>> list(_slicechunktosize(revlog, [3, 4, 5], 2))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	260 [[3], [5]]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	261
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	262 == mixed case ==
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	263 >>> revlog = _testrevlog(data, snapshot=[0, 1, 2])
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	264 >>> list(_slicechunktosize(revlog, list(range(9)), 5))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	265 [[0, 1], [2], [3, 4, 5], [6, 7, 8]]
38642 e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	266 """
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	267 assert targetsize is None or 0 <= targetsize
40685 9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	268 startdata = revlog.start(revs[0])
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	269 enddata = revlog.end(revs[-1])
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	270 fullspan = enddata - startdata
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	271 if targetsize is None or fullspan <= targetsize:
38642 e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	272 yield revs
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	273 return
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	274
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	275 startrevidx = 0
40698 2eb48aa0acce sparse-revlog: align endrevidx usages in the _slicechunktosize Boris Feld <boris.feld@octobus.net> parents: 40697 diff changeset	276 endrevidx = 1
38642 e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	277 iterrevs = enumerate(revs)
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	278 next(iterrevs) # skip first rev.
40685 9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	279 # first step: get snapshots out of the way
38642 e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	280 for idx, r in iterrevs:
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	281 span = revlog.end(r) - startdata
40685 9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	282 snapshot = revlog.issnapshot(r)
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	283 if span <= targetsize and snapshot:
40698 2eb48aa0acce sparse-revlog: align endrevidx usages in the _slicechunktosize Boris Feld <boris.feld@octobus.net> parents: 40697 diff changeset	284 endrevidx = idx + 1
38642 e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	285 else:
40698 2eb48aa0acce sparse-revlog: align endrevidx usages in the _slicechunktosize Boris Feld <boris.feld@octobus.net> parents: 40697 diff changeset	286 chunk = _trimchunk(revlog, revs, startrevidx, endrevidx)
38642 e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	287 if chunk:
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	288 yield chunk
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	289 startrevidx = idx
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	290 startdata = revlog.start(r)
40698 2eb48aa0acce sparse-revlog: align endrevidx usages in the _slicechunktosize Boris Feld <boris.feld@octobus.net> parents: 40697 diff changeset	291 endrevidx = idx + 1
40685 9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	292 if not snapshot:
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	293 break
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	294
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	295 # for the others, we use binary slicing to quickly converge toward valid
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	296 # chunks (otherwise, we might end up looking for start/end of many
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	297 # revisions). This logic is not looking for the perfect slicing point, it
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	298 # focuses on quickly converging toward valid chunks.
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	299 nbitem = len(revs)
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	300 while (enddata - startdata) > targetsize:
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	301 endrevidx = nbitem
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	302 if nbitem - startrevidx <= 1:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	303 break # protect against individual chunk larger than limit
40685 9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	304 localenddata = revlog.end(revs[endrevidx - 1])
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	305 span = localenddata - startdata
40697 fd1d41ccbe38 sparse-revlog: use `span` variable as intended Boris Feld <boris.feld@octobus.net> parents: 40685 diff changeset	306 while span > targetsize:
40685 9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	307 if endrevidx - startrevidx <= 1:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	308 break # protect against individual chunk larger than limit
40685 9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	309 endrevidx -= (endrevidx - startrevidx) // 2
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	310 localenddata = revlog.end(revs[endrevidx - 1])
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	311 span = localenddata - startdata
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	312 chunk = _trimchunk(revlog, revs, startrevidx, endrevidx)
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	313 if chunk:
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	314 yield chunk
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	315 startrevidx = endrevidx
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	316 startdata = revlog.start(revs[startrevidx])
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	317
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	318 chunk = _trimchunk(revlog, revs, startrevidx)
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	319 if chunk:
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	320 yield chunk
38642 e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	321
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	322
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	323 def _slicechunktodensity(revlog, revs, targetdensity=0.5, mingapsize=0):
38641 feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	324 """slice revs to reduce the amount of unrelated data to be read from disk.
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	325
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	326 ``revs`` is sliced into groups that should be read in one time.
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	327 Assume that revs are sorted.
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	328
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	329 The initial chunk is sliced until the overall density (payload/chunks-span
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	330 ratio) is above `targetdensity`. No gap smaller than `mingapsize` is
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	331 skipped.
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	332
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	333 >>> revlog = _testrevlog([
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	334 ... 5, #00 (5)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	335 ... 10, #01 (5)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	336 ... 12, #02 (2)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	337 ... 12, #03 (empty)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	338 ... 27, #04 (15)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	339 ... 31, #05 (4)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	340 ... 31, #06 (empty)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	341 ... 42, #07 (11)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	342 ... 47, #08 (5)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	343 ... 47, #09 (empty)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	344 ... 48, #10 (1)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	345 ... 51, #11 (3)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	346 ... 74, #12 (23)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	347 ... 85, #13 (11)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	348 ... 86, #14 (1)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	349 ... 91, #15 (5)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	350 ... ])
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	351
38655 cd1c484e31e8 revlog: adjust doctest examples to be portable to Python 3 Augie Fackler <augie@google.com> parents: 38644 diff changeset	352 >>> list(_slicechunktodensity(revlog, list(range(16))))
38641 feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	353 [[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]]
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	354 >>> list(_slicechunktodensity(revlog, [0, 15]))
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	355 [[0], [15]]
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	356 >>> list(_slicechunktodensity(revlog, [0, 11, 15]))
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	357 [[0], [11], [15]]
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	358 >>> list(_slicechunktodensity(revlog, [0, 11, 13, 15]))
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	359 [[0], [11, 13, 15]]
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	360 >>> list(_slicechunktodensity(revlog, [1, 2, 3, 5, 8, 10, 11, 14]))
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	361 [[1, 2], [5, 8, 10, 11], [14]]
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	362 >>> list(_slicechunktodensity(revlog, [1, 2, 3, 5, 8, 10, 11, 14],
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	363 ... mingapsize=20))
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	364 [[1, 2, 3, 5, 8, 10, 11], [14]]
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	365 >>> list(_slicechunktodensity(revlog, [1, 2, 3, 5, 8, 10, 11, 14],
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	366 ... targetdensity=0.95))
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	367 [[1, 2], [5], [8, 10, 11], [14]]
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	368 >>> list(_slicechunktodensity(revlog, [1, 2, 3, 5, 8, 10, 11, 14],
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	369 ... targetdensity=0.95, mingapsize=12))
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	370 [[1, 2], [5, 8, 10, 11], [14]]
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	371 """
34824 e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	372 start = revlog.start
e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	373 length = revlog.length
e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	374
34880 9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	375 if len(revs) <= 1:
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	376 yield revs
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	377 return
34824 e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	378
40651 3ac23dad6364 sparse-revlog: drop unused deltainfo parameter from _slicechunktodensity Boris Feld <boris.feld@octobus.net> parents: 40650 diff changeset	379 deltachainspan = segmentspan(revlog, revs)
34880 9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	380
38641 feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	381 if deltachainspan < mingapsize:
38635 d083ae26c325 revlog: early return in _slicechunk when span is already small enough Boris Feld <boris.feld@octobus.net> parents: 38634 diff changeset	382 yield revs
d083ae26c325 revlog: early return in _slicechunk when span is already small enough Boris Feld <boris.feld@octobus.net> parents: 38634 diff changeset	383 return
d083ae26c325 revlog: early return in _slicechunk when span is already small enough Boris Feld <boris.feld@octobus.net> parents: 38634 diff changeset	384
38718 f8762ea73e0d sparse-revlog: implement algorithm to write sparse delta chains (issue5480) Paul Morelle <paul.morelle@octobus.net> parents: 38717 diff changeset	385 readdata = deltachainspan
40653 bfbfd15d65bd sparse-revlog: fast-path before computing payload size Boris Feld <boris.feld@octobus.net> parents: 40652 diff changeset	386 chainpayload = sum(length(r) for r in revs)
34824 e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	387
34880 9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	388 if deltachainspan:
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	389 density = chainpayload / float(deltachainspan)
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	390 else:
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	391 density = 1.0
34824 e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	392
38641 feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	393 if density >= targetdensity:
38634 f0ea8b847831 revlog: early return in _slicechunk when density is already good Paul Morelle <paul.morelle@octobus.net> parents: 38632 diff changeset	394 yield revs
f0ea8b847831 revlog: early return in _slicechunk when density is already good Paul Morelle <paul.morelle@octobus.net> parents: 38632 diff changeset	395 return
f0ea8b847831 revlog: early return in _slicechunk when density is already good Paul Morelle <paul.morelle@octobus.net> parents: 38632 diff changeset	396
34880 9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	397 # Store the gaps in a heap to have them sorted by decreasing size
40654 54de23400b2a sparse-revlog: stop using a heap to track gaps Boris Feld <boris.feld@octobus.net> parents: 40653 diff changeset	398 gaps = []
34880 9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	399 prevend = None
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	400 for i, rev in enumerate(revs):
40651 3ac23dad6364 sparse-revlog: drop unused deltainfo parameter from _slicechunktodensity Boris Feld <boris.feld@octobus.net> parents: 40650 diff changeset	401 revstart = start(rev)
3ac23dad6364 sparse-revlog: drop unused deltainfo parameter from _slicechunktodensity Boris Feld <boris.feld@octobus.net> parents: 40650 diff changeset	402 revlen = length(rev)
34824 e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	403
34898 1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk Paul Morelle <paul.morelle@octobus.net> parents: 34881 diff changeset	404 # Skip empty revisions to form larger holes
1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk Paul Morelle <paul.morelle@octobus.net> parents: 34881 diff changeset	405 if revlen == 0:
1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk Paul Morelle <paul.morelle@octobus.net> parents: 34881 diff changeset	406 continue
1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk Paul Morelle <paul.morelle@octobus.net> parents: 34881 diff changeset	407
34880 9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	408 if prevend is not None:
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	409 gapsize = revstart - prevend
34881 8c9b08a0c48c sparse-read: skip gaps too small to be worth splitting Paul Morelle <paul.morelle@octobus.net> parents: 34880 diff changeset	410 # only consider holes that are large enough
38641 feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	411 if gapsize > mingapsize:
40654 54de23400b2a sparse-revlog: stop using a heap to track gaps Boris Feld <boris.feld@octobus.net> parents: 40653 diff changeset	412 gaps.append((gapsize, i))
34880 9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	413
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	414 prevend = revstart + revlen
40654 54de23400b2a sparse-revlog: stop using a heap to track gaps Boris Feld <boris.feld@octobus.net> parents: 40653 diff changeset	415 # sort the gaps to pop them from largest to small
54de23400b2a sparse-revlog: stop using a heap to track gaps Boris Feld <boris.feld@octobus.net> parents: 40653 diff changeset	416 gaps.sort()
34880 9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	417
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	418 # Collect the indices of the largest holes until the density is acceptable
40655 526ee887c4d5 sparse-revlog: stop using a heap to track selected gap Boris Feld <boris.feld@octobus.net> parents: 40654 diff changeset	419 selected = []
40654 54de23400b2a sparse-revlog: stop using a heap to track gaps Boris Feld <boris.feld@octobus.net> parents: 40653 diff changeset	420 while gaps and density < targetdensity:
54de23400b2a sparse-revlog: stop using a heap to track gaps Boris Feld <boris.feld@octobus.net> parents: 40653 diff changeset	421 gapsize, gapidx = gaps.pop()
34880 9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	422
40655 526ee887c4d5 sparse-revlog: stop using a heap to track selected gap Boris Feld <boris.feld@octobus.net> parents: 40654 diff changeset	423 selected.append(gapidx)
34824 e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	424
34880 9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	425 # the gap sizes are stored as negatives to be sorted decreasingly
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	426 # by the heap
40654 54de23400b2a sparse-revlog: stop using a heap to track gaps Boris Feld <boris.feld@octobus.net> parents: 40653 diff changeset	427 readdata -= gapsize
34880 9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	428 if readdata > 0:
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	429 density = chainpayload / float(readdata)
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	430 else:
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	431 density = 1.0
40655 526ee887c4d5 sparse-revlog: stop using a heap to track selected gap Boris Feld <boris.feld@octobus.net> parents: 40654 diff changeset	432 selected.sort()
34824 e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	433
34880 9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	434 # Cut the revs at collected indices
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	435 previdx = 0
40655 526ee887c4d5 sparse-revlog: stop using a heap to track selected gap Boris Feld <boris.feld@octobus.net> parents: 40654 diff changeset	436 for idx in selected:
34898 1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk Paul Morelle <paul.morelle@octobus.net> parents: 34881 diff changeset	437 chunk = _trimchunk(revlog, revs, previdx, idx)
1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk Paul Morelle <paul.morelle@octobus.net> parents: 34881 diff changeset	438 if chunk:
1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk Paul Morelle <paul.morelle@octobus.net> parents: 34881 diff changeset	439 yield chunk
1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk Paul Morelle <paul.morelle@octobus.net> parents: 34881 diff changeset	440
34880 9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	441 previdx = idx
34898 1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk Paul Morelle <paul.morelle@octobus.net> parents: 34881 diff changeset	442
1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk Paul Morelle <paul.morelle@octobus.net> parents: 34881 diff changeset	443 chunk = _trimchunk(revlog, revs, previdx)
1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk Paul Morelle <paul.morelle@octobus.net> parents: 34881 diff changeset	444 if chunk:
1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk Paul Morelle <paul.morelle@octobus.net> parents: 34881 diff changeset	445 yield chunk
34824 e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	446
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	447
39357 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	448 def _trimchunk(revlog, revs, startidx, endidx=None):
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	449 """returns revs[startidx:endidx] without empty trailing revs
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	450
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	451 Doctest Setup
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	452 >>> revlog = _testrevlog([
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	453 ... 5, #0
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	454 ... 10, #1
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	455 ... 12, #2
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	456 ... 12, #3 (empty)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	457 ... 17, #4
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	458 ... 21, #5
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	459 ... 21, #6 (empty)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	460 ... ])
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	461
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	462 Contiguous cases:
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	463 >>> _trimchunk(revlog, [0, 1, 2, 3, 4, 5, 6], 0)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	464 [0, 1, 2, 3, 4, 5]
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	465 >>> _trimchunk(revlog, [0, 1, 2, 3, 4, 5, 6], 0, 5)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	466 [0, 1, 2, 3, 4]
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	467 >>> _trimchunk(revlog, [0, 1, 2, 3, 4, 5, 6], 0, 4)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	468 [0, 1, 2]
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	469 >>> _trimchunk(revlog, [0, 1, 2, 3, 4, 5, 6], 2, 4)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	470 [2]
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	471 >>> _trimchunk(revlog, [0, 1, 2, 3, 4, 5, 6], 3)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	472 [3, 4, 5]
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	473 >>> _trimchunk(revlog, [0, 1, 2, 3, 4, 5, 6], 3, 5)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	474 [3, 4]
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	475
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	476 Discontiguous cases:
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	477 >>> _trimchunk(revlog, [1, 3, 5, 6], 0)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	478 [1, 3, 5]
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	479 >>> _trimchunk(revlog, [1, 3, 5, 6], 0, 2)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	480 [1]
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	481 >>> _trimchunk(revlog, [1, 3, 5, 6], 1, 3)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	482 [3, 5]
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	483 >>> _trimchunk(revlog, [1, 3, 5, 6], 1)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	484 [3, 5]
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	485 """
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	486 length = revlog.length
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	487
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	488 if endidx is None:
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	489 endidx = len(revs)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	490
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	491 # If we have a non-emtpy delta candidate, there are nothing to trim
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	492 if revs[endidx - 1] < len(revlog):
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	493 # Trim empty revs at the end, except the very first revision of a chain
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	494 while (
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	495 endidx > 1 and endidx > startidx and length(revs[endidx - 1]) == 0
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	496 ):
39357 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	497 endidx -= 1
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	498
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	499 return revs[startidx:endidx]
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	500
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	501
40652 a32ccd32982b sparse-revlog: drop unused deltainfo parameter from segmentspan Boris Feld <boris.feld@octobus.net> parents: 40651 diff changeset	502 def segmentspan(revlog, revs):
39357 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	503 """Get the byte span of a segment of revisions
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	504
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	505 revs is a sorted array of revision numbers
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	506
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	507 >>> revlog = _testrevlog([
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	508 ... 5, #0
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	509 ... 10, #1
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	510 ... 12, #2
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	511 ... 12, #3 (empty)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	512 ... 17, #4
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	513 ... ])
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	514
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	515 >>> segmentspan(revlog, [0, 1, 2, 3, 4])
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	516 17
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	517 >>> segmentspan(revlog, [0, 4])
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	518 17
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	519 >>> segmentspan(revlog, [3, 4])
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	520 5
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	521 >>> segmentspan(revlog, [1, 2, 3,])
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	522 7
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	523 >>> segmentspan(revlog, [1, 3])
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	524 7
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	525 """
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	526 if not revs:
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	527 return 0
40652 a32ccd32982b sparse-revlog: drop unused deltainfo parameter from segmentspan Boris Feld <boris.feld@octobus.net> parents: 40651 diff changeset	528 end = revlog.end(revs[-1])
39357 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	529 return end - revlog.start(revs[0])
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	530
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	531
51025 509f0f7fc89e delta-computer: stop explicitly taking file handle Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50952 diff changeset	532 def _textfromdelta(revlog, baserev, delta, p1, p2, flags, expectednode):
39358 fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39357 diff changeset	533 """build full text from a (base, delta) pair and other metadata"""
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39357 diff changeset	534 # special case deltas which replace entire base; no need to decode
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39357 diff changeset	535 # base revision. this neatly avoids censored bases, which throw when
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39357 diff changeset	536 # they're decoded.
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	537 hlen = struct.calcsize(b">lll")
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	538 if delta[:hlen] == mdiff.replacediffheader(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	539 revlog.rawsize(baserev), len(delta) - hlen
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	540 ):
39358 fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39357 diff changeset	541 fulltext = delta[hlen:]
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39357 diff changeset	542 else:
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39357 diff changeset	543 # deltabase is rawtext before changed by flag processors, which is
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39357 diff changeset	544 # equivalent to non-raw text
51025 509f0f7fc89e delta-computer: stop explicitly taking file handle Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50952 diff changeset	545 basetext = revlog.revision(baserev)
39358 fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39357 diff changeset	546 fulltext = mdiff.patch(basetext, delta)
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39357 diff changeset	547
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39357 diff changeset	548 try:
42994 dff95420480f flagprocessors: make `processflagsraw` a module level function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42884 diff changeset	549 validatehash = flagutil.processflagsraw(revlog, fulltext, flags)
39358 fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39357 diff changeset	550 if validatehash:
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39357 diff changeset	551 revlog.checkhash(fulltext, expectednode, p1=p1, p2=p2)
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39357 diff changeset	552 if flags & REVIDX_ISCENSORED:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	553 raise error.StorageError(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	554 _(b'node %s is not censored') % expectednode
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	555 )
39790 4a2466b2a434 revlog: drop some more error aliases (API) Gregory Szorc <gregory.szorc@gmail.com> parents: 39789 diff changeset	556 except error.CensoredNodeError:
39358 fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39357 diff changeset	557 # must pass the censored index flag to add censored revisions
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39357 diff changeset	558 if not flags & REVIDX_ISCENSORED:
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39357 diff changeset	559 raise
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39357 diff changeset	560 return fulltext
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39357 diff changeset	561
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	562
35638 edc9330acac1 revlog: introduce 'deltainfo' to distinguish from 'delta' Paul Morelle <paul.morelle@octobus.net> parents: 35637 diff changeset	563 @attr.s(slots=True, frozen=True)
49037 642e31cb55f0 py3: use class X: instead of class X(object): Gregory Szorc <gregory.szorc@gmail.com> parents: 48966 diff changeset	564 class _deltainfo:
35638 edc9330acac1 revlog: introduce 'deltainfo' to distinguish from 'delta' Paul Morelle <paul.morelle@octobus.net> parents: 35637 diff changeset	565 distance = attr.ib()
edc9330acac1 revlog: introduce 'deltainfo' to distinguish from 'delta' Paul Morelle <paul.morelle@octobus.net> parents: 35637 diff changeset	566 deltalen = attr.ib()
edc9330acac1 revlog: introduce 'deltainfo' to distinguish from 'delta' Paul Morelle <paul.morelle@octobus.net> parents: 35637 diff changeset	567 data = attr.ib()
edc9330acac1 revlog: introduce 'deltainfo' to distinguish from 'delta' Paul Morelle <paul.morelle@octobus.net> parents: 35637 diff changeset	568 base = attr.ib()
edc9330acac1 revlog: introduce 'deltainfo' to distinguish from 'delta' Paul Morelle <paul.morelle@octobus.net> parents: 35637 diff changeset	569 chainbase = attr.ib()
edc9330acac1 revlog: introduce 'deltainfo' to distinguish from 'delta' Paul Morelle <paul.morelle@octobus.net> parents: 35637 diff changeset	570 chainlen = attr.ib()
edc9330acac1 revlog: introduce 'deltainfo' to distinguish from 'delta' Paul Morelle <paul.morelle@octobus.net> parents: 35637 diff changeset	571 compresseddeltalen = attr.ib()
39187 e0da43e2f71f revlog: compute snapshot depth on delta info Boris Feld <boris.feld@octobus.net> parents: 39185 diff changeset	572 snapshotdepth = attr.ib()
35638 edc9330acac1 revlog: introduce 'deltainfo' to distinguish from 'delta' Paul Morelle <paul.morelle@octobus.net> parents: 35637 diff changeset	573
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	574
47264 b876f0bf7366 revlog: introduce a plain compression mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46819 diff changeset	575 def drop_u_compression(delta):
b876f0bf7366 revlog: introduce a plain compression mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46819 diff changeset	576 """turn into a "u" (no-compression) into no-compression without header
b876f0bf7366 revlog: introduce a plain compression mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46819 diff changeset	577
b876f0bf7366 revlog: introduce a plain compression mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46819 diff changeset	578 This is useful for revlog format that has better compression method.
b876f0bf7366 revlog: introduce a plain compression mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46819 diff changeset	579 """
b876f0bf7366 revlog: introduce a plain compression mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46819 diff changeset	580 assert delta.data[0] == b'u', delta.data[0]
b876f0bf7366 revlog: introduce a plain compression mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46819 diff changeset	581 return _deltainfo(
b876f0bf7366 revlog: introduce a plain compression mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46819 diff changeset	582 delta.distance,
b876f0bf7366 revlog: introduce a plain compression mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46819 diff changeset	583 delta.deltalen - 1,
b876f0bf7366 revlog: introduce a plain compression mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46819 diff changeset	584 (b'', delta.data[1]),
b876f0bf7366 revlog: introduce a plain compression mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46819 diff changeset	585 delta.base,
b876f0bf7366 revlog: introduce a plain compression mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46819 diff changeset	586 delta.chainbase,
b876f0bf7366 revlog: introduce a plain compression mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46819 diff changeset	587 delta.chainlen,
b876f0bf7366 revlog: introduce a plain compression mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46819 diff changeset	588 delta.compresseddeltalen,
b876f0bf7366 revlog: introduce a plain compression mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46819 diff changeset	589 delta.snapshotdepth,
b876f0bf7366 revlog: introduce a plain compression mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46819 diff changeset	590 )
b876f0bf7366 revlog: introduce a plain compression mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46819 diff changeset	591
b876f0bf7366 revlog: introduce a plain compression mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46819 diff changeset	592
40991 42f59d3f714d delta: exclude base candidate much smaller than the target Boris Feld <boris.feld@octobus.net> parents: 40970 diff changeset	593 # If a revision's full text is that much bigger than a base candidate full
42f59d3f714d delta: exclude base candidate much smaller than the target Boris Feld <boris.feld@octobus.net> parents: 40970 diff changeset	594 # text's, it is very unlikely that it will produce a valid delta. We no longer
42f59d3f714d delta: exclude base candidate much smaller than the target Boris Feld <boris.feld@octobus.net> parents: 40970 diff changeset	595 # consider these candidates.
41036 b373477948df revlog: limit base to rev size ratio to 500 instead of 50 Boris Feld <boris.feld@octobus.net> parents: 40992 diff changeset	596 LIMIT_BASE2TEXT = 500
40991 42f59d3f714d delta: exclude base candidate much smaller than the target Boris Feld <boris.feld@octobus.net> parents: 40970 diff changeset	597
51359 cc806f20d756 delta-find: explicitly track stage of the search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51358 diff changeset	598 ### stage of the search, used for debug and to select and to adjust some logic.
cc806f20d756 delta-find: explicitly track stage of the search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51358 diff changeset	599 # initial stage, next step is unknown
cc806f20d756 delta-find: explicitly track stage of the search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51358 diff changeset	600 _STAGE_UNSPECIFIED = "unspecified"
cc806f20d756 delta-find: explicitly track stage of the search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51358 diff changeset	601 # trying the cached delta
cc806f20d756 delta-find: explicitly track stage of the search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51358 diff changeset	602 _STAGE_CACHED = "cached"
cc806f20d756 delta-find: explicitly track stage of the search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51358 diff changeset	603 # trying delta based on parents
cc806f20d756 delta-find: explicitly track stage of the search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51358 diff changeset	604 _STAGE_PARENTS = "parents"
cc806f20d756 delta-find: explicitly track stage of the search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51358 diff changeset	605 # trying to build a valid snapshot of any level
cc806f20d756 delta-find: explicitly track stage of the search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51358 diff changeset	606 _STAGE_SNAPSHOT = "snapshot"
cc806f20d756 delta-find: explicitly track stage of the search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51358 diff changeset	607 # trying to build a delta based of the previous revision
cc806f20d756 delta-find: explicitly track stage of the search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51358 diff changeset	608 _STAGE_PREV = "prev"
cc806f20d756 delta-find: explicitly track stage of the search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51358 diff changeset	609 # trying to build a full snapshot
cc806f20d756 delta-find: explicitly track stage of the search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51358 diff changeset	610 _STAGE_FULL = "full"
cc806f20d756 delta-find: explicitly track stage of the search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51358 diff changeset	611
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	612
51356 2a333d791ecf delta-find: introduce a base class for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51355 diff changeset	613 class _BaseDeltaSearch(abc.ABC):
51330 c82e03b102a6 delta-find: introduce a _DeltaSearch object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51329 diff changeset	614 """perform the search of a good delta for a single revlog revision
50354 ca1522fe4ec8 delta-find: assume the target-rev if not specified Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49886 diff changeset	615
51330 c82e03b102a6 delta-find: introduce a _DeltaSearch object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51329 diff changeset	616 note: some of the deltacomputer.finddeltainfo logic should probably move
c82e03b102a6 delta-find: introduce a _DeltaSearch object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51329 diff changeset	617 here.
c82e03b102a6 delta-find: introduce a _DeltaSearch object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51329 diff changeset	618 """
50355 0232571255d3 delta-find: never do anything fancy when general delta is off Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50354 diff changeset	619
51330 c82e03b102a6 delta-find: introduce a _DeltaSearch object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51329 diff changeset	620 def __init__(
c82e03b102a6 delta-find: introduce a _DeltaSearch object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51329 diff changeset	621 self,
49739 35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	622 revlog,
51340 49401b7dec0c delta-find: feed revinfo to _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51338 diff changeset	623 revinfo,
49739 35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	624 p1,
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	625 p2,
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	626 cachedelta,
51330 c82e03b102a6 delta-find: introduce a _DeltaSearch object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51329 diff changeset	627 excluded_bases=None,
c82e03b102a6 delta-find: introduce a _DeltaSearch object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51329 diff changeset	628 target_rev=None,
c82e03b102a6 delta-find: introduce a _DeltaSearch object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51329 diff changeset	629 snapshot_cache=None,
c82e03b102a6 delta-find: introduce a _DeltaSearch object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51329 diff changeset	630 ):
51335 c9ab452e5611 delta-find: check DELTA_BASE_REUSE_FORCE in the _DeltaSearch.__init__ Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51334 diff changeset	631 # the DELTA_BASE_REUSE_FORCE case should have been taken care of sooner
c9ab452e5611 delta-find: check DELTA_BASE_REUSE_FORCE in the _DeltaSearch.__init__ Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51334 diff changeset	632 # so we should never end up asking such question. Adding the assert as
c9ab452e5611 delta-find: check DELTA_BASE_REUSE_FORCE in the _DeltaSearch.__init__ Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51334 diff changeset	633 # a safe-guard to detect anything that would be fishy in this regard.
c9ab452e5611 delta-find: check DELTA_BASE_REUSE_FORCE in the _DeltaSearch.__init__ Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51334 diff changeset	634 assert (
c9ab452e5611 delta-find: check DELTA_BASE_REUSE_FORCE in the _DeltaSearch.__init__ Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51334 diff changeset	635 cachedelta is None
c9ab452e5611 delta-find: check DELTA_BASE_REUSE_FORCE in the _DeltaSearch.__init__ Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51334 diff changeset	636 or cachedelta[2] != DELTA_BASE_REUSE_FORCE
c9ab452e5611 delta-find: check DELTA_BASE_REUSE_FORCE in the _DeltaSearch.__init__ Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51334 diff changeset	637 or not revlog.delta_config.general_delta
c9ab452e5611 delta-find: check DELTA_BASE_REUSE_FORCE in the _DeltaSearch.__init__ Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51334 diff changeset	638 )
51330 c82e03b102a6 delta-find: introduce a _DeltaSearch object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51329 diff changeset	639 self.revlog = revlog
51340 49401b7dec0c delta-find: feed revinfo to _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51338 diff changeset	640 self.revinfo = revinfo
49401b7dec0c delta-find: feed revinfo to _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51338 diff changeset	641 self.textlen = revinfo.textlen
51330 c82e03b102a6 delta-find: introduce a _DeltaSearch object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51329 diff changeset	642 self.p1 = p1
c82e03b102a6 delta-find: introduce a _DeltaSearch object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51329 diff changeset	643 self.p2 = p2
c82e03b102a6 delta-find: introduce a _DeltaSearch object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51329 diff changeset	644 self.cachedelta = cachedelta
c82e03b102a6 delta-find: introduce a _DeltaSearch object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51329 diff changeset	645 self.excluded_bases = excluded_bases
51334 047358417d01 delta-find: move target_rev in the _DeltaSearch.__init__ Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51333 diff changeset	646 if target_rev is None:
047358417d01 delta-find: move target_rev in the _DeltaSearch.__init__ Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51333 diff changeset	647 self.target_rev = len(self.revlog)
51330 c82e03b102a6 delta-find: introduce a _DeltaSearch object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51329 diff changeset	648 self.target_rev = target_rev
51333 87fce139e927 delta-find: move snapshot_cache in the _DeltaSearch.__init__ Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51332 diff changeset	649 if snapshot_cache is None:
87fce139e927 delta-find: move snapshot_cache in the _DeltaSearch.__init__ Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51332 diff changeset	650 # map: base-rev: [snapshot-revs]
87fce139e927 delta-find: move snapshot_cache in the _DeltaSearch.__init__ Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51332 diff changeset	651 snapshot_cache = SnapshotCache()
51330 c82e03b102a6 delta-find: introduce a _DeltaSearch object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51329 diff changeset	652 self.snapshot_cache = snapshot_cache
c82e03b102a6 delta-find: introduce a _DeltaSearch object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51329 diff changeset	653
51336 01db705bbe32 delta-find: move tested in the _DeltaSearch.__init__ Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51335 diff changeset	654 self.tested = {nullrev}
01db705bbe32 delta-find: move tested in the _DeltaSearch.__init__ Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51335 diff changeset	655
51359 cc806f20d756 delta-find: explicitly track stage of the search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51358 diff changeset	656 self.current_stage = _STAGE_UNSPECIFIED
51356 2a333d791ecf delta-find: introduce a base class for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51355 diff changeset	657 self.current_group = None
2a333d791ecf delta-find: introduce a base class for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51355 diff changeset	658 self._init_group()
51344 d0d869fccd20 delta-find: move away from the generator API for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51343 diff changeset	659
51354 f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	660 def is_good_delta_info(self, deltainfo):
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	661 """Returns True if the given delta is good.
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	662
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	663 Good means that it is within the disk span, disk size, and chain length
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	664 bounds that we know to be performant.
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	665 """
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	666 if not self._is_good_delta_info_universal(deltainfo):
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	667 return False
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	668 if not self._is_good_delta_info_chain_quality(deltainfo):
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	669 return False
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	670 return True
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	671
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	672 def _is_good_delta_info_universal(self, deltainfo):
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	673 """Returns True if the given delta is good.
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	674
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	675 This performs generic checks needed by all format variants.
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	676
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	677 This is used by is_good_delta_info.
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	678 """
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	679
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	680 if deltainfo is None:
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	681 return False
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	682
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	683 # the DELTA_BASE_REUSE_FORCE case should have been taken care of sooner
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	684 # so we should never end up asking such question. Adding the assert as
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	685 # a safe-guard to detect anything that would be fishy in this regard.
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	686 assert (
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	687 self.revinfo.cachedelta is None
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	688 or self.revinfo.cachedelta[2] != DELTA_BASE_REUSE_FORCE
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	689 or not self.revlog.delta_config.general_delta
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	690 )
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	691
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	692 # Bad delta from new delta size:
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	693 #
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	694 # If the delta size is larger than the target text, storing the delta
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	695 # will be inefficient.
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	696 if self.revinfo.textlen < deltainfo.deltalen:
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	697 return False
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	698
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	699 return True
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	700
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	701 def _is_good_delta_info_chain_quality(self, deltainfo):
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	702 """Returns True if the chain associated with the delta is good.
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	703
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	704 This performs checks for format that use delta chains.
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	705
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	706 This is used by is_good_delta_info.
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	707 """
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	708 # - 'deltainfo.distance' is the distance from the base revision --
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	709 # bounding it limits the amount of I/O we need to do.
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	710
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	711 defaultmax = self.revinfo.textlen * 4
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	712 maxdist = self.revlog.delta_config.max_deltachain_span
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	713 if not maxdist:
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	714 maxdist = deltainfo.distance # ensure the conditional pass
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	715 maxdist = max(maxdist, defaultmax)
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	716
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	717 # Bad delta from read span:
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	718 #
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	719 # If the span of data read is larger than the maximum allowed.
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	720 #
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	721 # In the sparse-revlog case, we rely on the associated "sparse
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	722 # reading" to avoid issue related to the span of data. In theory, it
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	723 # would be possible to build pathological revlog where delta pattern
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	724 # would lead to too many reads. However, they do not happen in
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	725 # practice at all. So we skip the span check entirely.
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	726 if (
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	727 not self.revlog.delta_config.sparse_revlog
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	728 and maxdist < deltainfo.distance
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	729 ):
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	730 return False
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	731
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	732 # Bad delta from cumulated payload size:
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	733 #
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	734 # - 'deltainfo.compresseddeltalen' is the sum of the total size of
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	735 # deltas we need to apply -- bounding it limits the amount of CPU
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	736 # we consume.
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	737 max_chain_data = self.revinfo.textlen * LIMIT_DELTA2TEXT
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	738 # If the sum of delta get larger than K * target text length.
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	739 if max_chain_data < deltainfo.compresseddeltalen:
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	740 return False
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	741
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	742 # Bad delta from chain length:
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	743 #
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	744 # If the number of delta in the chain gets too high.
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	745 if (
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	746 self.revlog.delta_config.max_chain_len
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	747 and self.revlog.delta_config.max_chain_len < deltainfo.chainlen
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	748 ):
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	749 return False
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	750 return True
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	751
51344 d0d869fccd20 delta-find: move away from the generator API for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51343 diff changeset	752 @property
d0d869fccd20 delta-find: move away from the generator API for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51343 diff changeset	753 def done(self):
d0d869fccd20 delta-find: move away from the generator API for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51343 diff changeset	754 """True when all possible candidate have been tested"""
d0d869fccd20 delta-find: move away from the generator API for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51343 diff changeset	755 return self.current_group is None
d0d869fccd20 delta-find: move away from the generator API for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51343 diff changeset	756
51356 2a333d791ecf delta-find: introduce a base class for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51355 diff changeset	757 @abc.abstractmethod
51344 d0d869fccd20 delta-find: move away from the generator API for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51343 diff changeset	758 def next_group(self, good_delta=None):
d0d869fccd20 delta-find: move away from the generator API for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51343 diff changeset	759 """move to the next group to test
d0d869fccd20 delta-find: move away from the generator API for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51343 diff changeset	760
d0d869fccd20 delta-find: move away from the generator API for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51343 diff changeset	761 The group of revision to test will be available in
d0d869fccd20 delta-find: move away from the generator API for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51343 diff changeset	762 `self.current_group`. If the previous group had any good delta, the
d0d869fccd20 delta-find: move away from the generator API for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51343 diff changeset	763 best one can be passed as the `good_delta` parameter to help selecting
d0d869fccd20 delta-find: move away from the generator API for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51343 diff changeset	764 the next group.
d0d869fccd20 delta-find: move away from the generator API for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51343 diff changeset	765
d0d869fccd20 delta-find: move away from the generator API for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51343 diff changeset	766 If not revision remains to be, `self.done` will be True and
d0d869fccd20 delta-find: move away from the generator API for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51343 diff changeset	767 `self.current_group` will be None.
d0d869fccd20 delta-find: move away from the generator API for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51343 diff changeset	768 """
51356 2a333d791ecf delta-find: introduce a base class for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51355 diff changeset	769 pass
2a333d791ecf delta-find: introduce a base class for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51355 diff changeset	770
2a333d791ecf delta-find: introduce a base class for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51355 diff changeset	771 @abc.abstractmethod
2a333d791ecf delta-find: introduce a base class for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51355 diff changeset	772 def _init_group(self):
2a333d791ecf delta-find: introduce a base class for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51355 diff changeset	773 pass
2a333d791ecf delta-find: introduce a base class for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51355 diff changeset	774
2a333d791ecf delta-find: introduce a base class for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51355 diff changeset	775
51357 555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51356 diff changeset	776 class _NoDeltaSearch(_BaseDeltaSearch):
555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51356 diff changeset	777 """Search for no delta.
555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51356 diff changeset	778
555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51356 diff changeset	779 This search variant is to be used in case where we should not store delta.
555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51356 diff changeset	780 """
555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51356 diff changeset	781
555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51356 diff changeset	782 def _init_group(self):
51359 cc806f20d756 delta-find: explicitly track stage of the search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51358 diff changeset	783 self.current_stage = _STAGE_FULL
51357 555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51356 diff changeset	784
555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51356 diff changeset	785 def next_group(self, good_delta=None):
555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51356 diff changeset	786 pass
555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51356 diff changeset	787
555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51356 diff changeset	788
555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51356 diff changeset	789 class _PrevDeltaSearch(_BaseDeltaSearch):
555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51356 diff changeset	790 """Search for delta against the previous revision only
555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51356 diff changeset	791
555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51356 diff changeset	792 This search variant is to be used when the format does not allow for delta
555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51356 diff changeset	793 against arbitrary bases.
555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51356 diff changeset	794 """
555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51356 diff changeset	795
555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51356 diff changeset	796 def _init_group(self):
51359 cc806f20d756 delta-find: explicitly track stage of the search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51358 diff changeset	797 self.current_stage = _STAGE_PREV
51357 555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51356 diff changeset	798 self.current_group = [self.target_rev - 1]
51361 bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51360 diff changeset	799 self.tested.update(self.current_group)
51357 555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51356 diff changeset	800
555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51356 diff changeset	801 def next_group(self, good_delta=None):
51359 cc806f20d756 delta-find: explicitly track stage of the search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51358 diff changeset	802 self.current_stage = _STAGE_FULL
51357 555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51356 diff changeset	803 self.current_group = None
555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51356 diff changeset	804
555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51356 diff changeset	805
51367 d7e2acdd50ba delta-find: split the _DeltaSearch class in two Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51366 diff changeset	806 class _GeneralDeltaSearch(_BaseDeltaSearch):
d7e2acdd50ba delta-find: split the _DeltaSearch class in two Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51366 diff changeset	807 """Delta search variant for general-delta repository"""
51357 555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51356 diff changeset	808
51356 2a333d791ecf delta-find: introduce a base class for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51355 diff changeset	809 def _init_group(self):
51357 555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51356 diff changeset	810 # Why search for delta base if we cannot use a delta base ?
555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51356 diff changeset	811 # also see issue6056
555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51356 diff changeset	812 assert self.revlog.delta_config.general_delta
51366 701caeabbee7 delta-find: finish reworking the snapshot logic and drop more layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51365 diff changeset	813 self._candidates_iterator = self._iter_groups()
51356 2a333d791ecf delta-find: introduce a base class for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51355 diff changeset	814 self._last_good = None
51362 7083b33a2699 delta-find: explicitly deal with usage of the cached revision Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51361 diff changeset	815 if (
7083b33a2699 delta-find: explicitly deal with usage of the cached revision Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51361 diff changeset	816 self.cachedelta is not None
7083b33a2699 delta-find: explicitly deal with usage of the cached revision Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51361 diff changeset	817 and self.cachedelta[2] > DELTA_BASE_REUSE_NO
7083b33a2699 delta-find: explicitly deal with usage of the cached revision Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51361 diff changeset	818 and self._pre_filter_rev(self.cachedelta[0])
7083b33a2699 delta-find: explicitly deal with usage of the cached revision Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51361 diff changeset	819 ):
7083b33a2699 delta-find: explicitly deal with usage of the cached revision Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51361 diff changeset	820 # First we try to reuse a the delta contained in the bundle. (or from
7083b33a2699 delta-find: explicitly deal with usage of the cached revision Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51361 diff changeset	821 # the source revlog)
7083b33a2699 delta-find: explicitly deal with usage of the cached revision Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51361 diff changeset	822 #
7083b33a2699 delta-find: explicitly deal with usage of the cached revision Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51361 diff changeset	823 # This logic only applies to general delta repositories and can be
7083b33a2699 delta-find: explicitly deal with usage of the cached revision Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51361 diff changeset	824 # disabled through configuration. Disabling reuse source delta is
7083b33a2699 delta-find: explicitly deal with usage of the cached revision Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51361 diff changeset	825 # useful when we want to make sure we recomputed "optimal" deltas.
7083b33a2699 delta-find: explicitly deal with usage of the cached revision Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51361 diff changeset	826 self.current_stage = _STAGE_CACHED
7083b33a2699 delta-find: explicitly deal with usage of the cached revision Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51361 diff changeset	827 self._internal_group = (self.cachedelta[0],)
7083b33a2699 delta-find: explicitly deal with usage of the cached revision Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51361 diff changeset	828 self._internal_idx = 0
7083b33a2699 delta-find: explicitly deal with usage of the cached revision Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51361 diff changeset	829 self.current_group = self._internal_group
7083b33a2699 delta-find: explicitly deal with usage of the cached revision Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51361 diff changeset	830 self.tested.update(self.current_group)
7083b33a2699 delta-find: explicitly deal with usage of the cached revision Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51361 diff changeset	831 else:
7083b33a2699 delta-find: explicitly deal with usage of the cached revision Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51361 diff changeset	832 self._next_internal_group()
51361 bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51360 diff changeset	833
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51360 diff changeset	834 def _next_internal_group(self):
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51360 diff changeset	835 # self._internal_group can be larger than self.current_group
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51360 diff changeset	836 self._internal_idx = 0
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51360 diff changeset	837 group = self._candidates_iterator.send(self._last_good)
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51360 diff changeset	838 if group is not None:
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51360 diff changeset	839 group = self._pre_filter_candidate_revs(group)
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51360 diff changeset	840 self._internal_group = group
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51360 diff changeset	841 if self._internal_group is None:
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51360 diff changeset	842 self.current_group = None
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51360 diff changeset	843 elif len(self._internal_group) == 0:
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51360 diff changeset	844 self.next_group()
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51360 diff changeset	845 else:
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51360 diff changeset	846 chunk_size = self.revlog.delta_config.candidate_group_chunk_size
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51360 diff changeset	847 if chunk_size > 0:
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51360 diff changeset	848 self.current_group = self._internal_group[:chunk_size]
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51360 diff changeset	849 self._internal_idx += chunk_size
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51360 diff changeset	850 else:
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51360 diff changeset	851 self.current_group = self._internal_group
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51360 diff changeset	852 self._internal_idx += len(self.current_group)
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51360 diff changeset	853
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51360 diff changeset	854 self.tested.update(self.current_group)
51356 2a333d791ecf delta-find: introduce a base class for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51355 diff changeset	855
2a333d791ecf delta-find: introduce a base class for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51355 diff changeset	856 def next_group(self, good_delta=None):
51361 bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51360 diff changeset	857 old_good = self._last_good
51344 d0d869fccd20 delta-find: move away from the generator API for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51343 diff changeset	858 if good_delta is not None:
51370 99869dcf3ba0 delta-find: pass the full deltainfo to the _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51369 diff changeset	859 self._last_good = good_delta
51362 7083b33a2699 delta-find: explicitly deal with usage of the cached revision Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51361 diff changeset	860 if self.current_stage == _STAGE_CACHED and good_delta is not None:
7083b33a2699 delta-find: explicitly deal with usage of the cached revision Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51361 diff changeset	861 # the cache is good, let us use the cache as requested
7083b33a2699 delta-find: explicitly deal with usage of the cached revision Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51361 diff changeset	862 self._candidates_iterator = None
7083b33a2699 delta-find: explicitly deal with usage of the cached revision Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51361 diff changeset	863 self._internal_group = None
7083b33a2699 delta-find: explicitly deal with usage of the cached revision Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51361 diff changeset	864 self._internal_idx = None
7083b33a2699 delta-find: explicitly deal with usage of the cached revision Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51361 diff changeset	865 self.current_group = None
7083b33a2699 delta-find: explicitly deal with usage of the cached revision Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51361 diff changeset	866 return
7083b33a2699 delta-find: explicitly deal with usage of the cached revision Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51361 diff changeset	867
51361 bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51360 diff changeset	868 if (self._internal_idx < len(self._internal_group)) and (
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51360 diff changeset	869 old_good != good_delta
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51360 diff changeset	870 ):
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51360 diff changeset	871 # When the size of the candidate group is big, it can result in
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51360 diff changeset	872 # a quite significant performance impact. To reduce this, we
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51360 diff changeset	873 # can send them in smaller batches until the new batch does not
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51360 diff changeset	874 # provide any improvements.
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51360 diff changeset	875 #
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51360 diff changeset	876 # This might reduce the overall efficiency of the compression
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51360 diff changeset	877 # in some corner cases, but that should also prevent very
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51360 diff changeset	878 # pathological cases from being an issue. (eg. 20 000
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51360 diff changeset	879 # candidates).
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51360 diff changeset	880 #
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51360 diff changeset	881 # XXX note that the ordering of the group becomes important as
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51360 diff changeset	882 # it now impacts the final result. The current order is
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51360 diff changeset	883 # unprocessed and can be improved.
51459 e7bbeaa4f0a7 delta-search: fix crash caused by unbound variable Rapha?l Gom?s <rgomes@octobus.net> parents: 51370 diff changeset	884 chunk_size = self.revlog.delta_config.candidate_group_chunk_size
e7bbeaa4f0a7 delta-search: fix crash caused by unbound variable Rapha?l Gom?s <rgomes@octobus.net> parents: 51370 diff changeset	885 next_idx = self._internal_idx + chunk_size
51361 bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51360 diff changeset	886 self.current_group = self._internal_group[
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51360 diff changeset	887 self._internal_idx : next_idx
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51360 diff changeset	888 ]
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51360 diff changeset	889 self.tested.update(self.current_group)
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51360 diff changeset	890 self._internal_idx = next_idx
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51360 diff changeset	891 else:
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51360 diff changeset	892 self._next_internal_group()
39363 1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39362 diff changeset	893
51345 02cc19f4f091 delta-find: move pre-filtering of candidates in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51344 diff changeset	894 def _pre_filter_candidate_revs(self, temptative):
02cc19f4f091 delta-find: move pre-filtering of candidates in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51344 diff changeset	895 """filter possible candidate before computing a delta
02cc19f4f091 delta-find: move pre-filtering of candidates in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51344 diff changeset	896
02cc19f4f091 delta-find: move pre-filtering of candidates in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51344 diff changeset	897 This function use various criteria to pre-filter candidate delta base
02cc19f4f091 delta-find: move pre-filtering of candidates in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51344 diff changeset	898 before we compute a delta and evaluate its quality.
02cc19f4f091 delta-find: move pre-filtering of candidates in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51344 diff changeset	899
02cc19f4f091 delta-find: move pre-filtering of candidates in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51344 diff changeset	900 Such pre-filter limit the number of computed delta, an expensive operation.
02cc19f4f091 delta-find: move pre-filtering of candidates in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51344 diff changeset	901
02cc19f4f091 delta-find: move pre-filtering of candidates in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51344 diff changeset	902 return the updated list of revision to test
02cc19f4f091 delta-find: move pre-filtering of candidates in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51344 diff changeset	903 """
02cc19f4f091 delta-find: move pre-filtering of candidates in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51344 diff changeset	904 deltalength = self.revlog.length
02cc19f4f091 delta-find: move pre-filtering of candidates in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51344 diff changeset	905 deltaparent = self.revlog.deltaparent
02cc19f4f091 delta-find: move pre-filtering of candidates in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51344 diff changeset	906
02cc19f4f091 delta-find: move pre-filtering of candidates in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51344 diff changeset	907 tested = self.tested
02cc19f4f091 delta-find: move pre-filtering of candidates in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51344 diff changeset	908 group = []
02cc19f4f091 delta-find: move pre-filtering of candidates in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51344 diff changeset	909 for rev in temptative:
02cc19f4f091 delta-find: move pre-filtering of candidates in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51344 diff changeset	910 # skip over empty delta (no need to include them in a chain)
02cc19f4f091 delta-find: move pre-filtering of candidates in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51344 diff changeset	911 while not (rev == nullrev or rev in tested or deltalength(rev)):
02cc19f4f091 delta-find: move pre-filtering of candidates in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51344 diff changeset	912 tested.add(rev)
02cc19f4f091 delta-find: move pre-filtering of candidates in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51344 diff changeset	913 rev = deltaparent(rev)
51346 898c212e1b2f delta-find: move pre-filtering of individual revision in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51345 diff changeset	914 if self._pre_filter_rev(rev):
898c212e1b2f delta-find: move pre-filtering of individual revision in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51345 diff changeset	915 group.append(rev)
898c212e1b2f delta-find: move pre-filtering of individual revision in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51345 diff changeset	916 else:
898c212e1b2f delta-find: move pre-filtering of individual revision in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51345 diff changeset	917 self.tested.add(rev)
898c212e1b2f delta-find: move pre-filtering of individual revision in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51345 diff changeset	918 return group
898c212e1b2f delta-find: move pre-filtering of individual revision in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51345 diff changeset	919
51348 2407af4f2eea delta-find: split the generic part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51347 diff changeset	920 def _pre_filter_rev_universal(self, rev):
2407af4f2eea delta-find: split the generic part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51347 diff changeset	921 """pre filtering that is need in all cases.
2407af4f2eea delta-find: split the generic part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51347 diff changeset	922
2407af4f2eea delta-find: split the generic part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51347 diff changeset	923 return True if it seems okay to test a rev, False otherwise.
2407af4f2eea delta-find: split the generic part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51347 diff changeset	924
2407af4f2eea delta-find: split the generic part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51347 diff changeset	925 used by _pre_filter_rev.
2407af4f2eea delta-find: split the generic part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51347 diff changeset	926 """
51347 ac8b798e602b delta-find: drop the temporary indent Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51346 diff changeset	927 # no need to try a delta against nullrev, this will be done as
ac8b798e602b delta-find: drop the temporary indent Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51346 diff changeset	928 # a last resort.
ac8b798e602b delta-find: drop the temporary indent Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51346 diff changeset	929 if rev == nullrev:
ac8b798e602b delta-find: drop the temporary indent Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51346 diff changeset	930 return False
ac8b798e602b delta-find: drop the temporary indent Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51346 diff changeset	931 # filter out revision we tested already
ac8b798e602b delta-find: drop the temporary indent Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51346 diff changeset	932 if rev in self.tested:
ac8b798e602b delta-find: drop the temporary indent Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51346 diff changeset	933 return False
51345 02cc19f4f091 delta-find: move pre-filtering of candidates in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51344 diff changeset	934
51347 ac8b798e602b delta-find: drop the temporary indent Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51346 diff changeset	935 # an higher authority deamed the base unworthy (e.g. censored)
ac8b798e602b delta-find: drop the temporary indent Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51346 diff changeset	936 if self.excluded_bases is not None and rev in self.excluded_bases:
ac8b798e602b delta-find: drop the temporary indent Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51346 diff changeset	937 return False
ac8b798e602b delta-find: drop the temporary indent Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51346 diff changeset	938 # We are in some recomputation cases and that rev is too high
ac8b798e602b delta-find: drop the temporary indent Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51346 diff changeset	939 # in the revlog
ac8b798e602b delta-find: drop the temporary indent Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51346 diff changeset	940 if self.target_rev is not None and rev >= self.target_rev:
ac8b798e602b delta-find: drop the temporary indent Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51346 diff changeset	941 return False
51348 2407af4f2eea delta-find: split the generic part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51347 diff changeset	942 # no delta for rawtext-changing revs (see "candelta" for why)
2407af4f2eea delta-find: split the generic part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51347 diff changeset	943 if self.revlog.flags(rev) & REVIDX_RAWTEXT_CHANGING_FLAGS:
2407af4f2eea delta-find: split the generic part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51347 diff changeset	944 return False
2407af4f2eea delta-find: split the generic part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51347 diff changeset	945 return True
2407af4f2eea delta-find: split the generic part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51347 diff changeset	946
51350 1ea56b10dd4a delta-find: split the delta-chain part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51349 diff changeset	947 def _pre_filter_rev_delta_chain(self, rev):
1ea56b10dd4a delta-find: split the delta-chain part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51349 diff changeset	948 """pre filtering that is needed in sparse revlog cases
51349 9a1239c362ae delta-find: split the "sparse" part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51348 diff changeset	949
51350 1ea56b10dd4a delta-find: split the delta-chain part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51349 diff changeset	950 return True if it seems okay to test a rev, False otherwise.
1ea56b10dd4a delta-find: split the delta-chain part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51349 diff changeset	951
1ea56b10dd4a delta-find: split the delta-chain part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51349 diff changeset	952 used by _pre_filter_rev.
1ea56b10dd4a delta-find: split the delta-chain part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51349 diff changeset	953 """
51349 9a1239c362ae delta-find: split the "sparse" part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51348 diff changeset	954 deltas_limit = self.revinfo.textlen * LIMIT_DELTA2TEXT
9a1239c362ae delta-find: split the "sparse" part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51348 diff changeset	955 # filter out delta base that will never produce good delta
9a1239c362ae delta-find: split the "sparse" part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51348 diff changeset	956 #
9a1239c362ae delta-find: split the "sparse" part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51348 diff changeset	957 # if the delta of that base is already bigger than the limit
9a1239c362ae delta-find: split the "sparse" part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51348 diff changeset	958 # for the delta chain size, doing a delta is hopeless.
9a1239c362ae delta-find: split the "sparse" part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51348 diff changeset	959 if deltas_limit < self.revlog.length(rev):
9a1239c362ae delta-find: split the "sparse" part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51348 diff changeset	960 return False
9a1239c362ae delta-find: split the "sparse" part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51348 diff changeset	961
9a1239c362ae delta-find: split the "sparse" part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51348 diff changeset	962 # If we reach here, we are about to build and test a delta.
9a1239c362ae delta-find: split the "sparse" part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51348 diff changeset	963 # The delta building process will compute the chaininfo in all
9a1239c362ae delta-find: split the "sparse" part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51348 diff changeset	964 # case, since that computation is cached, it is fine to access
9a1239c362ae delta-find: split the "sparse" part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51348 diff changeset	965 # it here too.
9a1239c362ae delta-find: split the "sparse" part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51348 diff changeset	966 chainlen, chainsize = self.revlog._chaininfo(rev)
9a1239c362ae delta-find: split the "sparse" part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51348 diff changeset	967 # if chain will be too long, skip base
9a1239c362ae delta-find: split the "sparse" part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51348 diff changeset	968 if (
9a1239c362ae delta-find: split the "sparse" part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51348 diff changeset	969 self.revlog.delta_config.max_chain_len
9a1239c362ae delta-find: split the "sparse" part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51348 diff changeset	970 and chainlen >= self.revlog.delta_config.max_chain_len
9a1239c362ae delta-find: split the "sparse" part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51348 diff changeset	971 ):
9a1239c362ae delta-find: split the "sparse" part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51348 diff changeset	972 return False
9a1239c362ae delta-find: split the "sparse" part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51348 diff changeset	973 # if chain already have too much data, skip base
9a1239c362ae delta-find: split the "sparse" part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51348 diff changeset	974 if deltas_limit < chainsize:
9a1239c362ae delta-find: split the "sparse" part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51348 diff changeset	975 return False
51350 1ea56b10dd4a delta-find: split the delta-chain part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51349 diff changeset	976 return True
51349 9a1239c362ae delta-find: split the "sparse" part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51348 diff changeset	977
51350 1ea56b10dd4a delta-find: split the delta-chain part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51349 diff changeset	978 def _pre_filter_rev(self, rev):
1ea56b10dd4a delta-find: split the delta-chain part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51349 diff changeset	979 """return True if it seems okay to test a rev, False otherwise"""
1ea56b10dd4a delta-find: split the delta-chain part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51349 diff changeset	980 if not self._pre_filter_rev_universal(rev):
1ea56b10dd4a delta-find: split the delta-chain part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51349 diff changeset	981 return False
1ea56b10dd4a delta-find: split the delta-chain part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51349 diff changeset	982 if not self._pre_filter_rev_delta_chain(rev):
1ea56b10dd4a delta-find: split the delta-chain part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51349 diff changeset	983 return False
51349 9a1239c362ae delta-find: split the "sparse" part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51348 diff changeset	984 return True
9a1239c362ae delta-find: split the "sparse" part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51348 diff changeset	985
51363 5cc04a6da19d delta-find: move the emotion of parents in a dedicated method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51362 diff changeset	986 def _iter_parents(self):
5cc04a6da19d delta-find: move the emotion of parents in a dedicated method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51362 diff changeset	987 # exclude already lazy tested base if any
5cc04a6da19d delta-find: move the emotion of parents in a dedicated method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51362 diff changeset	988 parents = [p for p in (self.p1, self.p2) if p != nullrev]
5cc04a6da19d delta-find: move the emotion of parents in a dedicated method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51362 diff changeset	989
5cc04a6da19d delta-find: move the emotion of parents in a dedicated method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51362 diff changeset	990 self.current_stage = _STAGE_PARENTS
5cc04a6da19d delta-find: move the emotion of parents in a dedicated method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51362 diff changeset	991 if (
5cc04a6da19d delta-find: move the emotion of parents in a dedicated method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51362 diff changeset	992 not self.revlog.delta_config.delta_both_parents
5cc04a6da19d delta-find: move the emotion of parents in a dedicated method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51362 diff changeset	993 and len(parents) == 2
5cc04a6da19d delta-find: move the emotion of parents in a dedicated method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51362 diff changeset	994 ):
5cc04a6da19d delta-find: move the emotion of parents in a dedicated method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51362 diff changeset	995 parents.sort()
5cc04a6da19d delta-find: move the emotion of parents in a dedicated method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51362 diff changeset	996 # To minimize the chance of having to build a fulltext,
5cc04a6da19d delta-find: move the emotion of parents in a dedicated method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51362 diff changeset	997 # pick first whichever parent is closest to us (max rev)
5cc04a6da19d delta-find: move the emotion of parents in a dedicated method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51362 diff changeset	998 yield (parents[1],)
5cc04a6da19d delta-find: move the emotion of parents in a dedicated method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51362 diff changeset	999 # then the other one (min rev) if the first did not fit
5cc04a6da19d delta-find: move the emotion of parents in a dedicated method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51362 diff changeset	1000 yield (parents[0],)
5cc04a6da19d delta-find: move the emotion of parents in a dedicated method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51362 diff changeset	1001 elif len(parents) > 0:
5cc04a6da19d delta-find: move the emotion of parents in a dedicated method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51362 diff changeset	1002 # Test all parents (1 or 2), and keep the best candidate
5cc04a6da19d delta-find: move the emotion of parents in a dedicated method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51362 diff changeset	1003 yield parents
5cc04a6da19d delta-find: move the emotion of parents in a dedicated method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51362 diff changeset	1004
51364 94fe4474b053 delta-find: move the emotion of prev in a dedicated method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51363 diff changeset	1005 def _iter_prev(self):
94fe4474b053 delta-find: move the emotion of prev in a dedicated method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51363 diff changeset	1006 # other approach failed try against prev to hopefully save us a
94fe4474b053 delta-find: move the emotion of prev in a dedicated method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51363 diff changeset	1007 # fulltext.
94fe4474b053 delta-find: move the emotion of prev in a dedicated method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51363 diff changeset	1008 self.current_stage = _STAGE_PREV
94fe4474b053 delta-find: move the emotion of prev in a dedicated method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51363 diff changeset	1009 yield (self.target_rev - 1,)
94fe4474b053 delta-find: move the emotion of prev in a dedicated method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51363 diff changeset	1010
51367 d7e2acdd50ba delta-find: split the _DeltaSearch class in two Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51366 diff changeset	1011 def _iter_groups(self):
d7e2acdd50ba delta-find: split the _DeltaSearch class in two Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51366 diff changeset	1012 good = None
d7e2acdd50ba delta-find: split the _DeltaSearch class in two Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51366 diff changeset	1013 for group in self._iter_parents():
d7e2acdd50ba delta-find: split the _DeltaSearch class in two Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51366 diff changeset	1014 good = yield group
d7e2acdd50ba delta-find: split the _DeltaSearch class in two Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51366 diff changeset	1015 if good is not None:
d7e2acdd50ba delta-find: split the _DeltaSearch class in two Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51366 diff changeset	1016 break
d7e2acdd50ba delta-find: split the _DeltaSearch class in two Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51366 diff changeset	1017 else:
d7e2acdd50ba delta-find: split the _DeltaSearch class in two Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51366 diff changeset	1018 assert good is None
d7e2acdd50ba delta-find: split the _DeltaSearch class in two Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51366 diff changeset	1019 yield from self._iter_prev()
d7e2acdd50ba delta-find: split the _DeltaSearch class in two Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51366 diff changeset	1020 yield None
d7e2acdd50ba delta-find: split the _DeltaSearch class in two Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51366 diff changeset	1021
d7e2acdd50ba delta-find: split the _DeltaSearch class in two Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51366 diff changeset	1022
d7e2acdd50ba delta-find: split the _DeltaSearch class in two Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51366 diff changeset	1023 class _SparseDeltaSearch(_GeneralDeltaSearch):
d7e2acdd50ba delta-find: split the _DeltaSearch class in two Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51366 diff changeset	1024 """Delta search variants for sparse-revlog"""
d7e2acdd50ba delta-find: split the _DeltaSearch class in two Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51366 diff changeset	1025
51368 383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51367 diff changeset	1026 def is_good_delta_info(self, deltainfo):
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51367 diff changeset	1027 """Returns True if the given delta is good.
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51367 diff changeset	1028
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51367 diff changeset	1029 Good means that it is within the disk span, disk size, and chain length
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51367 diff changeset	1030 bounds that we know to be performant.
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51367 diff changeset	1031 """
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51367 diff changeset	1032 if not self._is_good_delta_info_universal(deltainfo):
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51367 diff changeset	1033 return False
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51367 diff changeset	1034 if not self._is_good_delta_info_chain_quality(deltainfo):
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51367 diff changeset	1035 return False
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51367 diff changeset	1036 if not self._is_good_delta_info_snapshot_constraints(deltainfo):
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51367 diff changeset	1037 return False
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51367 diff changeset	1038 return True
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51367 diff changeset	1039
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51367 diff changeset	1040 def _is_good_delta_info_snapshot_constraints(self, deltainfo):
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51367 diff changeset	1041 """Returns True if the chain associated with snapshots
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51367 diff changeset	1042
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51367 diff changeset	1043 This performs checks for format that use sparse-revlog and intermediate
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51367 diff changeset	1044 snapshots.
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51367 diff changeset	1045
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51367 diff changeset	1046 This is used by is_good_delta_info.
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51367 diff changeset	1047 """
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51367 diff changeset	1048 # if not a snapshot, this method has no filtering to do
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51367 diff changeset	1049 if deltainfo.snapshotdepth is None:
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51367 diff changeset	1050 return True
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51367 diff changeset	1051 # bad delta from intermediate snapshot size limit
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51367 diff changeset	1052 #
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51367 diff changeset	1053 # If an intermediate snapshot size is higher than the limit. The
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51367 diff changeset	1054 # limit exist to prevent endless chain of intermediate delta to be
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51367 diff changeset	1055 # created.
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51367 diff changeset	1056 if (
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51367 diff changeset	1057 self.revinfo.textlen >> deltainfo.snapshotdepth
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51367 diff changeset	1058 ) < deltainfo.deltalen:
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51367 diff changeset	1059 return False
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51367 diff changeset	1060
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51367 diff changeset	1061 # bad delta if new intermediate snapshot is larger than the previous
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51367 diff changeset	1062 # snapshot
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51367 diff changeset	1063 if self.revlog.length(deltainfo.base) < deltainfo.deltalen:
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51367 diff changeset	1064 return False
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51367 diff changeset	1065
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51367 diff changeset	1066 return True
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51367 diff changeset	1067
51369 a224ce5694b3 delta-find: move sparse-revlog pre-filtering in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51368 diff changeset	1068 def _pre_filter_rev(self, rev):
a224ce5694b3 delta-find: move sparse-revlog pre-filtering in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51368 diff changeset	1069 """return True if it seems okay to test a rev, False otherwise"""
a224ce5694b3 delta-find: move sparse-revlog pre-filtering in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51368 diff changeset	1070 if not self._pre_filter_rev_universal(rev):
a224ce5694b3 delta-find: move sparse-revlog pre-filtering in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51368 diff changeset	1071 return False
a224ce5694b3 delta-find: move sparse-revlog pre-filtering in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51368 diff changeset	1072 if not self._pre_filter_rev_delta_chain(rev):
a224ce5694b3 delta-find: move sparse-revlog pre-filtering in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51368 diff changeset	1073 return False
a224ce5694b3 delta-find: move sparse-revlog pre-filtering in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51368 diff changeset	1074 if not self._pre_filter_rev_sparse(rev):
a224ce5694b3 delta-find: move sparse-revlog pre-filtering in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51368 diff changeset	1075 return False
a224ce5694b3 delta-find: move sparse-revlog pre-filtering in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51368 diff changeset	1076 return True
a224ce5694b3 delta-find: move sparse-revlog pre-filtering in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51368 diff changeset	1077
a224ce5694b3 delta-find: move sparse-revlog pre-filtering in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51368 diff changeset	1078 def _pre_filter_rev_sparse(self, rev):
a224ce5694b3 delta-find: move sparse-revlog pre-filtering in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51368 diff changeset	1079 """pre filtering that is needed in sparse revlog cases
a224ce5694b3 delta-find: move sparse-revlog pre-filtering in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51368 diff changeset	1080
a224ce5694b3 delta-find: move sparse-revlog pre-filtering in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51368 diff changeset	1081 return True if it seems okay to test a rev, False otherwise.
a224ce5694b3 delta-find: move sparse-revlog pre-filtering in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51368 diff changeset	1082
a224ce5694b3 delta-find: move sparse-revlog pre-filtering in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51368 diff changeset	1083 used by _pre_filter_rev.
a224ce5694b3 delta-find: move sparse-revlog pre-filtering in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51368 diff changeset	1084 """
a224ce5694b3 delta-find: move sparse-revlog pre-filtering in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51368 diff changeset	1085 assert self.revlog.delta_config.sparse_revlog
a224ce5694b3 delta-find: move sparse-revlog pre-filtering in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51368 diff changeset	1086 # if the revision we test again is too small, the resulting delta
a224ce5694b3 delta-find: move sparse-revlog pre-filtering in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51368 diff changeset	1087 # will be large anyway as that amount of data to be added is big
a224ce5694b3 delta-find: move sparse-revlog pre-filtering in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51368 diff changeset	1088 if self.revlog.rawsize(rev) < (self.textlen // LIMIT_BASE2TEXT):
a224ce5694b3 delta-find: move sparse-revlog pre-filtering in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51368 diff changeset	1089 return False
a224ce5694b3 delta-find: move sparse-revlog pre-filtering in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51368 diff changeset	1090
a224ce5694b3 delta-find: move sparse-revlog pre-filtering in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51368 diff changeset	1091 if self.revlog.delta_config.upper_bound_comp is not None:
a224ce5694b3 delta-find: move sparse-revlog pre-filtering in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51368 diff changeset	1092 maxcomp = self.revlog.delta_config.upper_bound_comp
a224ce5694b3 delta-find: move sparse-revlog pre-filtering in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51368 diff changeset	1093 basenotsnap = (self.p1, self.p2, nullrev)
a224ce5694b3 delta-find: move sparse-revlog pre-filtering in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51368 diff changeset	1094 if rev not in basenotsnap and self.revlog.issnapshot(rev):
a224ce5694b3 delta-find: move sparse-revlog pre-filtering in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51368 diff changeset	1095 snapshotdepth = self.revlog.snapshotdepth(rev)
a224ce5694b3 delta-find: move sparse-revlog pre-filtering in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51368 diff changeset	1096 # If text is significantly larger than the base, we can
a224ce5694b3 delta-find: move sparse-revlog pre-filtering in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51368 diff changeset	1097 # expect the resulting delta to be proportional to the size
a224ce5694b3 delta-find: move sparse-revlog pre-filtering in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51368 diff changeset	1098 # difference
a224ce5694b3 delta-find: move sparse-revlog pre-filtering in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51368 diff changeset	1099 revsize = self.revlog.rawsize(rev)
a224ce5694b3 delta-find: move sparse-revlog pre-filtering in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51368 diff changeset	1100 rawsizedistance = max(self.textlen - revsize, 0)
a224ce5694b3 delta-find: move sparse-revlog pre-filtering in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51368 diff changeset	1101 # use an estimate of the compression upper bound.
a224ce5694b3 delta-find: move sparse-revlog pre-filtering in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51368 diff changeset	1102 lowestrealisticdeltalen = rawsizedistance // maxcomp
a224ce5694b3 delta-find: move sparse-revlog pre-filtering in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51368 diff changeset	1103
a224ce5694b3 delta-find: move sparse-revlog pre-filtering in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51368 diff changeset	1104 # check the absolute constraint on the delta size
a224ce5694b3 delta-find: move sparse-revlog pre-filtering in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51368 diff changeset	1105 snapshotlimit = self.textlen >> snapshotdepth
a224ce5694b3 delta-find: move sparse-revlog pre-filtering in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51368 diff changeset	1106 if snapshotlimit < lowestrealisticdeltalen:
a224ce5694b3 delta-find: move sparse-revlog pre-filtering in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51368 diff changeset	1107 # delta lower bound is larger than accepted upper
a224ce5694b3 delta-find: move sparse-revlog pre-filtering in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51368 diff changeset	1108 # bound
a224ce5694b3 delta-find: move sparse-revlog pre-filtering in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51368 diff changeset	1109 return False
a224ce5694b3 delta-find: move sparse-revlog pre-filtering in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51368 diff changeset	1110
a224ce5694b3 delta-find: move sparse-revlog pre-filtering in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51368 diff changeset	1111 # check the relative constraint on the delta size
a224ce5694b3 delta-find: move sparse-revlog pre-filtering in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51368 diff changeset	1112 revlength = self.revlog.length(rev)
a224ce5694b3 delta-find: move sparse-revlog pre-filtering in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51368 diff changeset	1113 if revlength < lowestrealisticdeltalen:
a224ce5694b3 delta-find: move sparse-revlog pre-filtering in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51368 diff changeset	1114 # delta probable lower bound is larger than target
a224ce5694b3 delta-find: move sparse-revlog pre-filtering in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51368 diff changeset	1115 # base
a224ce5694b3 delta-find: move sparse-revlog pre-filtering in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51368 diff changeset	1116 return False
a224ce5694b3 delta-find: move sparse-revlog pre-filtering in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51368 diff changeset	1117 return True
a224ce5694b3 delta-find: move sparse-revlog pre-filtering in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51368 diff changeset	1118
51365 fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1119 def _iter_snapshots_base(self):
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1120 assert self.revlog.delta_config.sparse_revlog
51366 701caeabbee7 delta-find: finish reworking the snapshot logic and drop more layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51365 diff changeset	1121 assert self.current_stage == _STAGE_SNAPSHOT
51365 fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1122 prev = self.target_rev - 1
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1123 deltachain = lambda rev: self.revlog._deltachain(rev)[0]
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1124
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1125 parents = [p for p in (self.p1, self.p2) if p != nullrev]
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1126 if not parents:
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1127 return
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1128 # See if we can use an existing snapshot in the parent chains to
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1129 # use as a base for a new intermediate-snapshot
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1130 #
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1131 # search for snapshot in parents delta chain map: snapshot-level:
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1132 # snapshot-rev
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1133 parents_snaps = collections.defaultdict(set)
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1134 candidate_chains = [deltachain(p) for p in parents]
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1135 for chain in candidate_chains:
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1136 for idx, s in enumerate(chain):
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1137 if not self.revlog.issnapshot(s):
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1138 break
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1139 parents_snaps[idx].add(s)
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1140 snapfloor = min(parents_snaps[0]) + 1
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1141 self.snapshot_cache.update(self.revlog, snapfloor)
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1142 # search for the highest "unrelated" revision
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1143 #
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1144 # Adding snapshots used by "unrelated" revision increase the odd we
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1145 # reuse an independant, yet better snapshot chain.
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1146 #
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1147 # XXX instead of building a set of revisions, we could lazily
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1148 # enumerate over the chains. That would be more efficient, however
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1149 # we stick to simple code for now.
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1150 all_revs = set()
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1151 for chain in candidate_chains:
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1152 all_revs.update(chain)
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1153 other = None
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1154 for r in self.revlog.revs(prev, snapfloor):
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1155 if r not in all_revs:
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1156 other = r
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1157 break
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1158 if other is not None:
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1159 # To avoid unfair competition, we won't use unrelated
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1160 # intermediate snapshot that are deeper than the ones from the
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1161 # parent delta chain.
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1162 max_depth = max(parents_snaps.keys())
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1163 chain = deltachain(other)
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1164 for depth, s in enumerate(chain):
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1165 if s < snapfloor:
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1166 continue
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1167 if max_depth < depth:
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1168 break
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1169 if not self.revlog.issnapshot(s):
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1170 break
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1171 parents_snaps[depth].add(s)
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1172 # Test them as possible intermediate snapshot base We test them
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1173 # from highest to lowest level. High level one are more likely to
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1174 # result in small delta
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1175 floor = None
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1176 for idx, snaps in sorted(parents_snaps.items(), reverse=True):
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1177 siblings = set()
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1178 for s in snaps:
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1179 siblings.update(self.snapshot_cache.snapshots[s])
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1180 # Before considering making a new intermediate snapshot, we
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1181 # check if an existing snapshot, children of base we consider,
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1182 # would be suitable.
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1183 #
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1184 # It give a change to reuse a delta chain "unrelated" to the
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1185 # current revision instead of starting our own. Without such
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1186 # re-use, topological branches would keep reopening new chains.
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1187 # Creating more and more snapshot as the repository grow.
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1188
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1189 if floor is not None:
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1190 # We only do this for siblings created after the one in our
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1191 # parent's delta chain. Those created before has less
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1192 # chances to be valid base since our ancestors had to
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1193 # create a new snapshot.
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1194 siblings = [r for r in siblings if floor < r]
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1195 yield tuple(sorted(siblings))
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1196 # then test the base from our parent's delta chain.
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1197 yield tuple(sorted(snaps))
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1198 floor = min(snaps)
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1199 # No suitable base found in the parent chain, search if any full
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1200 # snapshots emitted since parent's base would be a suitable base
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1201 # for an intermediate snapshot.
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1202 #
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1203 # It give a chance to reuse a delta chain unrelated to the current
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1204 # revisions instead of starting our own. Without such re-use,
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1205 # topological branches would keep reopening new full chains.
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1206 # Creating more and more snapshot as the repository grow.
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1207 full = [
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1208 r for r in self.snapshot_cache.snapshots[nullrev] if snapfloor <= r
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1209 ]
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1210 yield tuple(sorted(full))
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1211
51366 701caeabbee7 delta-find: finish reworking the snapshot logic and drop more layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51365 diff changeset	1212 def _iter_snapshots(self):
701caeabbee7 delta-find: finish reworking the snapshot logic and drop more layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51365 diff changeset	1213 assert self.revlog.delta_config.sparse_revlog
701caeabbee7 delta-find: finish reworking the snapshot logic and drop more layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51365 diff changeset	1214 self.current_stage = _STAGE_SNAPSHOT
51331 c83074405276 delta-find: move `_refinedgroups` on the `_DeltaSearch` object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51330 diff changeset	1215 good = None
51366 701caeabbee7 delta-find: finish reworking the snapshot logic and drop more layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51365 diff changeset	1216 groups = self._iter_snapshots_base()
51331 c83074405276 delta-find: move `_refinedgroups` on the `_DeltaSearch` object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51330 diff changeset	1217 for candidates in groups:
c83074405276 delta-find: move `_refinedgroups` on the `_DeltaSearch` object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51330 diff changeset	1218 good = yield candidates
c83074405276 delta-find: move `_refinedgroups` on the `_DeltaSearch` object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51330 diff changeset	1219 if good is not None:
c83074405276 delta-find: move `_refinedgroups` on the `_DeltaSearch` object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51330 diff changeset	1220 break
c83074405276 delta-find: move `_refinedgroups` on the `_DeltaSearch` object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51330 diff changeset	1221 # if we have a refinable value, try to refine it
51370 99869dcf3ba0 delta-find: pass the full deltainfo to the _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51369 diff changeset	1222 if good is not None and good.snapshotdepth is not None:
51360 670e68729aa7 delta-find: stop using heuristic to determine if we are creating a snapshot Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51359 diff changeset	1223 assert self.current_stage == _STAGE_SNAPSHOT
51331 c83074405276 delta-find: move `_refinedgroups` on the `_DeltaSearch` object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51330 diff changeset	1224 # refine snapshot down
c83074405276 delta-find: move `_refinedgroups` on the `_DeltaSearch` object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51330 diff changeset	1225 previous = None
c83074405276 delta-find: move `_refinedgroups` on the `_DeltaSearch` object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51330 diff changeset	1226 while previous != good:
c83074405276 delta-find: move `_refinedgroups` on the `_DeltaSearch` object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51330 diff changeset	1227 previous = good
51370 99869dcf3ba0 delta-find: pass the full deltainfo to the _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51369 diff changeset	1228 base = self.revlog.deltaparent(good.base)
51331 c83074405276 delta-find: move `_refinedgroups` on the `_DeltaSearch` object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51330 diff changeset	1229 if base == nullrev:
c83074405276 delta-find: move `_refinedgroups` on the `_DeltaSearch` object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51330 diff changeset	1230 break
c83074405276 delta-find: move `_refinedgroups` on the `_DeltaSearch` object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51330 diff changeset	1231 good = yield (base,)
c83074405276 delta-find: move `_refinedgroups` on the `_DeltaSearch` object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51330 diff changeset	1232 # refine snapshot up
c83074405276 delta-find: move `_refinedgroups` on the `_DeltaSearch` object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51330 diff changeset	1233 if not self.snapshot_cache.snapshots:
51370 99869dcf3ba0 delta-find: pass the full deltainfo to the _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51369 diff changeset	1234 self.snapshot_cache.update(self.revlog, good.base + 1)
51331 c83074405276 delta-find: move `_refinedgroups` on the `_DeltaSearch` object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51330 diff changeset	1235 previous = None
c83074405276 delta-find: move `_refinedgroups` on the `_DeltaSearch` object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51330 diff changeset	1236 while good != previous:
c83074405276 delta-find: move `_refinedgroups` on the `_DeltaSearch` object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51330 diff changeset	1237 previous = good
c83074405276 delta-find: move `_refinedgroups` on the `_DeltaSearch` object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51330 diff changeset	1238 children = tuple(
51370 99869dcf3ba0 delta-find: pass the full deltainfo to the _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51369 diff changeset	1239 sorted(c for c in self.snapshot_cache.snapshots[good.base])
51331 c83074405276 delta-find: move `_refinedgroups` on the `_DeltaSearch` object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51330 diff changeset	1240 )
c83074405276 delta-find: move `_refinedgroups` on the `_DeltaSearch` object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51330 diff changeset	1241 good = yield children
c83074405276 delta-find: move `_refinedgroups` on the `_DeltaSearch` object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51330 diff changeset	1242 yield None
39513 2f9f7889549b snapshot: introduce an intermediate `_refinedgroups` generator Boris Feld <boris.feld@octobus.net> parents: 39512 diff changeset	1243
51366 701caeabbee7 delta-find: finish reworking the snapshot logic and drop more layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51365 diff changeset	1244 def _iter_groups(self):
701caeabbee7 delta-find: finish reworking the snapshot logic and drop more layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51365 diff changeset	1245 good = None
701caeabbee7 delta-find: finish reworking the snapshot logic and drop more layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51365 diff changeset	1246 for group in self._iter_parents():
701caeabbee7 delta-find: finish reworking the snapshot logic and drop more layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51365 diff changeset	1247 good = yield group
701caeabbee7 delta-find: finish reworking the snapshot logic and drop more layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51365 diff changeset	1248 if good is not None:
701caeabbee7 delta-find: finish reworking the snapshot logic and drop more layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51365 diff changeset	1249 break
51365 fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51364 diff changeset	1250 else:
51366 701caeabbee7 delta-find: finish reworking the snapshot logic and drop more layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51365 diff changeset	1251 assert good is None
51367 d7e2acdd50ba delta-find: split the _DeltaSearch class in two Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51366 diff changeset	1252 assert self.revlog.delta_config.sparse_revlog
d7e2acdd50ba delta-find: split the _DeltaSearch class in two Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51366 diff changeset	1253 # If sparse revlog is enabled, we can try to refine the
d7e2acdd50ba delta-find: split the _DeltaSearch class in two Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51366 diff changeset	1254 # available deltas
d7e2acdd50ba delta-find: split the _DeltaSearch class in two Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51366 diff changeset	1255 iter_snap = self._iter_snapshots()
d7e2acdd50ba delta-find: split the _DeltaSearch class in two Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51366 diff changeset	1256 group = iter_snap.send(None)
d7e2acdd50ba delta-find: split the _DeltaSearch class in two Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51366 diff changeset	1257 while group is not None:
d7e2acdd50ba delta-find: split the _DeltaSearch class in two Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51366 diff changeset	1258 good = yield group
d7e2acdd50ba delta-find: split the _DeltaSearch class in two Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51366 diff changeset	1259 group = iter_snap.send(good)
51366 701caeabbee7 delta-find: finish reworking the snapshot logic and drop more layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51365 diff changeset	1260 yield None
39361 507f5b1dd7c8 revlogdeltas: extract _getcandidaterevs in a function Boris Feld <boris.feld@octobus.net> parents: 39360 diff changeset	1261
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1262
49792 efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1263 class SnapshotCache:
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1264 __slots__ = ('snapshots', '_start_rev', '_end_rev')
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1265
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1266 def __init__(self):
49793 b670eb3dd6c9 delta-find: use sets instead of list in the snapshot cache Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49792 diff changeset	1267 self.snapshots = collections.defaultdict(set)
49792 efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1268 self._start_rev = None
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1269 self._end_rev = None
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1270
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1271 def update(self, revlog, start_rev=0):
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1272 """find snapshots from start_rev to tip"""
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1273 nb_revs = len(revlog)
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1274 end_rev = nb_revs - 1
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1275 if start_rev > end_rev:
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1276 return # range is empty
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1277
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1278 if self._start_rev is None:
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1279 assert self._end_rev is None
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1280 self._update(revlog, start_rev, end_rev)
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1281 elif not (self._start_rev <= start_rev and end_rev <= self._end_rev):
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1282 if start_rev < self._start_rev:
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1283 self._update(revlog, start_rev, self._start_rev - 1)
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1284 if self._end_rev < end_rev:
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1285 self._update(revlog, self._end_rev + 1, end_rev)
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1286
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1287 if self._start_rev is None:
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1288 assert self._end_rev is None
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1289 self._end_rev = end_rev
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1290 self._start_rev = start_rev
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1291 else:
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1292 self._start_rev = min(self._start_rev, start_rev)
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1293 self._end_rev = max(self._end_rev, end_rev)
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1294 assert self._start_rev <= self._end_rev, (
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1295 self._start_rev,
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1296 self._end_rev,
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1297 )
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1298
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1299 def _update(self, revlog, start_rev, end_rev):
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1300 """internal method that actually do update content"""
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1301 assert self._start_rev is None or (
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1302 start_rev < self._start_rev or start_rev > self._end_rev
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1303 ), (self._start_rev, self._end_rev, start_rev, end_rev)
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1304 assert self._start_rev is None or (
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1305 end_rev < self._start_rev or end_rev > self._end_rev
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1306 ), (self._start_rev, self._end_rev, start_rev, end_rev)
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1307 cache = self.snapshots
50951 d718eddf01d9 safehasattr: drop usage in favor of hasattr Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50698 diff changeset	1308 if hasattr(revlog.index, 'findsnapshots'):
49792 efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1309 revlog.index.findsnapshots(cache, start_rev, end_rev)
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1310 else:
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1311 deltaparent = revlog.deltaparent
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1312 issnapshot = revlog.issnapshot
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1313 for rev in revlog.revs(start_rev, end_rev):
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1314 if issnapshot(rev):
49793 b670eb3dd6c9 delta-find: use sets instead of list in the snapshot cache Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49792 diff changeset	1315 cache[deltaparent(rev)].add(rev)
49792 efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1316
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49791 diff changeset	1317
49037 642e31cb55f0 py3: use class X: instead of class X(object): Gregory Szorc <gregory.szorc@gmail.com> parents: 48966 diff changeset	1318 class deltacomputer:
51329 da7ecb4deaec delta-find: add a small docstring to deltacomputer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51095 diff changeset	1319 """object capable of computing delta and finding delta for multiple revision
da7ecb4deaec delta-find: add a small docstring to deltacomputer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51095 diff changeset	1320
da7ecb4deaec delta-find: add a small docstring to deltacomputer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51095 diff changeset	1321 This object is meant to compute and find multiple delta applied to the same
da7ecb4deaec delta-find: add a small docstring to deltacomputer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51095 diff changeset	1322 revlog.
da7ecb4deaec delta-find: add a small docstring to deltacomputer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51095 diff changeset	1323 """
da7ecb4deaec delta-find: add a small docstring to deltacomputer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51095 diff changeset	1324
49739 35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	1325 def __init__(
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	1326 self,
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	1327 revlog,
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	1328 write_debug=None,
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	1329 debug_search=False,
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	1330 debug_info=None,
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	1331 ):
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1332 self.revlog = revlog
49247 e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	1333 self._write_debug = write_debug
50378 c84cc0ac77e4 delta-fine: use the `_debug_search` attribute directly Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50377 diff changeset	1334 if write_debug is None:
c84cc0ac77e4 delta-fine: use the `_debug_search` attribute directly Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50377 diff changeset	1335 self._debug_search = False
c84cc0ac77e4 delta-fine: use the `_debug_search` attribute directly Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50377 diff changeset	1336 else:
c84cc0ac77e4 delta-fine: use the `_debug_search` attribute directly Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50377 diff changeset	1337 self._debug_search = debug_search
49739 35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	1338 self._debug_info = debug_info
49795 c261a628e525 delta-find: use a single snapshot cache when applying a group to an object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49794 diff changeset	1339 self._snapshot_cache = SnapshotCache()
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1340
50377 5d210ff4b657 delta-find: move the `gather_debug` logic in a property Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50357 diff changeset	1341 @property
5d210ff4b657 delta-find: move the `gather_debug` logic in a property Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50357 diff changeset	1342 def _gather_debug(self):
5d210ff4b657 delta-find: move the `gather_debug` logic in a property Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50357 diff changeset	1343 return self._write_debug is not None or self._debug_info is not None
5d210ff4b657 delta-find: move the `gather_debug` logic in a property Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50357 diff changeset	1344
51025 509f0f7fc89e delta-computer: stop explicitly taking file handle Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50952 diff changeset	1345 def buildtext(self, revinfo):
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1346 """Builds a fulltext version of a revision
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1347
47405 34cc102c73f5 revlog: move `revisioninfo` in `revlogutils` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47264 diff changeset	1348 revinfo: revisioninfo instance that contains all needed info
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1349 """
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1350 btext = revinfo.btext
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1351 if btext[0] is not None:
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1352 return btext[0]
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1353
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1354 revlog = self.revlog
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1355 cachedelta = revinfo.cachedelta
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1356 baserev = cachedelta[0]
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1357 delta = cachedelta[1]
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1358
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1359 fulltext = btext[0] = _textfromdelta(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1360 revlog,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1361 baserev,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1362 delta,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1363 revinfo.p1,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1364 revinfo.p2,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1365 revinfo.flags,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1366 revinfo.node,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1367 )
39358 fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39357 diff changeset	1368 return fulltext
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1369
51025 509f0f7fc89e delta-computer: stop explicitly taking file handle Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50952 diff changeset	1370 def _builddeltadiff(self, base, revinfo):
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1371 revlog = self.revlog
51025 509f0f7fc89e delta-computer: stop explicitly taking file handle Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50952 diff changeset	1372 t = self.buildtext(revinfo)
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1373 if revlog.iscensored(base):
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1374 # deltas based on a censored revision must replace the
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1375 # full content in one patch, so delta works everywhere
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1376 header = mdiff.replacediffheader(revlog.rawsize(base), len(t))
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1377 delta = header + t
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1378 else:
51025 509f0f7fc89e delta-computer: stop explicitly taking file handle Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50952 diff changeset	1379 ptext = revlog.rawdata(base)
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1380 delta = mdiff.textdiff(ptext, t)
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1381
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1382 return delta
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1383
51360 670e68729aa7 delta-find: stop using heuristic to determine if we are creating a snapshot Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51359 diff changeset	1384 def _builddeltainfo(
670e68729aa7 delta-find: stop using heuristic to determine if we are creating a snapshot Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51359 diff changeset	1385 self, revinfo, base, target_rev=None, as_snapshot=False
670e68729aa7 delta-find: stop using heuristic to determine if we are creating a snapshot Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51359 diff changeset	1386 ):
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1387 # can we use the cached delta?
42481 6e9ba867a946 delta: move some delta chain related computation earlier in deltainfo Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42480 diff changeset	1388 revlog = self.revlog
6e9ba867a946 delta: move some delta chain related computation earlier in deltainfo Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42480 diff changeset	1389 chainbase = revlog.chainbase(base)
51051 f636103c4d67 revlog: remove legacy usage of `_generaldelta` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51050 diff changeset	1390 if revlog.delta_config.general_delta:
42481 6e9ba867a946 delta: move some delta chain related computation earlier in deltainfo Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42480 diff changeset	1391 deltabase = base
6e9ba867a946 delta: move some delta chain related computation earlier in deltainfo Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42480 diff changeset	1392 else:
50356 2a6949ab9d23 delta-find: add a simple safeguard to prevent bad non-general-delta Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50355 diff changeset	1393 if target_rev is not None and base != target_rev - 1:
2a6949ab9d23 delta-find: add a simple safeguard to prevent bad non-general-delta Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50355 diff changeset	1394 msg = (
2a6949ab9d23 delta-find: add a simple safeguard to prevent bad non-general-delta Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50355 diff changeset	1395 b'general delta cannot use delta for something else '
2a6949ab9d23 delta-find: add a simple safeguard to prevent bad non-general-delta Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50355 diff changeset	1396 b'than `prev`: %d<-%d'
2a6949ab9d23 delta-find: add a simple safeguard to prevent bad non-general-delta Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50355 diff changeset	1397 )
2a6949ab9d23 delta-find: add a simple safeguard to prevent bad non-general-delta Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50355 diff changeset	1398 msg %= (base, target_rev)
2a6949ab9d23 delta-find: add a simple safeguard to prevent bad non-general-delta Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50355 diff changeset	1399 raise error.ProgrammingError(msg)
42481 6e9ba867a946 delta: move some delta chain related computation earlier in deltainfo Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42480 diff changeset	1400 deltabase = chainbase
6e9ba867a946 delta: move some delta chain related computation earlier in deltainfo Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42480 diff changeset	1401 snapshotdepth = None
51065 f71f07a679b4 revlog: remove legacy usage of `_sparserevlog` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51063 diff changeset	1402 if revlog.delta_config.sparse_revlog and deltabase == nullrev:
42481 6e9ba867a946 delta: move some delta chain related computation earlier in deltainfo Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42480 diff changeset	1403 snapshotdepth = 0
51360 670e68729aa7 delta-find: stop using heuristic to determine if we are creating a snapshot Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51359 diff changeset	1404 elif revlog.delta_config.sparse_revlog and as_snapshot:
670e68729aa7 delta-find: stop using heuristic to determine if we are creating a snapshot Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51359 diff changeset	1405 assert revlog.issnapshot(deltabase)
42481 6e9ba867a946 delta: move some delta chain related computation earlier in deltainfo Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42480 diff changeset	1406 # A delta chain should always be one full snapshot,
6e9ba867a946 delta: move some delta chain related computation earlier in deltainfo Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42480 diff changeset	1407 # zero or more semi-snapshots, and zero or more deltas
6e9ba867a946 delta: move some delta chain related computation earlier in deltainfo Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42480 diff changeset	1408 p1, p2 = revlog.rev(revinfo.p1), revlog.rev(revinfo.p2)
6e9ba867a946 delta: move some delta chain related computation earlier in deltainfo Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42480 diff changeset	1409 if deltabase not in (p1, p2) and revlog.issnapshot(deltabase):
6e9ba867a946 delta: move some delta chain related computation earlier in deltainfo Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42480 diff changeset	1410 snapshotdepth = len(revlog._deltachain(deltabase)[0])
39611 a911932d5003 revlog: reuse cached delta for identical base revision (issue5975) Boris Feld <boris.feld@octobus.net> parents: 39610 diff changeset	1411 delta = None
a911932d5003 revlog: reuse cached delta for identical base revision (issue5975) Boris Feld <boris.feld@octobus.net> parents: 39610 diff changeset	1412 if revinfo.cachedelta:
49789 0fca63953810 find-delta: minor preparatory change Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49788 diff changeset	1413 cachebase = revinfo.cachedelta[0]
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1414 # check if the diff still apply
39611 a911932d5003 revlog: reuse cached delta for identical base revision (issue5975) Boris Feld <boris.feld@octobus.net> parents: 39610 diff changeset	1415 currentbase = cachebase
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1416 while (
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1417 currentbase != nullrev
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1418 and currentbase != base
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1419 and self.revlog.length(currentbase) == 0
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1420 ):
39611 a911932d5003 revlog: reuse cached delta for identical base revision (issue5975) Boris Feld <boris.feld@octobus.net> parents: 39610 diff changeset	1421 currentbase = self.revlog.deltaparent(currentbase)
51072 26dcdbe15024 revlog: remove legacy usage of `_lazydeltabase` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51068 diff changeset	1422 if self.revlog.delta_config.lazy_delta and currentbase == base:
39611 a911932d5003 revlog: reuse cached delta for identical base revision (issue5975) Boris Feld <boris.feld@octobus.net> parents: 39610 diff changeset	1423 delta = revinfo.cachedelta[1]
a911932d5003 revlog: reuse cached delta for identical base revision (issue5975) Boris Feld <boris.feld@octobus.net> parents: 39610 diff changeset	1424 if delta is None:
51025 509f0f7fc89e delta-computer: stop explicitly taking file handle Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50952 diff changeset	1425 delta = self._builddeltadiff(base, revinfo)
50378 c84cc0ac77e4 delta-fine: use the `_debug_search` attribute directly Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50377 diff changeset	1426 if self._debug_search:
49249 b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1427 msg = b"DBG-DELTAS-SEARCH: uncompressed-delta-size=%d\n"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1428 msg %= len(delta)
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1429 self._write_debug(msg)
42483 c1c1872d25d1 deltas: skip if projected compressed size does not match text size constraint Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42482 diff changeset	1430 # snapshotdept need to be neither None nor 0 level snapshot
51087 5645524c6b62 revlog: also migrates `revlog.upperboundcomp` to ConfigClass Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51072 diff changeset	1431 if revlog.delta_config.upper_bound_comp is not None and snapshotdepth:
5645524c6b62 revlog: also migrates `revlog.upperboundcomp` to ConfigClass Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51072 diff changeset	1432 lowestrealisticdeltalen = (
5645524c6b62 revlog: also migrates `revlog.upperboundcomp` to ConfigClass Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51072 diff changeset	1433 len(delta) // revlog.delta_config.upper_bound_comp
5645524c6b62 revlog: also migrates `revlog.upperboundcomp` to ConfigClass Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51072 diff changeset	1434 )
42483 c1c1872d25d1 deltas: skip if projected compressed size does not match text size constraint Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42482 diff changeset	1435 snapshotlimit = revinfo.textlen >> snapshotdepth
50378 c84cc0ac77e4 delta-fine: use the `_debug_search` attribute directly Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50377 diff changeset	1436 if self._debug_search:
49249 b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1437 msg = b"DBG-DELTAS-SEARCH: projected-lower-size=%d\n"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1438 msg %= lowestrealisticdeltalen
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1439 self._write_debug(msg)
42483 c1c1872d25d1 deltas: skip if projected compressed size does not match text size constraint Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42482 diff changeset	1440 if snapshotlimit < lowestrealisticdeltalen:
50378 c84cc0ac77e4 delta-fine: use the `_debug_search` attribute directly Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50377 diff changeset	1441 if self._debug_search:
49249 b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1442 msg = b"DBG-DELTAS-SEARCH: DISCARDED (snapshot limit)\n"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1443 self._write_debug(msg)
42483 c1c1872d25d1 deltas: skip if projected compressed size does not match text size constraint Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42482 diff changeset	1444 return None
42484 9b5fbe5ead89 deltas: skip if projected compressed size is bigger than previous snapshot Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42483 diff changeset	1445 if revlog.length(base) < lowestrealisticdeltalen:
50378 c84cc0ac77e4 delta-fine: use the `_debug_search` attribute directly Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50377 diff changeset	1446 if self._debug_search:
49249 b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1447 msg = b"DBG-DELTAS-SEARCH: DISCARDED (prev size)\n"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1448 self._write_debug(msg)
42484 9b5fbe5ead89 deltas: skip if projected compressed size is bigger than previous snapshot Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42483 diff changeset	1449 return None
51095 a82704902db8 revlog: move the compression/decompression logic on the inner object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51087 diff changeset	1450 header, data = revlog._inner.compress(delta)
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1451 deltalen = len(header) + len(data)
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1452 offset = revlog.end(len(revlog) - 1)
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1453 dist = deltalen + offset - revlog.start(chainbase)
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1454 chainlen, compresseddeltalen = revlog._chaininfo(base)
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1455 chainlen += 1
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1456 compresseddeltalen += deltalen
39187 e0da43e2f71f revlog: compute snapshot depth on delta info Boris Feld <boris.feld@octobus.net> parents: 39185 diff changeset	1457
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1458 return _deltainfo(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1459 dist,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1460 deltalen,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1461 (header, data),
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1462 deltabase,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1463 chainbase,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1464 chainlen,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1465 compresseddeltalen,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1466 snapshotdepth,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1467 )
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1468
51025 509f0f7fc89e delta-computer: stop explicitly taking file handle Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50952 diff changeset	1469 def _fullsnapshotinfo(self, revinfo, curr):
509f0f7fc89e delta-computer: stop explicitly taking file handle Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50952 diff changeset	1470 rawtext = self.buildtext(revinfo)
51095 a82704902db8 revlog: move the compression/decompression logic on the inner object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51087 diff changeset	1471 data = self.revlog._inner.compress(rawtext)
39360 5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39359 diff changeset	1472 compresseddeltalen = deltalen = dist = len(data[1]) + len(data[0])
5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39359 diff changeset	1473 deltabase = chainbase = curr
5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39359 diff changeset	1474 snapshotdepth = 0
5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39359 diff changeset	1475 chainlen = 1
5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39359 diff changeset	1476
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1477 return _deltainfo(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1478 dist,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1479 deltalen,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1480 data,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1481 deltabase,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1482 chainbase,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1483 chainlen,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1484 compresseddeltalen,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1485 snapshotdepth,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1486 )
39360 5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39359 diff changeset	1487
51025 509f0f7fc89e delta-computer: stop explicitly taking file handle Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50952 diff changeset	1488 def finddeltainfo(self, revinfo, excluded_bases=None, target_rev=None):
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1489 """Find an acceptable delta against a candidate revision
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1490
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1491 revinfo: information about the revision (instance of _revisioninfo)
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1492
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1493 Returns the first acceptable candidate revision, as ordered by
39361 507f5b1dd7c8 revlogdeltas: extract _getcandidaterevs in a function Boris Feld <boris.feld@octobus.net> parents: 39360 diff changeset	1494 _candidategroups
39360 5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39359 diff changeset	1495
5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39359 diff changeset	1496 If no suitable deltabase is found, we return delta info for a full
5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39359 diff changeset	1497 snapshot.
47407 1efe3cdef53a revlog: add a ways to blacklist some revision when searching for a delta Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47405 diff changeset	1498
1efe3cdef53a revlog: add a ways to blacklist some revision when searching for a delta Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47405 diff changeset	1499 `excluded_bases` is an optional set of revision that cannot be used as
1efe3cdef53a revlog: add a ways to blacklist some revision when searching for a delta Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47405 diff changeset	1500 a delta base. Use this to recompute delta suitable in censor or strip
1efe3cdef53a revlog: add a ways to blacklist some revision when searching for a delta Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47405 diff changeset	1501 context.
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1502 """
47462 93f4e183b3f5 deltas: at a `target_rev` parameter to finddeltainfo Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47458 diff changeset	1503 if target_rev is None:
47463 f8330a3fc39f censor: implement censoring for revlogv2 Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47462 diff changeset	1504 target_rev = len(self.revlog)
47462 93f4e183b3f5 deltas: at a `target_rev` parameter to finddeltainfo Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47458 diff changeset	1505
50382 f930b1b1d671 delta-find: initialize the debug information much sooner (when possible) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50381 diff changeset	1506 gather_debug = self._gather_debug
f930b1b1d671 delta-find: initialize the debug information much sooner (when possible) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50381 diff changeset	1507 cachedelta = revinfo.cachedelta
f930b1b1d671 delta-find: initialize the debug information much sooner (when possible) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50381 diff changeset	1508 revlog = self.revlog
50386 e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50384 diff changeset	1509 p1r = p2r = None
50382 f930b1b1d671 delta-find: initialize the debug information much sooner (when possible) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50381 diff changeset	1510
50386 e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50384 diff changeset	1511 if excluded_bases is None:
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50384 diff changeset	1512 excluded_bases = set()
50382 f930b1b1d671 delta-find: initialize the debug information much sooner (when possible) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50381 diff changeset	1513
f930b1b1d671 delta-find: initialize the debug information much sooner (when possible) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50381 diff changeset	1514 if gather_debug:
f930b1b1d671 delta-find: initialize the debug information much sooner (when possible) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50381 diff changeset	1515 start = util.timer()
f930b1b1d671 delta-find: initialize the debug information much sooner (when possible) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50381 diff changeset	1516 dbg = self._one_dbg_data()
f930b1b1d671 delta-find: initialize the debug information much sooner (when possible) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50381 diff changeset	1517 dbg['revision'] = target_rev
f930b1b1d671 delta-find: initialize the debug information much sooner (when possible) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50381 diff changeset	1518 p1r = revlog.rev(revinfo.p1)
f930b1b1d671 delta-find: initialize the debug information much sooner (when possible) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50381 diff changeset	1519 p2r = revlog.rev(revinfo.p2)
f930b1b1d671 delta-find: initialize the debug information much sooner (when possible) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50381 diff changeset	1520 if p1r != nullrev:
f930b1b1d671 delta-find: initialize the debug information much sooner (when possible) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50381 diff changeset	1521 p1_chain_len = revlog._chaininfo(p1r)[0]
f930b1b1d671 delta-find: initialize the debug information much sooner (when possible) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50381 diff changeset	1522 else:
f930b1b1d671 delta-find: initialize the debug information much sooner (when possible) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50381 diff changeset	1523 p1_chain_len = -1
f930b1b1d671 delta-find: initialize the debug information much sooner (when possible) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50381 diff changeset	1524 if p2r != nullrev:
f930b1b1d671 delta-find: initialize the debug information much sooner (when possible) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50381 diff changeset	1525 p2_chain_len = revlog._chaininfo(p2r)[0]
f930b1b1d671 delta-find: initialize the debug information much sooner (when possible) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50381 diff changeset	1526 else:
f930b1b1d671 delta-find: initialize the debug information much sooner (when possible) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50381 diff changeset	1527 p2_chain_len = -1
f930b1b1d671 delta-find: initialize the debug information much sooner (when possible) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50381 diff changeset	1528 dbg['p1-chain-len'] = p1_chain_len
f930b1b1d671 delta-find: initialize the debug information much sooner (when possible) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50381 diff changeset	1529 dbg['p2-chain-len'] = p2_chain_len
f930b1b1d671 delta-find: initialize the debug information much sooner (when possible) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50381 diff changeset	1530
50383 bfb6404089a5 delta-find: gather the condition to blindly use a full snapshot together Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50382 diff changeset	1531 # 1) if the revision is empty, no amount of delta can beat it
bfb6404089a5 delta-find: gather the condition to blindly use a full snapshot together Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50382 diff changeset	1532 #
bfb6404089a5 delta-find: gather the condition to blindly use a full snapshot together Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50382 diff changeset	1533 # 2) no delta for flag processor revision (see "candelta" for why)
bfb6404089a5 delta-find: gather the condition to blindly use a full snapshot together Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50382 diff changeset	1534 # not calling candelta since only one revision needs test, also to
bfb6404089a5 delta-find: gather the condition to blindly use a full snapshot together Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50382 diff changeset	1535 # avoid overhead fetching flags again.
bfb6404089a5 delta-find: gather the condition to blindly use a full snapshot together Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50382 diff changeset	1536 if not revinfo.textlen or revinfo.flags & REVIDX_RAWTEXT_CHANGING_FLAGS:
51025 509f0f7fc89e delta-computer: stop explicitly taking file handle Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50952 diff changeset	1537 deltainfo = self._fullsnapshotinfo(revinfo, target_rev)
50384 485c9410b75a deltafind: issue debug information when we fast-path rivial case too Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50383 diff changeset	1538 if gather_debug:
485c9410b75a deltafind: issue debug information when we fast-path rivial case too Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50383 diff changeset	1539 end = util.timer()
485c9410b75a deltafind: issue debug information when we fast-path rivial case too Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50383 diff changeset	1540 dbg['duration'] = end - start
485c9410b75a deltafind: issue debug information when we fast-path rivial case too Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50383 diff changeset	1541 dbg[
485c9410b75a deltafind: issue debug information when we fast-path rivial case too Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50383 diff changeset	1542 'delta-base'
485c9410b75a deltafind: issue debug information when we fast-path rivial case too Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50383 diff changeset	1543 ] = deltainfo.base # pytype: disable=attribute-error
485c9410b75a deltafind: issue debug information when we fast-path rivial case too Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50383 diff changeset	1544 dbg['search_round_count'] = 0
50386 e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50384 diff changeset	1545 dbg['using-cached-base'] = False
50384 485c9410b75a deltafind: issue debug information when we fast-path rivial case too Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50383 diff changeset	1546 dbg['delta_try_count'] = 0
485c9410b75a deltafind: issue debug information when we fast-path rivial case too Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50383 diff changeset	1547 dbg['type'] = b"full"
485c9410b75a deltafind: issue debug information when we fast-path rivial case too Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50383 diff changeset	1548 dbg['snapshot-depth'] = 0
485c9410b75a deltafind: issue debug information when we fast-path rivial case too Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50383 diff changeset	1549 self._dbg_process_data(dbg)
485c9410b75a deltafind: issue debug information when we fast-path rivial case too Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50383 diff changeset	1550 return deltainfo
39122 dbb3e9e44fce revlog: do not search for delta for empty content Boris Feld <boris.feld@octobus.net> parents: 39121 diff changeset	1551
50386 e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50384 diff changeset	1552 deltainfo = None
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50384 diff changeset	1553
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50384 diff changeset	1554 # If this source delta are to be forcibly reuse, let us comply early.
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50384 diff changeset	1555 if (
51051 f636103c4d67 revlog: remove legacy usage of `_generaldelta` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51050 diff changeset	1556 revlog.delta_config.general_delta
50386 e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50384 diff changeset	1557 and revinfo.cachedelta is not None
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50384 diff changeset	1558 and revinfo.cachedelta[2] == DELTA_BASE_REUSE_FORCE
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50384 diff changeset	1559 ):
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50384 diff changeset	1560 base = revinfo.cachedelta[0]
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50384 diff changeset	1561 if base == nullrev:
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50384 diff changeset	1562 dbg_type = b"full"
51025 509f0f7fc89e delta-computer: stop explicitly taking file handle Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50952 diff changeset	1563 deltainfo = self._fullsnapshotinfo(revinfo, target_rev)
50386 e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50384 diff changeset	1564 if gather_debug:
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50384 diff changeset	1565 snapshotdepth = 0
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50384 diff changeset	1566 elif base not in excluded_bases:
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50384 diff changeset	1567 delta = revinfo.cachedelta[1]
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50384 diff changeset	1568 header, data = revlog.compress(delta)
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50384 diff changeset	1569 deltalen = len(header) + len(data)
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50384 diff changeset	1570 if gather_debug:
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50384 diff changeset	1571 offset = revlog.end(len(revlog) - 1)
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50384 diff changeset	1572 chainbase = revlog.chainbase(base)
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50384 diff changeset	1573 distance = deltalen + offset - revlog.start(chainbase)
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50384 diff changeset	1574 chainlen, compresseddeltalen = revlog._chaininfo(base)
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50384 diff changeset	1575 chainlen += 1
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50384 diff changeset	1576 compresseddeltalen += deltalen
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50384 diff changeset	1577 if base == p1r or base == p2r:
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50384 diff changeset	1578 dbg_type = b"delta"
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50384 diff changeset	1579 snapshotdepth = None
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50384 diff changeset	1580 elif not revlog.issnapshot(base):
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50384 diff changeset	1581 snapshotdepth = None
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50384 diff changeset	1582 else:
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50384 diff changeset	1583 dbg_type = b"snapshot"
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50384 diff changeset	1584 snapshotdepth = revlog.snapshotdepth(base) + 1
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50384 diff changeset	1585 else:
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50384 diff changeset	1586 distance = None
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50384 diff changeset	1587 chainbase = None
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50384 diff changeset	1588 chainlen = None
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50384 diff changeset	1589 compresseddeltalen = None
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50384 diff changeset	1590 snapshotdepth = None
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50384 diff changeset	1591 deltainfo = _deltainfo(
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50384 diff changeset	1592 distance=distance,
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50384 diff changeset	1593 deltalen=deltalen,
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50384 diff changeset	1594 data=(header, data),
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50384 diff changeset	1595 base=base,
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50384 diff changeset	1596 chainbase=chainbase,
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50384 diff changeset	1597 chainlen=chainlen,
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50384 diff changeset	1598 compresseddeltalen=compresseddeltalen,
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50384 diff changeset	1599 snapshotdepth=snapshotdepth,
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50384 diff changeset	1600 )
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50384 diff changeset	1601
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50384 diff changeset	1602 if deltainfo is not None:
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50384 diff changeset	1603 if gather_debug:
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50384 diff changeset	1604 end = util.timer()
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50384 diff changeset	1605 dbg['duration'] = end - start
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50384 diff changeset	1606 dbg[
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50384 diff changeset	1607 'delta-base'
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50384 diff changeset	1608 ] = deltainfo.base # pytype: disable=attribute-error
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50384 diff changeset	1609 dbg['search_round_count'] = 0
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50384 diff changeset	1610 dbg['using-cached-base'] = True
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50384 diff changeset	1611 dbg['delta_try_count'] = 0
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50384 diff changeset	1612 dbg['type'] = b"full"
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50384 diff changeset	1613 if snapshotdepth is None:
51343 0d92d62ecde0 delta-find: use "-1" as depth snapshot-dept for non snapshot in debug Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51342 diff changeset	1614 dbg['snapshot-depth'] = -1
50386 e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50384 diff changeset	1615 else:
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50384 diff changeset	1616 dbg['snapshot-depth'] = snapshotdepth
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50384 diff changeset	1617 self._dbg_process_data(dbg)
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50384 diff changeset	1618 return deltainfo
47407 1efe3cdef53a revlog: add a ways to blacklist some revision when searching for a delta Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47405 diff changeset	1619
49247 e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	1620 # count the number of different delta we tried (for debug purpose)
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	1621 dbg_try_count = 0
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	1622 # count the number of "search round" we did. (for debug purpose)
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	1623 dbg_try_rounds = 0
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	1624 dbg_type = b'unknown'
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	1625
50382 f930b1b1d671 delta-find: initialize the debug information much sooner (when possible) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50381 diff changeset	1626 if p1r is None:
f930b1b1d671 delta-find: initialize the debug information much sooner (when possible) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50381 diff changeset	1627 p1r = revlog.rev(revinfo.p1)
f930b1b1d671 delta-find: initialize the debug information much sooner (when possible) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50381 diff changeset	1628 p2r = revlog.rev(revinfo.p2)
49247 e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	1629
50378 c84cc0ac77e4 delta-fine: use the `_debug_search` attribute directly Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50377 diff changeset	1630 if self._debug_search:
49249 b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1631 msg = b"DBG-DELTAS-SEARCH: SEARCH rev=%d\n"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1632 msg %= target_rev
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1633 self._write_debug(msg)
49247 e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	1634
51357 555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51356 diff changeset	1635 # should we try to build a delta?
555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51356 diff changeset	1636 if not (len(self.revlog) and self.revlog._storedeltachains):
555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51356 diff changeset	1637 search_cls = _NoDeltaSearch
51367 d7e2acdd50ba delta-find: split the _DeltaSearch class in two Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51366 diff changeset	1638 elif self.revlog.delta_config.sparse_revlog:
d7e2acdd50ba delta-find: split the _DeltaSearch class in two Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51366 diff changeset	1639 search_cls = _SparseDeltaSearch
d7e2acdd50ba delta-find: split the _DeltaSearch class in two Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51366 diff changeset	1640 elif self.revlog.delta_config.general_delta:
d7e2acdd50ba delta-find: split the _DeltaSearch class in two Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51366 diff changeset	1641 search_cls = _GeneralDeltaSearch
d7e2acdd50ba delta-find: split the _DeltaSearch class in two Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51366 diff changeset	1642 else:
51357 555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51356 diff changeset	1643 # before general delta, there is only one possible delta base
555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51356 diff changeset	1644 search_cls = _PrevDeltaSearch
555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51356 diff changeset	1645
555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51356 diff changeset	1646 search = search_cls(
49740 2afee2176775 delta-find: expand a function definition and call before extendin it Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49739 diff changeset	1647 self.revlog,
51340 49401b7dec0c delta-find: feed revinfo to _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51338 diff changeset	1648 revinfo,
49740 2afee2176775 delta-find: expand a function definition and call before extendin it Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49739 diff changeset	1649 p1r,
2afee2176775 delta-find: expand a function definition and call before extendin it Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49739 diff changeset	1650 p2r,
2afee2176775 delta-find: expand a function definition and call before extendin it Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49739 diff changeset	1651 cachedelta,
49741 e706bb41fdb3 delta-find: move pre-filtering with other pre-filtering logic Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49740 diff changeset	1652 excluded_bases,
e706bb41fdb3 delta-find: move pre-filtering with other pre-filtering logic Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49740 diff changeset	1653 target_rev,
49795 c261a628e525 delta-find: use a single snapshot cache when applying a group to an object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49794 diff changeset	1654 snapshot_cache=self._snapshot_cache,
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42994 diff changeset	1655 )
51330 c82e03b102a6 delta-find: introduce a _DeltaSearch object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51329 diff changeset	1656
51344 d0d869fccd20 delta-find: move away from the generator API for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51343 diff changeset	1657 while not search.done:
d0d869fccd20 delta-find: move away from the generator API for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51343 diff changeset	1658 current_group = search.current_group
d0d869fccd20 delta-find: move away from the generator API for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51343 diff changeset	1659 # current_group can be `None`, but not is search.done is False
d0d869fccd20 delta-find: move away from the generator API for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51343 diff changeset	1660 # We add this assert to help pytype
d0d869fccd20 delta-find: move away from the generator API for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51343 diff changeset	1661 assert current_group is not None
d0d869fccd20 delta-find: move away from the generator API for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51343 diff changeset	1662 candidaterevs = current_group
49247 e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	1663 dbg_try_rounds += 1
50378 c84cc0ac77e4 delta-fine: use the `_debug_search` attribute directly Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50377 diff changeset	1664 if self._debug_search:
49249 b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1665 prev = None
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1666 if deltainfo is not None:
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1667 prev = deltainfo.base
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1668
49737 78ba41878f2e delta-find: add debug information about reuse of cached data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49249 diff changeset	1669 if (
78ba41878f2e delta-find: add debug information about reuse of cached data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49249 diff changeset	1670 cachedelta is not None
78ba41878f2e delta-find: add debug information about reuse of cached data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49249 diff changeset	1671 and len(candidaterevs) == 1
78ba41878f2e delta-find: add debug information about reuse of cached data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49249 diff changeset	1672 and cachedelta[0] in candidaterevs
78ba41878f2e delta-find: add debug information about reuse of cached data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49249 diff changeset	1673 ):
78ba41878f2e delta-find: add debug information about reuse of cached data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49249 diff changeset	1674 round_type = b"cached-delta"
50381 ebb292ffdf4d delta-find: fix `parents` round detection Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50380 diff changeset	1675 elif p1r in candidaterevs or p2r in candidaterevs:
49249 b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1676 round_type = b"parents"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1677 elif prev is not None and all(c < prev for c in candidaterevs):
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1678 round_type = b"refine-down"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1679 elif prev is not None and all(c > prev for c in candidaterevs):
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1680 round_type = b"refine-up"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1681 else:
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1682 round_type = b"search-down"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1683 msg = b"DBG-DELTAS-SEARCH: ROUND #%d - %d candidates - %s\n"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1684 msg %= (dbg_try_rounds, len(candidaterevs), round_type)
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1685 self._write_debug(msg)
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1686 nominateddeltas = []
39515 04b75f3a3f2a snapshot: add refining logic at the findeltainfo level Boris Feld <boris.feld@octobus.net> parents: 39514 diff changeset	1687 if deltainfo is not None:
50378 c84cc0ac77e4 delta-fine: use the `_debug_search` attribute directly Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50377 diff changeset	1688 if self._debug_search:
49249 b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1689 msg = (
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1690 b"DBG-DELTAS-SEARCH: CONTENDER: rev=%d - length=%d\n"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1691 )
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1692 msg %= (deltainfo.base, deltainfo.deltalen)
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1693 self._write_debug(msg)
39515 04b75f3a3f2a snapshot: add refining logic at the findeltainfo level Boris Feld <boris.feld@octobus.net> parents: 39514 diff changeset	1694 # if we already found a good delta,
04b75f3a3f2a snapshot: add refining logic at the findeltainfo level Boris Feld <boris.feld@octobus.net> parents: 39514 diff changeset	1695 # challenge it against refined candidates
04b75f3a3f2a snapshot: add refining logic at the findeltainfo level Boris Feld <boris.feld@octobus.net> parents: 39514 diff changeset	1696 nominateddeltas.append(deltainfo)
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1697 for candidaterev in candidaterevs:
50378 c84cc0ac77e4 delta-fine: use the `_debug_search` attribute directly Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50377 diff changeset	1698 if self._debug_search:
49249 b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1699 msg = b"DBG-DELTAS-SEARCH: CANDIDATE: rev=%d\n"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1700 msg %= candidaterev
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1701 self._write_debug(msg)
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1702 candidate_type = None
50381 ebb292ffdf4d delta-find: fix `parents` round detection Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50380 diff changeset	1703 if candidaterev == p1r:
49249 b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1704 candidate_type = b"p1"
50381 ebb292ffdf4d delta-find: fix `parents` round detection Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50380 diff changeset	1705 elif candidaterev == p2r:
49249 b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1706 candidate_type = b"p2"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1707 elif self.revlog.issnapshot(candidaterev):
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1708 candidate_type = b"snapshot-%d"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1709 candidate_type %= self.revlog.snapshotdepth(
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1710 candidaterev
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1711 )
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1712
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1713 if candidate_type is not None:
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1714 msg = b"DBG-DELTAS-SEARCH: type=%s\n"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1715 msg %= candidate_type
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1716 self._write_debug(msg)
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1717 msg = b"DBG-DELTAS-SEARCH: size=%d\n"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1718 msg %= self.revlog.length(candidaterev)
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1719 self._write_debug(msg)
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1720 msg = b"DBG-DELTAS-SEARCH: base=%d\n"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1721 msg %= self.revlog.deltaparent(candidaterev)
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1722 self._write_debug(msg)
49741 e706bb41fdb3 delta-find: move pre-filtering with other pre-filtering logic Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49740 diff changeset	1723
49247 e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	1724 dbg_try_count += 1
49249 b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1725
50378 c84cc0ac77e4 delta-fine: use the `_debug_search` attribute directly Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50377 diff changeset	1726 if self._debug_search:
49249 b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1727 delta_start = util.timer()
50356 2a6949ab9d23 delta-find: add a simple safeguard to prevent bad non-general-delta Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50355 diff changeset	1728 candidatedelta = self._builddeltainfo(
2a6949ab9d23 delta-find: add a simple safeguard to prevent bad non-general-delta Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50355 diff changeset	1729 revinfo,
2a6949ab9d23 delta-find: add a simple safeguard to prevent bad non-general-delta Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50355 diff changeset	1730 candidaterev,
2a6949ab9d23 delta-find: add a simple safeguard to prevent bad non-general-delta Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50355 diff changeset	1731 target_rev=target_rev,
51360 670e68729aa7 delta-find: stop using heuristic to determine if we are creating a snapshot Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51359 diff changeset	1732 as_snapshot=search.current_stage == _STAGE_SNAPSHOT,
50356 2a6949ab9d23 delta-find: add a simple safeguard to prevent bad non-general-delta Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50355 diff changeset	1733 )
50378 c84cc0ac77e4 delta-fine: use the `_debug_search` attribute directly Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50377 diff changeset	1734 if self._debug_search:
49249 b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1735 delta_end = util.timer()
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1736 msg = b"DBG-DELTAS-SEARCH: delta-search-time=%f\n"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1737 msg %= delta_end - delta_start
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1738 self._write_debug(msg)
42482 465f2d0df9ae deltas: accept and skip None return for delta info Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42481 diff changeset	1739 if candidatedelta is not None:
51341 7455cae67260 delta-find: move is_good_delta_info on the _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	1740 if search.is_good_delta_info(candidatedelta):
50378 c84cc0ac77e4 delta-fine: use the `_debug_search` attribute directly Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50377 diff changeset	1741 if self._debug_search:
49249 b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1742 msg = b"DBG-DELTAS-SEARCH: DELTA: length=%d (GOOD)\n"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1743 msg %= candidatedelta.deltalen
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1744 self._write_debug(msg)
42482 465f2d0df9ae deltas: accept and skip None return for delta info Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42481 diff changeset	1745 nominateddeltas.append(candidatedelta)
50378 c84cc0ac77e4 delta-fine: use the `_debug_search` attribute directly Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50377 diff changeset	1746 elif self._debug_search:
49249 b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1747 msg = b"DBG-DELTAS-SEARCH: DELTA: length=%d (BAD)\n"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1748 msg %= candidatedelta.deltalen
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1749 self._write_debug(msg)
50378 c84cc0ac77e4 delta-fine: use the `_debug_search` attribute directly Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50377 diff changeset	1750 elif self._debug_search:
49249 b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1751 msg = b"DBG-DELTAS-SEARCH: NO-DELTA\n"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1752 self._write_debug(msg)
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1753 if nominateddeltas:
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1754 deltainfo = min(nominateddeltas, key=lambda x: x.deltalen)
51344 d0d869fccd20 delta-find: move away from the generator API for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51343 diff changeset	1755 search.next_group(deltainfo)
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1756
39360 5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39359 diff changeset	1757 if deltainfo is None:
49247 e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	1758 dbg_type = b"full"
51025 509f0f7fc89e delta-computer: stop explicitly taking file handle Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50952 diff changeset	1759 deltainfo = self._fullsnapshotinfo(revinfo, target_rev)
49249 b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1760 elif deltainfo.snapshotdepth: # pytype: disable=attribute-error
49247 e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	1761 dbg_type = b"snapshot"
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	1762 else:
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	1763 dbg_type = b"delta"
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	1764
49739 35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	1765 if gather_debug:
49247 e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	1766 end = util.timer()
49885 d57b966cdeb1 delta-find: properly report full snapshot used from cache as such Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49879 diff changeset	1767 if dbg_type == b'full':
d57b966cdeb1 delta-find: properly report full snapshot used from cache as such Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49879 diff changeset	1768 used_cached = (
d57b966cdeb1 delta-find: properly report full snapshot used from cache as such Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49879 diff changeset	1769 cachedelta is not None
d57b966cdeb1 delta-find: properly report full snapshot used from cache as such Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49879 diff changeset	1770 and dbg_try_rounds == 0
d57b966cdeb1 delta-find: properly report full snapshot used from cache as such Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49879 diff changeset	1771 and dbg_try_count == 0
d57b966cdeb1 delta-find: properly report full snapshot used from cache as such Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49879 diff changeset	1772 and cachedelta[0] == nullrev
d57b966cdeb1 delta-find: properly report full snapshot used from cache as such Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49879 diff changeset	1773 )
d57b966cdeb1 delta-find: properly report full snapshot used from cache as such Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49879 diff changeset	1774 else:
d57b966cdeb1 delta-find: properly report full snapshot used from cache as such Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49879 diff changeset	1775 used_cached = (
d57b966cdeb1 delta-find: properly report full snapshot used from cache as such Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49879 diff changeset	1776 cachedelta is not None
d57b966cdeb1 delta-find: properly report full snapshot used from cache as such Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49879 diff changeset	1777 and dbg_try_rounds == 1
d57b966cdeb1 delta-find: properly report full snapshot used from cache as such Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49879 diff changeset	1778 and dbg_try_count == 1
d57b966cdeb1 delta-find: properly report full snapshot used from cache as such Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49879 diff changeset	1779 and deltainfo.base == cachedelta[0]
d57b966cdeb1 delta-find: properly report full snapshot used from cache as such Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49879 diff changeset	1780 )
50380 d1dc4a03125e delta-find: intrduce a `_one_dbg_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50379 diff changeset	1781 dbg['duration'] = end - start
d1dc4a03125e delta-find: intrduce a `_one_dbg_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50379 diff changeset	1782 dbg[
d1dc4a03125e delta-find: intrduce a `_one_dbg_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50379 diff changeset	1783 'delta-base'
d1dc4a03125e delta-find: intrduce a `_one_dbg_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50379 diff changeset	1784 ] = deltainfo.base # pytype: disable=attribute-error
d1dc4a03125e delta-find: intrduce a `_one_dbg_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50379 diff changeset	1785 dbg['search_round_count'] = dbg_try_rounds
d1dc4a03125e delta-find: intrduce a `_one_dbg_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50379 diff changeset	1786 dbg['using-cached-base'] = used_cached
d1dc4a03125e delta-find: intrduce a `_one_dbg_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50379 diff changeset	1787 dbg['delta_try_count'] = dbg_try_count
d1dc4a03125e delta-find: intrduce a `_one_dbg_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50379 diff changeset	1788 dbg['type'] = dbg_type
49249 b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1789 if (
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1790 deltainfo.snapshotdepth # pytype: disable=attribute-error
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1791 is not None
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1792 ):
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1793 dbg[
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1794 'snapshot-depth'
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49247 diff changeset	1795 ] = deltainfo.snapshotdepth # pytype: disable=attribute-error
49247 e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	1796 else:
51343 0d92d62ecde0 delta-find: use "-1" as depth snapshot-dept for non snapshot in debug Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51342 diff changeset	1797 dbg['snapshot-depth'] = -1
50379 4d84b6d52e93 delta-find: move final debug processing in a `_dbg_process_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50378 diff changeset	1798 self._dbg_process_data(dbg)
4d84b6d52e93 delta-find: move final debug processing in a `_dbg_process_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50378 diff changeset	1799 return deltainfo
49247 e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49037 diff changeset	1800
50380 d1dc4a03125e delta-find: intrduce a `_one_dbg_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50379 diff changeset	1801 def _one_dbg_data(self):
51338 46415df4eb1c delta-find: move filing of some debug data in `_one_dbg_data` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51337 diff changeset	1802 dbg = {
50380 d1dc4a03125e delta-find: intrduce a `_one_dbg_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50379 diff changeset	1803 'duration': None,
d1dc4a03125e delta-find: intrduce a `_one_dbg_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50379 diff changeset	1804 'revision': None,
d1dc4a03125e delta-find: intrduce a `_one_dbg_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50379 diff changeset	1805 'delta-base': None,
d1dc4a03125e delta-find: intrduce a `_one_dbg_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50379 diff changeset	1806 'search_round_count': None,
d1dc4a03125e delta-find: intrduce a `_one_dbg_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50379 diff changeset	1807 'using-cached-base': None,
d1dc4a03125e delta-find: intrduce a `_one_dbg_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50379 diff changeset	1808 'delta_try_count': None,
d1dc4a03125e delta-find: intrduce a `_one_dbg_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50379 diff changeset	1809 'type': None,
d1dc4a03125e delta-find: intrduce a `_one_dbg_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50379 diff changeset	1810 'p1-chain-len': None,
d1dc4a03125e delta-find: intrduce a `_one_dbg_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50379 diff changeset	1811 'p2-chain-len': None,
d1dc4a03125e delta-find: intrduce a `_one_dbg_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50379 diff changeset	1812 'snapshot-depth': None,
d1dc4a03125e delta-find: intrduce a `_one_dbg_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50379 diff changeset	1813 'target-revlog': None,
d1dc4a03125e delta-find: intrduce a `_one_dbg_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50379 diff changeset	1814 }
51338 46415df4eb1c delta-find: move filing of some debug data in `_one_dbg_data` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51337 diff changeset	1815 target_revlog = b"UNKNOWN"
46415df4eb1c delta-find: move filing of some debug data in `_one_dbg_data` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51337 diff changeset	1816 target_type = self.revlog.target[0]
46415df4eb1c delta-find: move filing of some debug data in `_one_dbg_data` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51337 diff changeset	1817 target_key = self.revlog.target[1]
46415df4eb1c delta-find: move filing of some debug data in `_one_dbg_data` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51337 diff changeset	1818 if target_type == KIND_CHANGELOG:
46415df4eb1c delta-find: move filing of some debug data in `_one_dbg_data` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51337 diff changeset	1819 target_revlog = b'CHANGELOG:'
46415df4eb1c delta-find: move filing of some debug data in `_one_dbg_data` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51337 diff changeset	1820 elif target_type == KIND_MANIFESTLOG:
46415df4eb1c delta-find: move filing of some debug data in `_one_dbg_data` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51337 diff changeset	1821 target_revlog = b'MANIFESTLOG:'
46415df4eb1c delta-find: move filing of some debug data in `_one_dbg_data` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51337 diff changeset	1822 if target_key:
46415df4eb1c delta-find: move filing of some debug data in `_one_dbg_data` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51337 diff changeset	1823 target_revlog += b'%s:' % target_key
46415df4eb1c delta-find: move filing of some debug data in `_one_dbg_data` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51337 diff changeset	1824 elif target_type == KIND_FILELOG:
46415df4eb1c delta-find: move filing of some debug data in `_one_dbg_data` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51337 diff changeset	1825 target_revlog = b'FILELOG:'
46415df4eb1c delta-find: move filing of some debug data in `_one_dbg_data` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51337 diff changeset	1826 if target_key:
46415df4eb1c delta-find: move filing of some debug data in `_one_dbg_data` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51337 diff changeset	1827 target_revlog += b'%s:' % target_key
46415df4eb1c delta-find: move filing of some debug data in `_one_dbg_data` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51337 diff changeset	1828 dbg['target-revlog'] = target_revlog
46415df4eb1c delta-find: move filing of some debug data in `_one_dbg_data` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51337 diff changeset	1829 return dbg
50380 d1dc4a03125e delta-find: intrduce a `_one_dbg_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50379 diff changeset	1830
50379 4d84b6d52e93 delta-find: move final debug processing in a `_dbg_process_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50378 diff changeset	1831 def _dbg_process_data(self, dbg):
4d84b6d52e93 delta-find: move final debug processing in a `_dbg_process_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50378 diff changeset	1832 if self._debug_info is not None:
4d84b6d52e93 delta-find: move final debug processing in a `_dbg_process_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50378 diff changeset	1833 self._debug_info.append(dbg)
49739 35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49737 diff changeset	1834
50379 4d84b6d52e93 delta-find: move final debug processing in a `_dbg_process_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50378 diff changeset	1835 if self._write_debug is not None:
4d84b6d52e93 delta-find: move final debug processing in a `_dbg_process_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50378 diff changeset	1836 msg = (
4d84b6d52e93 delta-find: move final debug processing in a `_dbg_process_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50378 diff changeset	1837 b"DBG-DELTAS:"
4d84b6d52e93 delta-find: move final debug processing in a `_dbg_process_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50378 diff changeset	1838 b" %-12s"
4d84b6d52e93 delta-find: move final debug processing in a `_dbg_process_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50378 diff changeset	1839 b" rev=%d:"
4d84b6d52e93 delta-find: move final debug processing in a `_dbg_process_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50378 diff changeset	1840 b" delta-base=%d"
4d84b6d52e93 delta-find: move final debug processing in a `_dbg_process_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50378 diff changeset	1841 b" is-cached=%d"
4d84b6d52e93 delta-find: move final debug processing in a `_dbg_process_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50378 diff changeset	1842 b" - search-rounds=%d"
4d84b6d52e93 delta-find: move final debug processing in a `_dbg_process_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50378 diff changeset	1843 b" try-count=%d"
4d84b6d52e93 delta-find: move final debug processing in a `_dbg_process_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50378 diff changeset	1844 b" - delta-type=%-6s"
4d84b6d52e93 delta-find: move final debug processing in a `_dbg_process_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50378 diff changeset	1845 b" snap-depth=%d"
4d84b6d52e93 delta-find: move final debug processing in a `_dbg_process_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50378 diff changeset	1846 b" - p1-chain-length=%d"
4d84b6d52e93 delta-find: move final debug processing in a `_dbg_process_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50378 diff changeset	1847 b" p2-chain-length=%d"
4d84b6d52e93 delta-find: move final debug processing in a `_dbg_process_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50378 diff changeset	1848 b" - duration=%f"
4d84b6d52e93 delta-find: move final debug processing in a `_dbg_process_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50378 diff changeset	1849 b"\n"
4d84b6d52e93 delta-find: move final debug processing in a `_dbg_process_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50378 diff changeset	1850 )
4d84b6d52e93 delta-find: move final debug processing in a `_dbg_process_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50378 diff changeset	1851 msg %= (
4d84b6d52e93 delta-find: move final debug processing in a `_dbg_process_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50378 diff changeset	1852 dbg["target-revlog"],
4d84b6d52e93 delta-find: move final debug processing in a `_dbg_process_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50378 diff changeset	1853 dbg["revision"],
4d84b6d52e93 delta-find: move final debug processing in a `_dbg_process_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50378 diff changeset	1854 dbg["delta-base"],
4d84b6d52e93 delta-find: move final debug processing in a `_dbg_process_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50378 diff changeset	1855 dbg["using-cached-base"],
4d84b6d52e93 delta-find: move final debug processing in a `_dbg_process_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50378 diff changeset	1856 dbg["search_round_count"],
4d84b6d52e93 delta-find: move final debug processing in a `_dbg_process_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50378 diff changeset	1857 dbg["delta_try_count"],
4d84b6d52e93 delta-find: move final debug processing in a `_dbg_process_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50378 diff changeset	1858 dbg["type"],
4d84b6d52e93 delta-find: move final debug processing in a `_dbg_process_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50378 diff changeset	1859 dbg["snapshot-depth"],
4d84b6d52e93 delta-find: move final debug processing in a `_dbg_process_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50378 diff changeset	1860 dbg["p1-chain-len"],
4d84b6d52e93 delta-find: move final debug processing in a `_dbg_process_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50378 diff changeset	1861 dbg["p2-chain-len"],
4d84b6d52e93 delta-find: move final debug processing in a `_dbg_process_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50378 diff changeset	1862 dbg["duration"],
4d84b6d52e93 delta-find: move final debug processing in a `_dbg_process_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50378 diff changeset	1863 )
4d84b6d52e93 delta-find: move final debug processing in a `_dbg_process_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50378 diff changeset	1864 self._write_debug(msg)
47458 c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47407 diff changeset	1865
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47407 diff changeset	1866
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47407 diff changeset	1867 def delta_compression(default_compression_header, deltainfo):
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47407 diff changeset	1868 """return (COMPRESSION_MODE, deltainfo)
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47407 diff changeset	1869
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47407 diff changeset	1870 used by revlog v2+ format to dispatch between PLAIN and DEFAULT
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47407 diff changeset	1871 compression.
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47407 diff changeset	1872 """
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47407 diff changeset	1873 h, d = deltainfo.data
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47407 diff changeset	1874 compression_mode = COMP_MODE_INLINE
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47407 diff changeset	1875 if not h and not d:
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47407 diff changeset	1876 # not data to store at all... declare them uncompressed
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47407 diff changeset	1877 compression_mode = COMP_MODE_PLAIN
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47407 diff changeset	1878 elif not h:
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47407 diff changeset	1879 t = d[0:1]
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47407 diff changeset	1880 if t == b'\0':
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47407 diff changeset	1881 compression_mode = COMP_MODE_PLAIN
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47407 diff changeset	1882 elif t == default_compression_header:
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47407 diff changeset	1883 compression_mode = COMP_MODE_DEFAULT
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47407 diff changeset	1884 elif h == b'u':
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47407 diff changeset	1885 # we have a more efficient way to declare uncompressed
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47407 diff changeset	1886 h = b''
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47407 diff changeset	1887 compression_mode = COMP_MODE_PLAIN
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47407 diff changeset	1888 deltainfo = drop_u_compression(deltainfo)
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47407 diff changeset	1889 return compression_mode, deltainfo

Mercurial > public > mercurial-scm > hg-stable

annotate mercurial/revlogutils/deltas.py @ 52669:e627cc25b6f3