mercurial-scm/hg: mercurial/revlogutils/deltas.py annotate

annotate mercurial/revlogutils/deltas.py @ 51355:383e99f6bc99

delta-find: move sparse-revlog delta checks in the associated class Lets move the specialized code in the specialized class.

author	Pierre-Yves David <pierre-yves.david@octobus.net>
date	Sun, 07 Jan 2024 05:16:08 +0100
parents	d7e2acdd50ba
children	a224ce5694b3

rev	line source
39330 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	1 # revlogdeltas.py - Logic around delta computation for revlog
8226 8b2cd04a6e97 put license and copyright info into comment blocks Martin Geisler <mg@lazybytes.net> parents: 8225 diff changeset	2 #
46819 d4ba4d51f85f contributor: change mentions of mpm to olivia Rapha?l Gom?s <rgomes@octobus.net> parents: 43089 diff changeset	3 # Copyright 2005-2007 Olivia Mackall <olivia@selenic.com>
39330 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	4 # Copyright 2018 Octobus <contact@octobus.net>
8226 8b2cd04a6e97 put license and copyright info into comment blocks Martin Geisler <mg@lazybytes.net> parents: 8225 diff changeset	5 #
8b2cd04a6e97 put license and copyright info into comment blocks Martin Geisler <mg@lazybytes.net> parents: 8225 diff changeset	6 # This software may be used and distributed according to the terms of the
10263 25e572394f5c Update license to GPLv2+ Matt Mackall <mpm@selenic.com> parents: 10047 diff changeset	7 # GNU General Public License version 2 or any later version.
39330 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	8 """Helper class to compute deltas stored inside revlogs"""
8227 0a9542703300 turn some comments back into module docstrings Martin Geisler <mg@lazybytes.net> parents: 8226 diff changeset	9
27361 29f50344fa83 revlog: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27251 diff changeset	10
51343 2a333d791ecf delta-find: introduce a base class for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51342 diff changeset	11 import abc
39493 3ca144f1c8dd snapshot: search for unrelated but reusable full-snapshot Boris Feld <boris.feld@octobus.net> parents: 39492 diff changeset	12 import collections
27361 29f50344fa83 revlog: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27251 diff changeset	13 import struct
29f50344fa83 revlog: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27251 diff changeset	14
29f50344fa83 revlog: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27251 diff changeset	15 # import stuff from node for others to import from revlog
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	16 from ..node import nullrev
39330 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	17 from ..i18n import _
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	18
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	19 from .constants import (
47452 c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47401 diff changeset	20 COMP_MODE_DEFAULT,
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47401 diff changeset	21 COMP_MODE_INLINE,
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47401 diff changeset	22 COMP_MODE_PLAIN,
49768 bcae90c53def delta-find: add a delta-reuse policy that blindly accepts incoming deltas Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49681 diff changeset	23 DELTA_BASE_REUSE_FORCE,
49677 05db41701ece find-delta: pass the cache-delta usage policy alongside the cache-delta Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49675 diff changeset	24 DELTA_BASE_REUSE_NO,
49226 e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	25 KIND_CHANGELOG,
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	26 KIND_FILELOG,
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	27 KIND_MANIFESTLOG,
39329 729082bb9938 revlog: split constants into a new `revlogutils.constants` module Boris Feld <boris.feld@octobus.net> parents: 39232 diff changeset	28 REVIDX_ISCENSORED,
729082bb9938 revlog: split constants into a new `revlogutils.constants` module Boris Feld <boris.feld@octobus.net> parents: 39232 diff changeset	29 REVIDX_RAWTEXT_CHANGING_FLAGS,
729082bb9938 revlog: split constants into a new `revlogutils.constants` module Boris Feld <boris.feld@octobus.net> parents: 39232 diff changeset	30 )
39330 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	31
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	32 from ..thirdparty import attr
39330 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	33
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	34 from .. import (
27361 29f50344fa83 revlog: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27251 diff changeset	35 error,
29f50344fa83 revlog: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27251 diff changeset	36 mdiff,
41108 38e88450138c delta: have a native implementation of _findsnapshot Boris Feld <boris.feld@octobus.net> parents: 41079 diff changeset	37 util,
27361 29f50344fa83 revlog: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27251 diff changeset	38 )
10913 f2ecc5733c89 revlog: factor out _maxinline global. Greg Ward <greg-hg@gerg.ca> parents: 10404 diff changeset	39
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	40 from . import flagutil
42992 dff95420480f flagprocessors: make `processflagsraw` a module level function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42878 diff changeset	41
39330 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	42 # maximum <delta-chain-data>/<revision-text-length> ratio
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	43 LIMIT_DELTA2TEXT = 2
1091 d62130f99a73 Move hash function back to revlog from node mpm@selenic.com parents: 1089 diff changeset	44
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	45
48946 642e31cb55f0 py3: use class X: instead of class X(object): Gregory Szorc <gregory.szorc@gmail.com> parents: 48875 diff changeset	46 class _testrevlog:
38637 e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	47 """minimalist fake revlog to use in doctests"""
e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	48
40641 85b14f0dc334 doctest: add a `issnapshot` method to _testrevlog Boris Feld <boris.feld@octobus.net> parents: 40608 diff changeset	49 def __init__(self, data, density=0.5, mingap=0, snapshot=()):
38637 e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	50 """data is an list of revision payload boundaries"""
51042 127656e0b97b revlog: use the new Config classes in _testrevlog Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51017 diff changeset	51 from .. import revlog
127656e0b97b revlog: use the new Config classes in _testrevlog Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51017 diff changeset	52
38637 e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	53 self._data = data
51042 127656e0b97b revlog: use the new Config classes in _testrevlog Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51017 diff changeset	54 self.data_config = revlog.DataConfig()
127656e0b97b revlog: use the new Config classes in _testrevlog Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51017 diff changeset	55 self.data_config.sr_density_threshold = density
127656e0b97b revlog: use the new Config classes in _testrevlog Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51017 diff changeset	56 self.data_config.sr_min_gap_size = mingap
127656e0b97b revlog: use the new Config classes in _testrevlog Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51017 diff changeset	57 self.delta_config = revlog.DeltaConfig()
127656e0b97b revlog: use the new Config classes in _testrevlog Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51017 diff changeset	58 self.feature_config = revlog.FeatureConfig()
40641 85b14f0dc334 doctest: add a `issnapshot` method to _testrevlog Boris Feld <boris.feld@octobus.net> parents: 40608 diff changeset	59 self._snapshot = set(snapshot)
40709 39d29542fe40 sparse-revlog: put the native implementation of slicechunktodensity to use Boris Feld <boris.feld@octobus.net> parents: 40657 diff changeset	60 self.index = None
38637 e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	61
e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	62 def start(self, rev):
41079 88d813cd9acd revlog: fix pure python slicing test when chain contains nullrev Boris Feld <boris.feld@octobus.net> parents: 41033 diff changeset	63 if rev == nullrev:
88d813cd9acd revlog: fix pure python slicing test when chain contains nullrev Boris Feld <boris.feld@octobus.net> parents: 41033 diff changeset	64 return 0
38637 e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	65 if rev == 0:
e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	66 return 0
e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	67 return self._data[rev - 1]
e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	68
e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	69 def end(self, rev):
41079 88d813cd9acd revlog: fix pure python slicing test when chain contains nullrev Boris Feld <boris.feld@octobus.net> parents: 41033 diff changeset	70 if rev == nullrev:
88d813cd9acd revlog: fix pure python slicing test when chain contains nullrev Boris Feld <boris.feld@octobus.net> parents: 41033 diff changeset	71 return 0
38637 e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	72 return self._data[rev]
e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	73
e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	74 def length(self, rev):
e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	75 return self.end(rev) - self.start(rev)
e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	76
38718 f8762ea73e0d sparse-revlog: implement algorithm to write sparse delta chains (issue5480) Paul Morelle <paul.morelle@octobus.net> parents: 38717 diff changeset	77 def __len__(self):
f8762ea73e0d sparse-revlog: implement algorithm to write sparse delta chains (issue5480) Paul Morelle <paul.morelle@octobus.net> parents: 38717 diff changeset	78 return len(self._data)
f8762ea73e0d sparse-revlog: implement algorithm to write sparse delta chains (issue5480) Paul Morelle <paul.morelle@octobus.net> parents: 38717 diff changeset	79
40641 85b14f0dc334 doctest: add a `issnapshot` method to _testrevlog Boris Feld <boris.feld@octobus.net> parents: 40608 diff changeset	80 def issnapshot(self, rev):
41079 88d813cd9acd revlog: fix pure python slicing test when chain contains nullrev Boris Feld <boris.feld@octobus.net> parents: 41033 diff changeset	81 if rev == nullrev:
88d813cd9acd revlog: fix pure python slicing test when chain contains nullrev Boris Feld <boris.feld@octobus.net> parents: 41033 diff changeset	82 return True
40641 85b14f0dc334 doctest: add a `issnapshot` method to _testrevlog Boris Feld <boris.feld@octobus.net> parents: 40608 diff changeset	83 return rev in self._snapshot
85b14f0dc334 doctest: add a `issnapshot` method to _testrevlog Boris Feld <boris.feld@octobus.net> parents: 40608 diff changeset	84
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	85
40604 3ac23dad6364 sparse-revlog: drop unused deltainfo parameter from _slicechunktodensity Boris Feld <boris.feld@octobus.net> parents: 40603 diff changeset	86 def slicechunk(revlog, revs, targetsize=None):
34824 e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	87 """slice revs to reduce the amount of unrelated data to be read from disk.
e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	88
e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	89 ``revs`` is sliced into groups that should be read in one time.
e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	90 Assume that revs are sorted.
38640 f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	91
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	92 The initial chunk is sliced until the overall density (payload/chunks-span
51059 8c614fa16330 revlog: remove legacy usage of `_srdensitythreshold` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51057 diff changeset	93 ratio) is above `revlog.data_config.sr_density_threshold`. No gap smaller
51060 533d6943f6a3 revlog: remove legacy usage of `_srmingapsize` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51059 diff changeset	94 than `revlog.data_config.sr_min_gap_size` is skipped.
38640 f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	95
38643 967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size Boris Feld <boris.feld@octobus.net> parents: 38642 diff changeset	96 If `targetsize` is set, no chunk larger than `targetsize` will be yield.
967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size Boris Feld <boris.feld@octobus.net> parents: 38642 diff changeset	97 For consistency with other slicing choice, this limit won't go lower than
51060 533d6943f6a3 revlog: remove legacy usage of `_srmingapsize` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51059 diff changeset	98 `revlog.data_config.sr_min_gap_size`.
38643 967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size Boris Feld <boris.feld@octobus.net> parents: 38642 diff changeset	99
967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size Boris Feld <boris.feld@octobus.net> parents: 38642 diff changeset	100 If individual revisions chunk are larger than this limit, they will still
967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size Boris Feld <boris.feld@octobus.net> parents: 38642 diff changeset	101 be raised individually.
967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size Boris Feld <boris.feld@octobus.net> parents: 38642 diff changeset	102
40642 9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	103 >>> data = [
38640 f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	104 ... 5, #00 (5)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	105 ... 10, #01 (5)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	106 ... 12, #02 (2)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	107 ... 12, #03 (empty)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	108 ... 27, #04 (15)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	109 ... 31, #05 (4)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	110 ... 31, #06 (empty)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	111 ... 42, #07 (11)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	112 ... 47, #08 (5)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	113 ... 47, #09 (empty)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	114 ... 48, #10 (1)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	115 ... 51, #11 (3)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	116 ... 74, #12 (23)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	117 ... 85, #13 (11)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	118 ... 86, #14 (1)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	119 ... 91, #15 (5)
40642 9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	120 ... ]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	121 >>> revlog = _testrevlog(data, snapshot=range(16))
38640 f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	122
39330 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	123 >>> list(slicechunk(revlog, list(range(16))))
38640 f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	124 [[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]]
39330 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	125 >>> list(slicechunk(revlog, [0, 15]))
38640 f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	126 [[0], [15]]
39330 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	127 >>> list(slicechunk(revlog, [0, 11, 15]))
38640 f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	128 [[0], [11], [15]]
39330 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	129 >>> list(slicechunk(revlog, [0, 11, 13, 15]))
38640 f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	130 [[0], [11, 13, 15]]
39330 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	131 >>> list(slicechunk(revlog, [1, 2, 3, 5, 8, 10, 11, 14]))
38640 f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	132 [[1, 2], [5, 8, 10, 11], [14]]
38643 967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size Boris Feld <boris.feld@octobus.net> parents: 38642 diff changeset	133
967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size Boris Feld <boris.feld@octobus.net> parents: 38642 diff changeset	134 Slicing with a maximum chunk size
39330 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	135 >>> list(slicechunk(revlog, [0, 11, 13, 15], targetsize=15))
38643 967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size Boris Feld <boris.feld@octobus.net> parents: 38642 diff changeset	136 [[0], [11], [13], [15]]
39330 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	137 >>> list(slicechunk(revlog, [0, 11, 13, 15], targetsize=20))
38643 967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size Boris Feld <boris.feld@octobus.net> parents: 38642 diff changeset	138 [[0], [11], [13, 15]]
41079 88d813cd9acd revlog: fix pure python slicing test when chain contains nullrev Boris Feld <boris.feld@octobus.net> parents: 41033 diff changeset	139
88d813cd9acd revlog: fix pure python slicing test when chain contains nullrev Boris Feld <boris.feld@octobus.net> parents: 41033 diff changeset	140 Slicing involving nullrev
88d813cd9acd revlog: fix pure python slicing test when chain contains nullrev Boris Feld <boris.feld@octobus.net> parents: 41033 diff changeset	141 >>> list(slicechunk(revlog, [-1, 0, 11, 13, 15], targetsize=20))
88d813cd9acd revlog: fix pure python slicing test when chain contains nullrev Boris Feld <boris.feld@octobus.net> parents: 41033 diff changeset	142 [[-1, 0], [11], [13, 15]]
88d813cd9acd revlog: fix pure python slicing test when chain contains nullrev Boris Feld <boris.feld@octobus.net> parents: 41033 diff changeset	143 >>> list(slicechunk(revlog, [-1, 13, 15], targetsize=5))
88d813cd9acd revlog: fix pure python slicing test when chain contains nullrev Boris Feld <boris.feld@octobus.net> parents: 41033 diff changeset	144 [[-1], [13], [15]]
34824 e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	145 """
38643 967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size Boris Feld <boris.feld@octobus.net> parents: 38642 diff changeset	146 if targetsize is not None:
51060 533d6943f6a3 revlog: remove legacy usage of `_srmingapsize` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51059 diff changeset	147 targetsize = max(targetsize, revlog.data_config.sr_min_gap_size)
38718 f8762ea73e0d sparse-revlog: implement algorithm to write sparse delta chains (issue5480) Paul Morelle <paul.morelle@octobus.net> parents: 38717 diff changeset	148 # targetsize should not be specified when evaluating delta candidates:
f8762ea73e0d sparse-revlog: implement algorithm to write sparse delta chains (issue5480) Paul Morelle <paul.morelle@octobus.net> parents: 38717 diff changeset	149 # * targetsize is used to ensure we stay within specification when reading,
40709 39d29542fe40 sparse-revlog: put the native implementation of slicechunktodensity to use Boris Feld <boris.feld@octobus.net> parents: 40657 diff changeset	150 densityslicing = getattr(revlog.index, 'slicechunktodensity', None)
39d29542fe40 sparse-revlog: put the native implementation of slicechunktodensity to use Boris Feld <boris.feld@octobus.net> parents: 40657 diff changeset	151 if densityslicing is None:
39d29542fe40 sparse-revlog: put the native implementation of slicechunktodensity to use Boris Feld <boris.feld@octobus.net> parents: 40657 diff changeset	152 densityslicing = lambda x, y, z: _slicechunktodensity(revlog, x, y, z)
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	153 for chunk in densityslicing(
51060 533d6943f6a3 revlog: remove legacy usage of `_srmingapsize` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51059 diff changeset	154 revs,
533d6943f6a3 revlog: remove legacy usage of `_srmingapsize` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51059 diff changeset	155 revlog.data_config.sr_density_threshold,
533d6943f6a3 revlog: remove legacy usage of `_srmingapsize` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51059 diff changeset	156 revlog.data_config.sr_min_gap_size,
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	157 ):
38643 967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size Boris Feld <boris.feld@octobus.net> parents: 38642 diff changeset	158 for subchunk in _slicechunktosize(revlog, chunk, targetsize):
967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size Boris Feld <boris.feld@octobus.net> parents: 38642 diff changeset	159 yield subchunk
38641 feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	160
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	161
38718 f8762ea73e0d sparse-revlog: implement algorithm to write sparse delta chains (issue5480) Paul Morelle <paul.morelle@octobus.net> parents: 38717 diff changeset	162 def _slicechunktosize(revlog, revs, targetsize=None):
38642 e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	163 """slice revs to match the target size
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	164
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	165 This is intended to be used on chunk that density slicing selected by that
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	166 are still too large compared to the read garantee of revlog. This might
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	167 happens when "minimal gap size" interrupted the slicing or when chain are
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	168 built in a way that create large blocks next to each other.
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	169
40642 9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	170 >>> data = [
38642 e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	171 ... 3, #0 (3)
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	172 ... 5, #1 (2)
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	173 ... 6, #2 (1)
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	174 ... 8, #3 (2)
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	175 ... 8, #4 (empty)
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	176 ... 11, #5 (3)
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	177 ... 12, #6 (1)
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	178 ... 13, #7 (1)
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	179 ... 14, #8 (1)
40642 9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	180 ... ]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	181
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	182 == All snapshots cases ==
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	183 >>> revlog = _testrevlog(data, snapshot=range(9))
38642 e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	184
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	185 Cases where chunk is already small enough
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	186 >>> list(_slicechunktosize(revlog, [0], 3))
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	187 [[0]]
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	188 >>> list(_slicechunktosize(revlog, [6, 7], 3))
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	189 [[6, 7]]
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	190 >>> list(_slicechunktosize(revlog, [0], None))
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	191 [[0]]
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	192 >>> list(_slicechunktosize(revlog, [6, 7], None))
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	193 [[6, 7]]
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	194
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	195 cases where we need actual slicing
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	196 >>> list(_slicechunktosize(revlog, [0, 1], 3))
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	197 [[0], [1]]
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	198 >>> list(_slicechunktosize(revlog, [1, 3], 3))
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	199 [[1], [3]]
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	200 >>> list(_slicechunktosize(revlog, [1, 2, 3], 3))
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	201 [[1, 2], [3]]
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	202 >>> list(_slicechunktosize(revlog, [3, 5], 3))
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	203 [[3], [5]]
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	204 >>> list(_slicechunktosize(revlog, [3, 4, 5], 3))
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	205 [[3], [5]]
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	206 >>> list(_slicechunktosize(revlog, [5, 6, 7, 8], 3))
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	207 [[5], [6, 7, 8]]
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	208 >>> list(_slicechunktosize(revlog, [0, 1, 2, 3, 4, 5, 6, 7, 8], 3))
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	209 [[0], [1, 2], [3], [5], [6, 7, 8]]
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	210
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	211 Case with too large individual chunk (must return valid chunk)
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	212 >>> list(_slicechunktosize(revlog, [0, 1], 2))
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	213 [[0], [1]]
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	214 >>> list(_slicechunktosize(revlog, [1, 3], 1))
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	215 [[1], [3]]
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	216 >>> list(_slicechunktosize(revlog, [3, 4, 5], 2))
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	217 [[3], [5]]
40642 9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	218
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	219 == No Snapshot cases ==
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	220 >>> revlog = _testrevlog(data)
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	221
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	222 Cases where chunk is already small enough
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	223 >>> list(_slicechunktosize(revlog, [0], 3))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	224 [[0]]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	225 >>> list(_slicechunktosize(revlog, [6, 7], 3))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	226 [[6, 7]]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	227 >>> list(_slicechunktosize(revlog, [0], None))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	228 [[0]]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	229 >>> list(_slicechunktosize(revlog, [6, 7], None))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	230 [[6, 7]]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	231
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	232 cases where we need actual slicing
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	233 >>> list(_slicechunktosize(revlog, [0, 1], 3))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	234 [[0], [1]]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	235 >>> list(_slicechunktosize(revlog, [1, 3], 3))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	236 [[1], [3]]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	237 >>> list(_slicechunktosize(revlog, [1, 2, 3], 3))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	238 [[1], [2, 3]]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	239 >>> list(_slicechunktosize(revlog, [3, 5], 3))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	240 [[3], [5]]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	241 >>> list(_slicechunktosize(revlog, [3, 4, 5], 3))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	242 [[3], [4, 5]]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	243 >>> list(_slicechunktosize(revlog, [5, 6, 7, 8], 3))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	244 [[5], [6, 7, 8]]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	245 >>> list(_slicechunktosize(revlog, [0, 1, 2, 3, 4, 5, 6, 7, 8], 3))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	246 [[0], [1, 2], [3], [5], [6, 7, 8]]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	247
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	248 Case with too large individual chunk (must return valid chunk)
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	249 >>> list(_slicechunktosize(revlog, [0, 1], 2))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	250 [[0], [1]]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	251 >>> list(_slicechunktosize(revlog, [1, 3], 1))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	252 [[1], [3]]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	253 >>> list(_slicechunktosize(revlog, [3, 4, 5], 2))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	254 [[3], [5]]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	255
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	256 == mixed case ==
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	257 >>> revlog = _testrevlog(data, snapshot=[0, 1, 2])
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	258 >>> list(_slicechunktosize(revlog, list(range(9)), 5))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	259 [[0, 1], [2], [3, 4, 5], [6, 7, 8]]
38642 e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	260 """
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	261 assert targetsize is None or 0 <= targetsize
40642 9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	262 startdata = revlog.start(revs[0])
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	263 enddata = revlog.end(revs[-1])
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	264 fullspan = enddata - startdata
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	265 if targetsize is None or fullspan <= targetsize:
38642 e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	266 yield revs
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	267 return
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	268
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	269 startrevidx = 0
40657 2eb48aa0acce sparse-revlog: align endrevidx usages in the _slicechunktosize Boris Feld <boris.feld@octobus.net> parents: 40654 diff changeset	270 endrevidx = 1
38642 e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	271 iterrevs = enumerate(revs)
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	272 next(iterrevs) # skip first rev.
40642 9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	273 # first step: get snapshots out of the way
38642 e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	274 for idx, r in iterrevs:
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	275 span = revlog.end(r) - startdata
40642 9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	276 snapshot = revlog.issnapshot(r)
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	277 if span <= targetsize and snapshot:
40657 2eb48aa0acce sparse-revlog: align endrevidx usages in the _slicechunktosize Boris Feld <boris.feld@octobus.net> parents: 40654 diff changeset	278 endrevidx = idx + 1
38642 e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	279 else:
40657 2eb48aa0acce sparse-revlog: align endrevidx usages in the _slicechunktosize Boris Feld <boris.feld@octobus.net> parents: 40654 diff changeset	280 chunk = _trimchunk(revlog, revs, startrevidx, endrevidx)
38642 e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	281 if chunk:
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	282 yield chunk
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	283 startrevidx = idx
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	284 startdata = revlog.start(r)
40657 2eb48aa0acce sparse-revlog: align endrevidx usages in the _slicechunktosize Boris Feld <boris.feld@octobus.net> parents: 40654 diff changeset	285 endrevidx = idx + 1
40642 9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	286 if not snapshot:
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	287 break
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	288
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	289 # for the others, we use binary slicing to quickly converge toward valid
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	290 # chunks (otherwise, we might end up looking for start/end of many
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	291 # revisions). This logic is not looking for the perfect slicing point, it
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	292 # focuses on quickly converging toward valid chunks.
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	293 nbitem = len(revs)
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	294 while (enddata - startdata) > targetsize:
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	295 endrevidx = nbitem
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	296 if nbitem - startrevidx <= 1:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	297 break # protect against individual chunk larger than limit
40642 9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	298 localenddata = revlog.end(revs[endrevidx - 1])
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	299 span = localenddata - startdata
40654 fd1d41ccbe38 sparse-revlog: use `span` variable as intended Boris Feld <boris.feld@octobus.net> parents: 40642 diff changeset	300 while span > targetsize:
40642 9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	301 if endrevidx - startrevidx <= 1:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	302 break # protect against individual chunk larger than limit
40642 9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	303 endrevidx -= (endrevidx - startrevidx) // 2
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	304 localenddata = revlog.end(revs[endrevidx - 1])
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	305 span = localenddata - startdata
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	306 chunk = _trimchunk(revlog, revs, startrevidx, endrevidx)
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	307 if chunk:
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	308 yield chunk
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	309 startrevidx = endrevidx
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	310 startdata = revlog.start(revs[startrevidx])
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	311
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	312 chunk = _trimchunk(revlog, revs, startrevidx)
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	313 if chunk:
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	314 yield chunk
38642 e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	315
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	316
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	317 def _slicechunktodensity(revlog, revs, targetdensity=0.5, mingapsize=0):
38641 feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	318 """slice revs to reduce the amount of unrelated data to be read from disk.
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	319
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	320 ``revs`` is sliced into groups that should be read in one time.
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	321 Assume that revs are sorted.
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	322
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	323 The initial chunk is sliced until the overall density (payload/chunks-span
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	324 ratio) is above `targetdensity`. No gap smaller than `mingapsize` is
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	325 skipped.
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	326
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	327 >>> revlog = _testrevlog([
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	328 ... 5, #00 (5)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	329 ... 10, #01 (5)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	330 ... 12, #02 (2)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	331 ... 12, #03 (empty)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	332 ... 27, #04 (15)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	333 ... 31, #05 (4)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	334 ... 31, #06 (empty)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	335 ... 42, #07 (11)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	336 ... 47, #08 (5)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	337 ... 47, #09 (empty)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	338 ... 48, #10 (1)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	339 ... 51, #11 (3)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	340 ... 74, #12 (23)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	341 ... 85, #13 (11)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	342 ... 86, #14 (1)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	343 ... 91, #15 (5)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	344 ... ])
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	345
38655 cd1c484e31e8 revlog: adjust doctest examples to be portable to Python 3 Augie Fackler <augie@google.com> parents: 38644 diff changeset	346 >>> list(_slicechunktodensity(revlog, list(range(16))))
38641 feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	347 [[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]]
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	348 >>> list(_slicechunktodensity(revlog, [0, 15]))
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	349 [[0], [15]]
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	350 >>> list(_slicechunktodensity(revlog, [0, 11, 15]))
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	351 [[0], [11], [15]]
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	352 >>> list(_slicechunktodensity(revlog, [0, 11, 13, 15]))
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	353 [[0], [11, 13, 15]]
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	354 >>> list(_slicechunktodensity(revlog, [1, 2, 3, 5, 8, 10, 11, 14]))
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	355 [[1, 2], [5, 8, 10, 11], [14]]
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	356 >>> list(_slicechunktodensity(revlog, [1, 2, 3, 5, 8, 10, 11, 14],
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	357 ... mingapsize=20))
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	358 [[1, 2, 3, 5, 8, 10, 11], [14]]
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	359 >>> list(_slicechunktodensity(revlog, [1, 2, 3, 5, 8, 10, 11, 14],
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	360 ... targetdensity=0.95))
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	361 [[1, 2], [5], [8, 10, 11], [14]]
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	362 >>> list(_slicechunktodensity(revlog, [1, 2, 3, 5, 8, 10, 11, 14],
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	363 ... targetdensity=0.95, mingapsize=12))
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	364 [[1, 2], [5, 8, 10, 11], [14]]
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	365 """
34824 e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	366 start = revlog.start
e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	367 length = revlog.length
e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	368
34880 9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	369 if len(revs) <= 1:
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	370 yield revs
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	371 return
34824 e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	372
40604 3ac23dad6364 sparse-revlog: drop unused deltainfo parameter from _slicechunktodensity Boris Feld <boris.feld@octobus.net> parents: 40603 diff changeset	373 deltachainspan = segmentspan(revlog, revs)
34880 9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	374
38641 feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	375 if deltachainspan < mingapsize:
38635 d083ae26c325 revlog: early return in _slicechunk when span is already small enough Boris Feld <boris.feld@octobus.net> parents: 38634 diff changeset	376 yield revs
d083ae26c325 revlog: early return in _slicechunk when span is already small enough Boris Feld <boris.feld@octobus.net> parents: 38634 diff changeset	377 return
d083ae26c325 revlog: early return in _slicechunk when span is already small enough Boris Feld <boris.feld@octobus.net> parents: 38634 diff changeset	378
38718 f8762ea73e0d sparse-revlog: implement algorithm to write sparse delta chains (issue5480) Paul Morelle <paul.morelle@octobus.net> parents: 38717 diff changeset	379 readdata = deltachainspan
40606 bfbfd15d65bd sparse-revlog: fast-path before computing payload size Boris Feld <boris.feld@octobus.net> parents: 40605 diff changeset	380 chainpayload = sum(length(r) for r in revs)
34824 e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	381
34880 9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	382 if deltachainspan:
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	383 density = chainpayload / float(deltachainspan)
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	384 else:
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	385 density = 1.0
34824 e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	386
38641 feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	387 if density >= targetdensity:
38634 f0ea8b847831 revlog: early return in _slicechunk when density is already good Paul Morelle <paul.morelle@octobus.net> parents: 38632 diff changeset	388 yield revs
f0ea8b847831 revlog: early return in _slicechunk when density is already good Paul Morelle <paul.morelle@octobus.net> parents: 38632 diff changeset	389 return
f0ea8b847831 revlog: early return in _slicechunk when density is already good Paul Morelle <paul.morelle@octobus.net> parents: 38632 diff changeset	390
34880 9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	391 # Store the gaps in a heap to have them sorted by decreasing size
40607 54de23400b2a sparse-revlog: stop using a heap to track gaps Boris Feld <boris.feld@octobus.net> parents: 40606 diff changeset	392 gaps = []
34880 9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	393 prevend = None
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	394 for i, rev in enumerate(revs):
40604 3ac23dad6364 sparse-revlog: drop unused deltainfo parameter from _slicechunktodensity Boris Feld <boris.feld@octobus.net> parents: 40603 diff changeset	395 revstart = start(rev)
3ac23dad6364 sparse-revlog: drop unused deltainfo parameter from _slicechunktodensity Boris Feld <boris.feld@octobus.net> parents: 40603 diff changeset	396 revlen = length(rev)
34824 e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	397
34898 1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk Paul Morelle <paul.morelle@octobus.net> parents: 34881 diff changeset	398 # Skip empty revisions to form larger holes
1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk Paul Morelle <paul.morelle@octobus.net> parents: 34881 diff changeset	399 if revlen == 0:
1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk Paul Morelle <paul.morelle@octobus.net> parents: 34881 diff changeset	400 continue
1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk Paul Morelle <paul.morelle@octobus.net> parents: 34881 diff changeset	401
34880 9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	402 if prevend is not None:
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	403 gapsize = revstart - prevend
34881 8c9b08a0c48c sparse-read: skip gaps too small to be worth splitting Paul Morelle <paul.morelle@octobus.net> parents: 34880 diff changeset	404 # only consider holes that are large enough
38641 feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	405 if gapsize > mingapsize:
40607 54de23400b2a sparse-revlog: stop using a heap to track gaps Boris Feld <boris.feld@octobus.net> parents: 40606 diff changeset	406 gaps.append((gapsize, i))
34880 9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	407
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	408 prevend = revstart + revlen
40607 54de23400b2a sparse-revlog: stop using a heap to track gaps Boris Feld <boris.feld@octobus.net> parents: 40606 diff changeset	409 # sort the gaps to pop them from largest to small
54de23400b2a sparse-revlog: stop using a heap to track gaps Boris Feld <boris.feld@octobus.net> parents: 40606 diff changeset	410 gaps.sort()
34880 9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	411
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	412 # Collect the indices of the largest holes until the density is acceptable
40608 526ee887c4d5 sparse-revlog: stop using a heap to track selected gap Boris Feld <boris.feld@octobus.net> parents: 40607 diff changeset	413 selected = []
40607 54de23400b2a sparse-revlog: stop using a heap to track gaps Boris Feld <boris.feld@octobus.net> parents: 40606 diff changeset	414 while gaps and density < targetdensity:
54de23400b2a sparse-revlog: stop using a heap to track gaps Boris Feld <boris.feld@octobus.net> parents: 40606 diff changeset	415 gapsize, gapidx = gaps.pop()
34880 9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	416
40608 526ee887c4d5 sparse-revlog: stop using a heap to track selected gap Boris Feld <boris.feld@octobus.net> parents: 40607 diff changeset	417 selected.append(gapidx)
34824 e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	418
34880 9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	419 # the gap sizes are stored as negatives to be sorted decreasingly
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	420 # by the heap
40607 54de23400b2a sparse-revlog: stop using a heap to track gaps Boris Feld <boris.feld@octobus.net> parents: 40606 diff changeset	421 readdata -= gapsize
34880 9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	422 if readdata > 0:
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	423 density = chainpayload / float(readdata)
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	424 else:
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	425 density = 1.0
40608 526ee887c4d5 sparse-revlog: stop using a heap to track selected gap Boris Feld <boris.feld@octobus.net> parents: 40607 diff changeset	426 selected.sort()
34824 e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	427
34880 9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	428 # Cut the revs at collected indices
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	429 previdx = 0
40608 526ee887c4d5 sparse-revlog: stop using a heap to track selected gap Boris Feld <boris.feld@octobus.net> parents: 40607 diff changeset	430 for idx in selected:
34898 1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk Paul Morelle <paul.morelle@octobus.net> parents: 34881 diff changeset	431
1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk Paul Morelle <paul.morelle@octobus.net> parents: 34881 diff changeset	432 chunk = _trimchunk(revlog, revs, previdx, idx)
1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk Paul Morelle <paul.morelle@octobus.net> parents: 34881 diff changeset	433 if chunk:
1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk Paul Morelle <paul.morelle@octobus.net> parents: 34881 diff changeset	434 yield chunk
1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk Paul Morelle <paul.morelle@octobus.net> parents: 34881 diff changeset	435
34880 9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	436 previdx = idx
34898 1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk Paul Morelle <paul.morelle@octobus.net> parents: 34881 diff changeset	437
1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk Paul Morelle <paul.morelle@octobus.net> parents: 34881 diff changeset	438 chunk = _trimchunk(revlog, revs, previdx)
1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk Paul Morelle <paul.morelle@octobus.net> parents: 34881 diff changeset	439 if chunk:
1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk Paul Morelle <paul.morelle@octobus.net> parents: 34881 diff changeset	440 yield chunk
34824 e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	441
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	442
39330 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	443 def _trimchunk(revlog, revs, startidx, endidx=None):
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	444 """returns revs[startidx:endidx] without empty trailing revs
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	445
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	446 Doctest Setup
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	447 >>> revlog = _testrevlog([
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	448 ... 5, #0
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	449 ... 10, #1
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	450 ... 12, #2
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	451 ... 12, #3 (empty)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	452 ... 17, #4
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	453 ... 21, #5
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	454 ... 21, #6 (empty)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	455 ... ])
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	456
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	457 Contiguous cases:
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	458 >>> _trimchunk(revlog, [0, 1, 2, 3, 4, 5, 6], 0)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	459 [0, 1, 2, 3, 4, 5]
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	460 >>> _trimchunk(revlog, [0, 1, 2, 3, 4, 5, 6], 0, 5)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	461 [0, 1, 2, 3, 4]
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	462 >>> _trimchunk(revlog, [0, 1, 2, 3, 4, 5, 6], 0, 4)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	463 [0, 1, 2]
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	464 >>> _trimchunk(revlog, [0, 1, 2, 3, 4, 5, 6], 2, 4)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	465 [2]
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	466 >>> _trimchunk(revlog, [0, 1, 2, 3, 4, 5, 6], 3)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	467 [3, 4, 5]
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	468 >>> _trimchunk(revlog, [0, 1, 2, 3, 4, 5, 6], 3, 5)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	469 [3, 4]
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	470
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	471 Discontiguous cases:
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	472 >>> _trimchunk(revlog, [1, 3, 5, 6], 0)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	473 [1, 3, 5]
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	474 >>> _trimchunk(revlog, [1, 3, 5, 6], 0, 2)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	475 [1]
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	476 >>> _trimchunk(revlog, [1, 3, 5, 6], 1, 3)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	477 [3, 5]
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	478 >>> _trimchunk(revlog, [1, 3, 5, 6], 1)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	479 [3, 5]
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	480 """
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	481 length = revlog.length
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	482
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	483 if endidx is None:
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	484 endidx = len(revs)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	485
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	486 # If we have a non-emtpy delta candidate, there are nothing to trim
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	487 if revs[endidx - 1] < len(revlog):
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	488 # Trim empty revs at the end, except the very first revision of a chain
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	489 while (
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	490 endidx > 1 and endidx > startidx and length(revs[endidx - 1]) == 0
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	491 ):
39330 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	492 endidx -= 1
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	493
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	494 return revs[startidx:endidx]
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	495
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	496
40605 a32ccd32982b sparse-revlog: drop unused deltainfo parameter from segmentspan Boris Feld <boris.feld@octobus.net> parents: 40604 diff changeset	497 def segmentspan(revlog, revs):
39330 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	498 """Get the byte span of a segment of revisions
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	499
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	500 revs is a sorted array of revision numbers
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	501
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	502 >>> revlog = _testrevlog([
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	503 ... 5, #0
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	504 ... 10, #1
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	505 ... 12, #2
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	506 ... 12, #3 (empty)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	507 ... 17, #4
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	508 ... ])
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	509
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	510 >>> segmentspan(revlog, [0, 1, 2, 3, 4])
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	511 17
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	512 >>> segmentspan(revlog, [0, 4])
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	513 17
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	514 >>> segmentspan(revlog, [3, 4])
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	515 5
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	516 >>> segmentspan(revlog, [1, 2, 3,])
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	517 7
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	518 >>> segmentspan(revlog, [1, 3])
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	519 7
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	520 """
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	521 if not revs:
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	522 return 0
40605 a32ccd32982b sparse-revlog: drop unused deltainfo parameter from segmentspan Boris Feld <boris.feld@octobus.net> parents: 40604 diff changeset	523 end = revlog.end(revs[-1])
39330 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	524 return end - revlog.start(revs[0])
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	525
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	526
51017 509f0f7fc89e delta-computer: stop explicitly taking file handle Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50926 diff changeset	527 def _textfromdelta(revlog, baserev, delta, p1, p2, flags, expectednode):
39331 fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39330 diff changeset	528 """build full text from a (base, delta) pair and other metadata"""
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39330 diff changeset	529 # special case deltas which replace entire base; no need to decode
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39330 diff changeset	530 # base revision. this neatly avoids censored bases, which throw when
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39330 diff changeset	531 # they're decoded.
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	532 hlen = struct.calcsize(b">lll")
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	533 if delta[:hlen] == mdiff.replacediffheader(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	534 revlog.rawsize(baserev), len(delta) - hlen
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	535 ):
39331 fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39330 diff changeset	536 fulltext = delta[hlen:]
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39330 diff changeset	537 else:
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39330 diff changeset	538 # deltabase is rawtext before changed by flag processors, which is
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39330 diff changeset	539 # equivalent to non-raw text
51017 509f0f7fc89e delta-computer: stop explicitly taking file handle Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50926 diff changeset	540 basetext = revlog.revision(baserev)
39331 fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39330 diff changeset	541 fulltext = mdiff.patch(basetext, delta)
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39330 diff changeset	542
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39330 diff changeset	543 try:
42992 dff95420480f flagprocessors: make `processflagsraw` a module level function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42878 diff changeset	544 validatehash = flagutil.processflagsraw(revlog, fulltext, flags)
39331 fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39330 diff changeset	545 if validatehash:
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39330 diff changeset	546 revlog.checkhash(fulltext, expectednode, p1=p1, p2=p2)
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39330 diff changeset	547 if flags & REVIDX_ISCENSORED:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	548 raise error.StorageError(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	549 _(b'node %s is not censored') % expectednode
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	550 )
39774 4a2466b2a434 revlog: drop some more error aliases (API) Gregory Szorc <gregory.szorc@gmail.com> parents: 39773 diff changeset	551 except error.CensoredNodeError:
39331 fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39330 diff changeset	552 # must pass the censored index flag to add censored revisions
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39330 diff changeset	553 if not flags & REVIDX_ISCENSORED:
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39330 diff changeset	554 raise
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39330 diff changeset	555 return fulltext
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39330 diff changeset	556
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	557
35638 edc9330acac1 revlog: introduce 'deltainfo' to distinguish from 'delta' Paul Morelle <paul.morelle@octobus.net> parents: 35637 diff changeset	558 @attr.s(slots=True, frozen=True)
48946 642e31cb55f0 py3: use class X: instead of class X(object): Gregory Szorc <gregory.szorc@gmail.com> parents: 48875 diff changeset	559 class _deltainfo:
35638 edc9330acac1 revlog: introduce 'deltainfo' to distinguish from 'delta' Paul Morelle <paul.morelle@octobus.net> parents: 35637 diff changeset	560 distance = attr.ib()
edc9330acac1 revlog: introduce 'deltainfo' to distinguish from 'delta' Paul Morelle <paul.morelle@octobus.net> parents: 35637 diff changeset	561 deltalen = attr.ib()
edc9330acac1 revlog: introduce 'deltainfo' to distinguish from 'delta' Paul Morelle <paul.morelle@octobus.net> parents: 35637 diff changeset	562 data = attr.ib()
edc9330acac1 revlog: introduce 'deltainfo' to distinguish from 'delta' Paul Morelle <paul.morelle@octobus.net> parents: 35637 diff changeset	563 base = attr.ib()
edc9330acac1 revlog: introduce 'deltainfo' to distinguish from 'delta' Paul Morelle <paul.morelle@octobus.net> parents: 35637 diff changeset	564 chainbase = attr.ib()
edc9330acac1 revlog: introduce 'deltainfo' to distinguish from 'delta' Paul Morelle <paul.morelle@octobus.net> parents: 35637 diff changeset	565 chainlen = attr.ib()
edc9330acac1 revlog: introduce 'deltainfo' to distinguish from 'delta' Paul Morelle <paul.morelle@octobus.net> parents: 35637 diff changeset	566 compresseddeltalen = attr.ib()
39154 e0da43e2f71f revlog: compute snapshot depth on delta info Boris Feld <boris.feld@octobus.net> parents: 39152 diff changeset	567 snapshotdepth = attr.ib()
35638 edc9330acac1 revlog: introduce 'deltainfo' to distinguish from 'delta' Paul Morelle <paul.morelle@octobus.net> parents: 35637 diff changeset	568
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	569
47253 b876f0bf7366 revlog: introduce a plain compression mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46819 diff changeset	570 def drop_u_compression(delta):
b876f0bf7366 revlog: introduce a plain compression mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46819 diff changeset	571 """turn into a "u" (no-compression) into no-compression without header
b876f0bf7366 revlog: introduce a plain compression mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46819 diff changeset	572
b876f0bf7366 revlog: introduce a plain compression mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46819 diff changeset	573 This is useful for revlog format that has better compression method.
b876f0bf7366 revlog: introduce a plain compression mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46819 diff changeset	574 """
b876f0bf7366 revlog: introduce a plain compression mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46819 diff changeset	575 assert delta.data[0] == b'u', delta.data[0]
b876f0bf7366 revlog: introduce a plain compression mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46819 diff changeset	576 return _deltainfo(
b876f0bf7366 revlog: introduce a plain compression mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46819 diff changeset	577 delta.distance,
b876f0bf7366 revlog: introduce a plain compression mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46819 diff changeset	578 delta.deltalen - 1,
b876f0bf7366 revlog: introduce a plain compression mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46819 diff changeset	579 (b'', delta.data[1]),
b876f0bf7366 revlog: introduce a plain compression mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46819 diff changeset	580 delta.base,
b876f0bf7366 revlog: introduce a plain compression mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46819 diff changeset	581 delta.chainbase,
b876f0bf7366 revlog: introduce a plain compression mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46819 diff changeset	582 delta.chainlen,
b876f0bf7366 revlog: introduce a plain compression mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46819 diff changeset	583 delta.compresseddeltalen,
b876f0bf7366 revlog: introduce a plain compression mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46819 diff changeset	584 delta.snapshotdepth,
b876f0bf7366 revlog: introduce a plain compression mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46819 diff changeset	585 )
b876f0bf7366 revlog: introduce a plain compression mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46819 diff changeset	586
b876f0bf7366 revlog: introduce a plain compression mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46819 diff changeset	587
40978 42f59d3f714d delta: exclude base candidate much smaller than the target Boris Feld <boris.feld@octobus.net> parents: 40957 diff changeset	588 # If a revision's full text is that much bigger than a base candidate full
42f59d3f714d delta: exclude base candidate much smaller than the target Boris Feld <boris.feld@octobus.net> parents: 40957 diff changeset	589 # text's, it is very unlikely that it will produce a valid delta. We no longer
42f59d3f714d delta: exclude base candidate much smaller than the target Boris Feld <boris.feld@octobus.net> parents: 40957 diff changeset	590 # consider these candidates.
41033 b373477948df revlog: limit base to rev size ratio to 500 instead of 50 Boris Feld <boris.feld@octobus.net> parents: 40979 diff changeset	591 LIMIT_BASE2TEXT = 500
40978 42f59d3f714d delta: exclude base candidate much smaller than the target Boris Feld <boris.feld@octobus.net> parents: 40957 diff changeset	592
51346 cc806f20d756 delta-find: explicitly track stage of the search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51345 diff changeset	593 ### stage of the search, used for debug and to select and to adjust some logic.
cc806f20d756 delta-find: explicitly track stage of the search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51345 diff changeset	594 # initial stage, next step is unknown
cc806f20d756 delta-find: explicitly track stage of the search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51345 diff changeset	595 _STAGE_UNSPECIFIED = "unspecified"
cc806f20d756 delta-find: explicitly track stage of the search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51345 diff changeset	596 # trying the cached delta
cc806f20d756 delta-find: explicitly track stage of the search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51345 diff changeset	597 _STAGE_CACHED = "cached"
cc806f20d756 delta-find: explicitly track stage of the search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51345 diff changeset	598 # trying delta based on parents
cc806f20d756 delta-find: explicitly track stage of the search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51345 diff changeset	599 _STAGE_PARENTS = "parents"
cc806f20d756 delta-find: explicitly track stage of the search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51345 diff changeset	600 # trying to build a valid snapshot of any level
cc806f20d756 delta-find: explicitly track stage of the search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51345 diff changeset	601 _STAGE_SNAPSHOT = "snapshot"
cc806f20d756 delta-find: explicitly track stage of the search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51345 diff changeset	602 # trying to build a delta based of the previous revision
cc806f20d756 delta-find: explicitly track stage of the search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51345 diff changeset	603 _STAGE_PREV = "prev"
cc806f20d756 delta-find: explicitly track stage of the search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51345 diff changeset	604 # trying to build a full snapshot
cc806f20d756 delta-find: explicitly track stage of the search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51345 diff changeset	605 _STAGE_FULL = "full"
cc806f20d756 delta-find: explicitly track stage of the search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51345 diff changeset	606
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	607
51343 2a333d791ecf delta-find: introduce a base class for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51342 diff changeset	608 class _BaseDeltaSearch(abc.ABC):
51317 c82e03b102a6 delta-find: introduce a _DeltaSearch object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51316 diff changeset	609 """perform the search of a good delta for a single revlog revision
50354 ca1522fe4ec8 delta-find: assume the target-rev if not specified Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49776 diff changeset	610
51317 c82e03b102a6 delta-find: introduce a _DeltaSearch object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51316 diff changeset	611 note: some of the deltacomputer.finddeltainfo logic should probably move
c82e03b102a6 delta-find: introduce a _DeltaSearch object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51316 diff changeset	612 here.
c82e03b102a6 delta-find: introduce a _DeltaSearch object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51316 diff changeset	613 """
50355 0232571255d3 delta-find: never do anything fancy when general delta is off Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50354 diff changeset	614
51317 c82e03b102a6 delta-find: introduce a _DeltaSearch object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51316 diff changeset	615 def __init__(
c82e03b102a6 delta-find: introduce a _DeltaSearch object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51316 diff changeset	616 self,
49610 35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	617 revlog,
51327 49401b7dec0c delta-find: feed revinfo to _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51325 diff changeset	618 revinfo,
49610 35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	619 p1,
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	620 p2,
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	621 cachedelta,
51317 c82e03b102a6 delta-find: introduce a _DeltaSearch object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51316 diff changeset	622 excluded_bases=None,
c82e03b102a6 delta-find: introduce a _DeltaSearch object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51316 diff changeset	623 target_rev=None,
c82e03b102a6 delta-find: introduce a _DeltaSearch object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51316 diff changeset	624 snapshot_cache=None,
c82e03b102a6 delta-find: introduce a _DeltaSearch object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51316 diff changeset	625 ):
51322 c9ab452e5611 delta-find: check DELTA_BASE_REUSE_FORCE in the _DeltaSearch.__init__ Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51321 diff changeset	626 # the DELTA_BASE_REUSE_FORCE case should have been taken care of sooner
c9ab452e5611 delta-find: check DELTA_BASE_REUSE_FORCE in the _DeltaSearch.__init__ Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51321 diff changeset	627 # so we should never end up asking such question. Adding the assert as
c9ab452e5611 delta-find: check DELTA_BASE_REUSE_FORCE in the _DeltaSearch.__init__ Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51321 diff changeset	628 # a safe-guard to detect anything that would be fishy in this regard.
c9ab452e5611 delta-find: check DELTA_BASE_REUSE_FORCE in the _DeltaSearch.__init__ Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51321 diff changeset	629 assert (
c9ab452e5611 delta-find: check DELTA_BASE_REUSE_FORCE in the _DeltaSearch.__init__ Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51321 diff changeset	630 cachedelta is None
c9ab452e5611 delta-find: check DELTA_BASE_REUSE_FORCE in the _DeltaSearch.__init__ Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51321 diff changeset	631 or cachedelta[2] != DELTA_BASE_REUSE_FORCE
c9ab452e5611 delta-find: check DELTA_BASE_REUSE_FORCE in the _DeltaSearch.__init__ Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51321 diff changeset	632 or not revlog.delta_config.general_delta
c9ab452e5611 delta-find: check DELTA_BASE_REUSE_FORCE in the _DeltaSearch.__init__ Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51321 diff changeset	633 )
51317 c82e03b102a6 delta-find: introduce a _DeltaSearch object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51316 diff changeset	634 self.revlog = revlog
51327 49401b7dec0c delta-find: feed revinfo to _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51325 diff changeset	635 self.revinfo = revinfo
49401b7dec0c delta-find: feed revinfo to _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51325 diff changeset	636 self.textlen = revinfo.textlen
51317 c82e03b102a6 delta-find: introduce a _DeltaSearch object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51316 diff changeset	637 self.p1 = p1
c82e03b102a6 delta-find: introduce a _DeltaSearch object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51316 diff changeset	638 self.p2 = p2
c82e03b102a6 delta-find: introduce a _DeltaSearch object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51316 diff changeset	639 self.cachedelta = cachedelta
c82e03b102a6 delta-find: introduce a _DeltaSearch object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51316 diff changeset	640 self.excluded_bases = excluded_bases
51321 047358417d01 delta-find: move target_rev in the _DeltaSearch.__init__ Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51320 diff changeset	641 if target_rev is None:
047358417d01 delta-find: move target_rev in the _DeltaSearch.__init__ Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51320 diff changeset	642 self.target_rev = len(self.revlog)
51317 c82e03b102a6 delta-find: introduce a _DeltaSearch object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51316 diff changeset	643 self.target_rev = target_rev
51320 87fce139e927 delta-find: move snapshot_cache in the _DeltaSearch.__init__ Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51319 diff changeset	644 if snapshot_cache is None:
87fce139e927 delta-find: move snapshot_cache in the _DeltaSearch.__init__ Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51319 diff changeset	645 # map: base-rev: [snapshot-revs]
87fce139e927 delta-find: move snapshot_cache in the _DeltaSearch.__init__ Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51319 diff changeset	646 snapshot_cache = SnapshotCache()
51317 c82e03b102a6 delta-find: introduce a _DeltaSearch object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51316 diff changeset	647 self.snapshot_cache = snapshot_cache
c82e03b102a6 delta-find: introduce a _DeltaSearch object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51316 diff changeset	648
51323 01db705bbe32 delta-find: move tested in the _DeltaSearch.__init__ Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51322 diff changeset	649 self.tested = {nullrev}
01db705bbe32 delta-find: move tested in the _DeltaSearch.__init__ Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51322 diff changeset	650
51346 cc806f20d756 delta-find: explicitly track stage of the search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51345 diff changeset	651 self.current_stage = _STAGE_UNSPECIFIED
51343 2a333d791ecf delta-find: introduce a base class for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51342 diff changeset	652 self.current_group = None
2a333d791ecf delta-find: introduce a base class for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51342 diff changeset	653 self._init_group()
51331 d0d869fccd20 delta-find: move away from the generator API for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51330 diff changeset	654
51341 f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	655 def is_good_delta_info(self, deltainfo):
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	656 """Returns True if the given delta is good.
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	657
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	658 Good means that it is within the disk span, disk size, and chain length
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	659 bounds that we know to be performant.
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	660 """
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	661 if not self._is_good_delta_info_universal(deltainfo):
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	662 return False
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	663 if not self._is_good_delta_info_chain_quality(deltainfo):
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	664 return False
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	665 return True
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	666
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	667 def _is_good_delta_info_universal(self, deltainfo):
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	668 """Returns True if the given delta is good.
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	669
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	670 This performs generic checks needed by all format variants.
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	671
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	672 This is used by is_good_delta_info.
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	673 """
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	674
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	675 if deltainfo is None:
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	676 return False
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	677
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	678 # the DELTA_BASE_REUSE_FORCE case should have been taken care of sooner
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	679 # so we should never end up asking such question. Adding the assert as
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	680 # a safe-guard to detect anything that would be fishy in this regard.
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	681 assert (
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	682 self.revinfo.cachedelta is None
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	683 or self.revinfo.cachedelta[2] != DELTA_BASE_REUSE_FORCE
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	684 or not self.revlog.delta_config.general_delta
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	685 )
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	686
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	687 # Bad delta from new delta size:
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	688 #
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	689 # If the delta size is larger than the target text, storing the delta
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	690 # will be inefficient.
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	691 if self.revinfo.textlen < deltainfo.deltalen:
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	692 return False
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	693
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	694 return True
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	695
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	696 def _is_good_delta_info_chain_quality(self, deltainfo):
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	697 """Returns True if the chain associated with the delta is good.
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	698
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	699 This performs checks for format that use delta chains.
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	700
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	701 This is used by is_good_delta_info.
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	702 """
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	703 # - 'deltainfo.distance' is the distance from the base revision --
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	704 # bounding it limits the amount of I/O we need to do.
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	705
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	706 defaultmax = self.revinfo.textlen * 4
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	707 maxdist = self.revlog.delta_config.max_deltachain_span
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	708 if not maxdist:
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	709 maxdist = deltainfo.distance # ensure the conditional pass
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	710 maxdist = max(maxdist, defaultmax)
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	711
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	712 # Bad delta from read span:
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	713 #
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	714 # If the span of data read is larger than the maximum allowed.
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	715 #
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	716 # In the sparse-revlog case, we rely on the associated "sparse
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	717 # reading" to avoid issue related to the span of data. In theory, it
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	718 # would be possible to build pathological revlog where delta pattern
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	719 # would lead to too many reads. However, they do not happen in
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	720 # practice at all. So we skip the span check entirely.
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	721 if (
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	722 not self.revlog.delta_config.sparse_revlog
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	723 and maxdist < deltainfo.distance
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	724 ):
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	725 return False
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	726
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	727 # Bad delta from cumulated payload size:
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	728 #
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	729 # - 'deltainfo.compresseddeltalen' is the sum of the total size of
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	730 # deltas we need to apply -- bounding it limits the amount of CPU
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	731 # we consume.
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	732 max_chain_data = self.revinfo.textlen * LIMIT_DELTA2TEXT
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	733 # If the sum of delta get larger than K * target text length.
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	734 if max_chain_data < deltainfo.compresseddeltalen:
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	735 return False
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	736
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	737 # Bad delta from chain length:
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	738 #
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	739 # If the number of delta in the chain gets too high.
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	740 if (
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	741 self.revlog.delta_config.max_chain_len
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	742 and self.revlog.delta_config.max_chain_len < deltainfo.chainlen
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	743 ):
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	744 return False
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	745 return True
f3f35b37f4b2 delta-find: move good delta code earlier in the class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51340 diff changeset	746
51331 d0d869fccd20 delta-find: move away from the generator API for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51330 diff changeset	747 @property
d0d869fccd20 delta-find: move away from the generator API for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51330 diff changeset	748 def done(self):
d0d869fccd20 delta-find: move away from the generator API for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51330 diff changeset	749 """True when all possible candidate have been tested"""
d0d869fccd20 delta-find: move away from the generator API for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51330 diff changeset	750 return self.current_group is None
d0d869fccd20 delta-find: move away from the generator API for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51330 diff changeset	751
51343 2a333d791ecf delta-find: introduce a base class for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51342 diff changeset	752 @abc.abstractmethod
51331 d0d869fccd20 delta-find: move away from the generator API for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51330 diff changeset	753 def next_group(self, good_delta=None):
d0d869fccd20 delta-find: move away from the generator API for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51330 diff changeset	754 """move to the next group to test
d0d869fccd20 delta-find: move away from the generator API for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51330 diff changeset	755
d0d869fccd20 delta-find: move away from the generator API for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51330 diff changeset	756 The group of revision to test will be available in
d0d869fccd20 delta-find: move away from the generator API for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51330 diff changeset	757 `self.current_group`. If the previous group had any good delta, the
d0d869fccd20 delta-find: move away from the generator API for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51330 diff changeset	758 best one can be passed as the `good_delta` parameter to help selecting
d0d869fccd20 delta-find: move away from the generator API for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51330 diff changeset	759 the next group.
d0d869fccd20 delta-find: move away from the generator API for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51330 diff changeset	760
d0d869fccd20 delta-find: move away from the generator API for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51330 diff changeset	761 If not revision remains to be, `self.done` will be True and
d0d869fccd20 delta-find: move away from the generator API for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51330 diff changeset	762 `self.current_group` will be None.
d0d869fccd20 delta-find: move away from the generator API for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51330 diff changeset	763 """
51343 2a333d791ecf delta-find: introduce a base class for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51342 diff changeset	764 pass
2a333d791ecf delta-find: introduce a base class for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51342 diff changeset	765
2a333d791ecf delta-find: introduce a base class for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51342 diff changeset	766 @abc.abstractmethod
2a333d791ecf delta-find: introduce a base class for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51342 diff changeset	767 def _init_group(self):
2a333d791ecf delta-find: introduce a base class for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51342 diff changeset	768 pass
2a333d791ecf delta-find: introduce a base class for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51342 diff changeset	769
2a333d791ecf delta-find: introduce a base class for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51342 diff changeset	770
51344 555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51343 diff changeset	771 class _NoDeltaSearch(_BaseDeltaSearch):
555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51343 diff changeset	772 """Search for no delta.
555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51343 diff changeset	773
555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51343 diff changeset	774 This search variant is to be used in case where we should not store delta.
555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51343 diff changeset	775 """
555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51343 diff changeset	776
555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51343 diff changeset	777 def _init_group(self):
51346 cc806f20d756 delta-find: explicitly track stage of the search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51345 diff changeset	778 self.current_stage = _STAGE_FULL
51344 555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51343 diff changeset	779
555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51343 diff changeset	780 def next_group(self, good_delta=None):
555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51343 diff changeset	781 pass
555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51343 diff changeset	782
555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51343 diff changeset	783
555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51343 diff changeset	784 class _PrevDeltaSearch(_BaseDeltaSearch):
555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51343 diff changeset	785 """Search for delta against the previous revision only
555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51343 diff changeset	786
555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51343 diff changeset	787 This search variant is to be used when the format does not allow for delta
555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51343 diff changeset	788 against arbitrary bases.
555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51343 diff changeset	789 """
555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51343 diff changeset	790
555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51343 diff changeset	791 def _init_group(self):
51346 cc806f20d756 delta-find: explicitly track stage of the search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51345 diff changeset	792 self.current_stage = _STAGE_PREV
51344 555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51343 diff changeset	793 self.current_group = [self.target_rev - 1]
51348 bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51347 diff changeset	794 self.tested.update(self.current_group)
51344 555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51343 diff changeset	795
555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51343 diff changeset	796 def next_group(self, good_delta=None):
51346 cc806f20d756 delta-find: explicitly track stage of the search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51345 diff changeset	797 self.current_stage = _STAGE_FULL
51344 555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51343 diff changeset	798 self.current_group = None
555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51343 diff changeset	799
555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51343 diff changeset	800
51354 d7e2acdd50ba delta-find: split the _DeltaSearch class in two Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	801 class _GeneralDeltaSearch(_BaseDeltaSearch):
d7e2acdd50ba delta-find: split the _DeltaSearch class in two Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	802 """Delta search variant for general-delta repository"""
51344 555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51343 diff changeset	803
51343 2a333d791ecf delta-find: introduce a base class for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51342 diff changeset	804 def _init_group(self):
51344 555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51343 diff changeset	805 # Why search for delta base if we cannot use a delta base ?
555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51343 diff changeset	806 # also see issue6056
555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51343 diff changeset	807 assert self.revlog.delta_config.general_delta
51353 701caeabbee7 delta-find: finish reworking the snapshot logic and drop more layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51352 diff changeset	808 self._candidates_iterator = self._iter_groups()
51343 2a333d791ecf delta-find: introduce a base class for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51342 diff changeset	809 self._last_good = None
51349 7083b33a2699 delta-find: explicitly deal with usage of the cached revision Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51348 diff changeset	810 if (
7083b33a2699 delta-find: explicitly deal with usage of the cached revision Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51348 diff changeset	811 self.cachedelta is not None
7083b33a2699 delta-find: explicitly deal with usage of the cached revision Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51348 diff changeset	812 and self.cachedelta[2] > DELTA_BASE_REUSE_NO
7083b33a2699 delta-find: explicitly deal with usage of the cached revision Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51348 diff changeset	813 and self._pre_filter_rev(self.cachedelta[0])
7083b33a2699 delta-find: explicitly deal with usage of the cached revision Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51348 diff changeset	814 ):
7083b33a2699 delta-find: explicitly deal with usage of the cached revision Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51348 diff changeset	815 # First we try to reuse a the delta contained in the bundle. (or from
7083b33a2699 delta-find: explicitly deal with usage of the cached revision Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51348 diff changeset	816 # the source revlog)
7083b33a2699 delta-find: explicitly deal with usage of the cached revision Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51348 diff changeset	817 #
7083b33a2699 delta-find: explicitly deal with usage of the cached revision Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51348 diff changeset	818 # This logic only applies to general delta repositories and can be
7083b33a2699 delta-find: explicitly deal with usage of the cached revision Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51348 diff changeset	819 # disabled through configuration. Disabling reuse source delta is
7083b33a2699 delta-find: explicitly deal with usage of the cached revision Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51348 diff changeset	820 # useful when we want to make sure we recomputed "optimal" deltas.
7083b33a2699 delta-find: explicitly deal with usage of the cached revision Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51348 diff changeset	821 self.current_stage = _STAGE_CACHED
7083b33a2699 delta-find: explicitly deal with usage of the cached revision Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51348 diff changeset	822 self._internal_group = (self.cachedelta[0],)
7083b33a2699 delta-find: explicitly deal with usage of the cached revision Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51348 diff changeset	823 self._internal_idx = 0
7083b33a2699 delta-find: explicitly deal with usage of the cached revision Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51348 diff changeset	824 self.current_group = self._internal_group
7083b33a2699 delta-find: explicitly deal with usage of the cached revision Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51348 diff changeset	825 self.tested.update(self.current_group)
7083b33a2699 delta-find: explicitly deal with usage of the cached revision Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51348 diff changeset	826 else:
7083b33a2699 delta-find: explicitly deal with usage of the cached revision Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51348 diff changeset	827 self._next_internal_group()
51348 bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51347 diff changeset	828
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51347 diff changeset	829 def _next_internal_group(self):
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51347 diff changeset	830 # self._internal_group can be larger than self.current_group
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51347 diff changeset	831 self._internal_idx = 0
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51347 diff changeset	832 group = self._candidates_iterator.send(self._last_good)
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51347 diff changeset	833 if group is not None:
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51347 diff changeset	834 group = self._pre_filter_candidate_revs(group)
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51347 diff changeset	835 self._internal_group = group
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51347 diff changeset	836 if self._internal_group is None:
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51347 diff changeset	837 self.current_group = None
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51347 diff changeset	838 elif len(self._internal_group) == 0:
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51347 diff changeset	839 self.next_group()
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51347 diff changeset	840 else:
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51347 diff changeset	841 chunk_size = self.revlog.delta_config.candidate_group_chunk_size
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51347 diff changeset	842 if chunk_size > 0:
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51347 diff changeset	843 self.current_group = self._internal_group[:chunk_size]
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51347 diff changeset	844 self._internal_idx += chunk_size
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51347 diff changeset	845 else:
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51347 diff changeset	846 self.current_group = self._internal_group
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51347 diff changeset	847 self._internal_idx += len(self.current_group)
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51347 diff changeset	848
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51347 diff changeset	849 self.tested.update(self.current_group)
51343 2a333d791ecf delta-find: introduce a base class for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51342 diff changeset	850
2a333d791ecf delta-find: introduce a base class for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51342 diff changeset	851 def next_group(self, good_delta=None):
51348 bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51347 diff changeset	852 old_good = self._last_good
51331 d0d869fccd20 delta-find: move away from the generator API for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51330 diff changeset	853 if good_delta is not None:
d0d869fccd20 delta-find: move away from the generator API for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51330 diff changeset	854 self._last_good = good_delta.base
51349 7083b33a2699 delta-find: explicitly deal with usage of the cached revision Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51348 diff changeset	855 if self.current_stage == _STAGE_CACHED and good_delta is not None:
7083b33a2699 delta-find: explicitly deal with usage of the cached revision Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51348 diff changeset	856 # the cache is good, let us use the cache as requested
7083b33a2699 delta-find: explicitly deal with usage of the cached revision Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51348 diff changeset	857 self._candidates_iterator = None
7083b33a2699 delta-find: explicitly deal with usage of the cached revision Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51348 diff changeset	858 self._internal_group = None
7083b33a2699 delta-find: explicitly deal with usage of the cached revision Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51348 diff changeset	859 self._internal_idx = None
7083b33a2699 delta-find: explicitly deal with usage of the cached revision Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51348 diff changeset	860 self.current_group = None
7083b33a2699 delta-find: explicitly deal with usage of the cached revision Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51348 diff changeset	861 return
7083b33a2699 delta-find: explicitly deal with usage of the cached revision Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51348 diff changeset	862
51348 bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51347 diff changeset	863 if (self._internal_idx < len(self._internal_group)) and (
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51347 diff changeset	864 old_good != good_delta
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51347 diff changeset	865 ):
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51347 diff changeset	866 # When the size of the candidate group is big, it can result in
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51347 diff changeset	867 # a quite significant performance impact. To reduce this, we
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51347 diff changeset	868 # can send them in smaller batches until the new batch does not
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51347 diff changeset	869 # provide any improvements.
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51347 diff changeset	870 #
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51347 diff changeset	871 # This might reduce the overall efficiency of the compression
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51347 diff changeset	872 # in some corner cases, but that should also prevent very
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51347 diff changeset	873 # pathological cases from being an issue. (eg. 20 000
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51347 diff changeset	874 # candidates).
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51347 diff changeset	875 #
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51347 diff changeset	876 # XXX note that the ordering of the group becomes important as
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51347 diff changeset	877 # it now impacts the final result. The current order is
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51347 diff changeset	878 # unprocessed and can be improved.
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51347 diff changeset	879 next_idx = self._internal_idx + self._group_chunk_size
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51347 diff changeset	880 self.current_group = self._internal_group[
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51347 diff changeset	881 self._internal_idx : next_idx
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51347 diff changeset	882 ]
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51347 diff changeset	883 self.tested.update(self.current_group)
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51347 diff changeset	884 self._internal_idx = next_idx
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51347 diff changeset	885 else:
bbbfa00d1e67 delta-find: remove the "candidate groups" layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51347 diff changeset	886 self._next_internal_group()
39336 1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39335 diff changeset	887
51332 02cc19f4f091 delta-find: move pre-filtering of candidates in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51331 diff changeset	888 def _pre_filter_candidate_revs(self, temptative):
02cc19f4f091 delta-find: move pre-filtering of candidates in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51331 diff changeset	889 """filter possible candidate before computing a delta
02cc19f4f091 delta-find: move pre-filtering of candidates in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51331 diff changeset	890
02cc19f4f091 delta-find: move pre-filtering of candidates in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51331 diff changeset	891 This function use various criteria to pre-filter candidate delta base
02cc19f4f091 delta-find: move pre-filtering of candidates in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51331 diff changeset	892 before we compute a delta and evaluate its quality.
02cc19f4f091 delta-find: move pre-filtering of candidates in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51331 diff changeset	893
02cc19f4f091 delta-find: move pre-filtering of candidates in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51331 diff changeset	894 Such pre-filter limit the number of computed delta, an expensive operation.
02cc19f4f091 delta-find: move pre-filtering of candidates in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51331 diff changeset	895
02cc19f4f091 delta-find: move pre-filtering of candidates in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51331 diff changeset	896 return the updated list of revision to test
02cc19f4f091 delta-find: move pre-filtering of candidates in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51331 diff changeset	897 """
02cc19f4f091 delta-find: move pre-filtering of candidates in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51331 diff changeset	898 deltalength = self.revlog.length
02cc19f4f091 delta-find: move pre-filtering of candidates in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51331 diff changeset	899 deltaparent = self.revlog.deltaparent
02cc19f4f091 delta-find: move pre-filtering of candidates in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51331 diff changeset	900
02cc19f4f091 delta-find: move pre-filtering of candidates in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51331 diff changeset	901 tested = self.tested
02cc19f4f091 delta-find: move pre-filtering of candidates in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51331 diff changeset	902 group = []
02cc19f4f091 delta-find: move pre-filtering of candidates in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51331 diff changeset	903 for rev in temptative:
02cc19f4f091 delta-find: move pre-filtering of candidates in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51331 diff changeset	904 # skip over empty delta (no need to include them in a chain)
02cc19f4f091 delta-find: move pre-filtering of candidates in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51331 diff changeset	905 while not (rev == nullrev or rev in tested or deltalength(rev)):
02cc19f4f091 delta-find: move pre-filtering of candidates in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51331 diff changeset	906 tested.add(rev)
02cc19f4f091 delta-find: move pre-filtering of candidates in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51331 diff changeset	907 rev = deltaparent(rev)
51333 898c212e1b2f delta-find: move pre-filtering of individual revision in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51332 diff changeset	908 if self._pre_filter_rev(rev):
898c212e1b2f delta-find: move pre-filtering of individual revision in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51332 diff changeset	909 group.append(rev)
898c212e1b2f delta-find: move pre-filtering of individual revision in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51332 diff changeset	910 else:
898c212e1b2f delta-find: move pre-filtering of individual revision in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51332 diff changeset	911 self.tested.add(rev)
898c212e1b2f delta-find: move pre-filtering of individual revision in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51332 diff changeset	912 return group
898c212e1b2f delta-find: move pre-filtering of individual revision in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51332 diff changeset	913
51335 2407af4f2eea delta-find: split the generic part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51334 diff changeset	914 def _pre_filter_rev_universal(self, rev):
2407af4f2eea delta-find: split the generic part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51334 diff changeset	915 """pre filtering that is need in all cases.
2407af4f2eea delta-find: split the generic part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51334 diff changeset	916
2407af4f2eea delta-find: split the generic part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51334 diff changeset	917 return True if it seems okay to test a rev, False otherwise.
2407af4f2eea delta-find: split the generic part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51334 diff changeset	918
2407af4f2eea delta-find: split the generic part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51334 diff changeset	919 used by _pre_filter_rev.
2407af4f2eea delta-find: split the generic part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51334 diff changeset	920 """
51334 ac8b798e602b delta-find: drop the temporary indent Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51333 diff changeset	921 # no need to try a delta against nullrev, this will be done as
ac8b798e602b delta-find: drop the temporary indent Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51333 diff changeset	922 # a last resort.
ac8b798e602b delta-find: drop the temporary indent Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51333 diff changeset	923 if rev == nullrev:
ac8b798e602b delta-find: drop the temporary indent Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51333 diff changeset	924 return False
ac8b798e602b delta-find: drop the temporary indent Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51333 diff changeset	925 # filter out revision we tested already
ac8b798e602b delta-find: drop the temporary indent Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51333 diff changeset	926 if rev in self.tested:
ac8b798e602b delta-find: drop the temporary indent Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51333 diff changeset	927 return False
51332 02cc19f4f091 delta-find: move pre-filtering of candidates in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51331 diff changeset	928
51334 ac8b798e602b delta-find: drop the temporary indent Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51333 diff changeset	929 # an higher authority deamed the base unworthy (e.g. censored)
ac8b798e602b delta-find: drop the temporary indent Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51333 diff changeset	930 if self.excluded_bases is not None and rev in self.excluded_bases:
ac8b798e602b delta-find: drop the temporary indent Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51333 diff changeset	931 return False
ac8b798e602b delta-find: drop the temporary indent Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51333 diff changeset	932 # We are in some recomputation cases and that rev is too high
ac8b798e602b delta-find: drop the temporary indent Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51333 diff changeset	933 # in the revlog
ac8b798e602b delta-find: drop the temporary indent Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51333 diff changeset	934 if self.target_rev is not None and rev >= self.target_rev:
ac8b798e602b delta-find: drop the temporary indent Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51333 diff changeset	935 return False
51335 2407af4f2eea delta-find: split the generic part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51334 diff changeset	936 # no delta for rawtext-changing revs (see "candelta" for why)
2407af4f2eea delta-find: split the generic part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51334 diff changeset	937 if self.revlog.flags(rev) & REVIDX_RAWTEXT_CHANGING_FLAGS:
2407af4f2eea delta-find: split the generic part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51334 diff changeset	938 return False
2407af4f2eea delta-find: split the generic part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51334 diff changeset	939 return True
2407af4f2eea delta-find: split the generic part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51334 diff changeset	940
51336 9a1239c362ae delta-find: split the "sparse" part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51335 diff changeset	941 def _pre_filter_rev_sparse(self, rev):
9a1239c362ae delta-find: split the "sparse" part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51335 diff changeset	942 """pre filtering that is needed in sparse revlog cases
51333 898c212e1b2f delta-find: move pre-filtering of individual revision in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51332 diff changeset	943
51336 9a1239c362ae delta-find: split the "sparse" part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51335 diff changeset	944 return True if it seems okay to test a rev, False otherwise.
51332 02cc19f4f091 delta-find: move pre-filtering of candidates in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51331 diff changeset	945
51336 9a1239c362ae delta-find: split the "sparse" part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51335 diff changeset	946 used by _pre_filter_rev.
9a1239c362ae delta-find: split the "sparse" part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51335 diff changeset	947 """
9a1239c362ae delta-find: split the "sparse" part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51335 diff changeset	948 assert self.revlog.delta_config.sparse_revlog
51334 ac8b798e602b delta-find: drop the temporary indent Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51333 diff changeset	949 # if the revision we test again is too small, the resulting delta
ac8b798e602b delta-find: drop the temporary indent Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51333 diff changeset	950 # will be large anyway as that amount of data to be added is big
51336 9a1239c362ae delta-find: split the "sparse" part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51335 diff changeset	951 if self.revlog.rawsize(rev) < (self.textlen // LIMIT_BASE2TEXT):
51334 ac8b798e602b delta-find: drop the temporary indent Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51333 diff changeset	952 return False
51332 02cc19f4f091 delta-find: move pre-filtering of candidates in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51331 diff changeset	953
51336 9a1239c362ae delta-find: split the "sparse" part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51335 diff changeset	954 if self.revlog.delta_config.upper_bound_comp is not None:
51334 ac8b798e602b delta-find: drop the temporary indent Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51333 diff changeset	955 maxcomp = self.revlog.delta_config.upper_bound_comp
ac8b798e602b delta-find: drop the temporary indent Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51333 diff changeset	956 basenotsnap = (self.p1, self.p2, nullrev)
ac8b798e602b delta-find: drop the temporary indent Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51333 diff changeset	957 if rev not in basenotsnap and self.revlog.issnapshot(rev):
ac8b798e602b delta-find: drop the temporary indent Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51333 diff changeset	958 snapshotdepth = self.revlog.snapshotdepth(rev)
ac8b798e602b delta-find: drop the temporary indent Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51333 diff changeset	959 # If text is significantly larger than the base, we can
ac8b798e602b delta-find: drop the temporary indent Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51333 diff changeset	960 # expect the resulting delta to be proportional to the size
ac8b798e602b delta-find: drop the temporary indent Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51333 diff changeset	961 # difference
ac8b798e602b delta-find: drop the temporary indent Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51333 diff changeset	962 revsize = self.revlog.rawsize(rev)
ac8b798e602b delta-find: drop the temporary indent Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51333 diff changeset	963 rawsizedistance = max(self.textlen - revsize, 0)
ac8b798e602b delta-find: drop the temporary indent Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51333 diff changeset	964 # use an estimate of the compression upper bound.
ac8b798e602b delta-find: drop the temporary indent Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51333 diff changeset	965 lowestrealisticdeltalen = rawsizedistance // maxcomp
51332 02cc19f4f091 delta-find: move pre-filtering of candidates in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51331 diff changeset	966
51334 ac8b798e602b delta-find: drop the temporary indent Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51333 diff changeset	967 # check the absolute constraint on the delta size
ac8b798e602b delta-find: drop the temporary indent Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51333 diff changeset	968 snapshotlimit = self.textlen >> snapshotdepth
ac8b798e602b delta-find: drop the temporary indent Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51333 diff changeset	969 if snapshotlimit < lowestrealisticdeltalen:
ac8b798e602b delta-find: drop the temporary indent Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51333 diff changeset	970 # delta lower bound is larger than accepted upper
ac8b798e602b delta-find: drop the temporary indent Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51333 diff changeset	971 # bound
ac8b798e602b delta-find: drop the temporary indent Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51333 diff changeset	972 return False
51332 02cc19f4f091 delta-find: move pre-filtering of candidates in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51331 diff changeset	973
51334 ac8b798e602b delta-find: drop the temporary indent Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51333 diff changeset	974 # check the relative constraint on the delta size
ac8b798e602b delta-find: drop the temporary indent Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51333 diff changeset	975 revlength = self.revlog.length(rev)
ac8b798e602b delta-find: drop the temporary indent Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51333 diff changeset	976 if revlength < lowestrealisticdeltalen:
ac8b798e602b delta-find: drop the temporary indent Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51333 diff changeset	977 # delta probable lower bound is larger than target
ac8b798e602b delta-find: drop the temporary indent Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51333 diff changeset	978 # base
ac8b798e602b delta-find: drop the temporary indent Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51333 diff changeset	979 return False
51333 898c212e1b2f delta-find: move pre-filtering of individual revision in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51332 diff changeset	980 return True
51332 02cc19f4f091 delta-find: move pre-filtering of candidates in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51331 diff changeset	981
51337 1ea56b10dd4a delta-find: split the delta-chain part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51336 diff changeset	982 def _pre_filter_rev_delta_chain(self, rev):
1ea56b10dd4a delta-find: split the delta-chain part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51336 diff changeset	983 """pre filtering that is needed in sparse revlog cases
51336 9a1239c362ae delta-find: split the "sparse" part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51335 diff changeset	984
51337 1ea56b10dd4a delta-find: split the delta-chain part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51336 diff changeset	985 return True if it seems okay to test a rev, False otherwise.
1ea56b10dd4a delta-find: split the delta-chain part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51336 diff changeset	986
1ea56b10dd4a delta-find: split the delta-chain part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51336 diff changeset	987 used by _pre_filter_rev.
1ea56b10dd4a delta-find: split the delta-chain part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51336 diff changeset	988 """
51336 9a1239c362ae delta-find: split the "sparse" part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51335 diff changeset	989 deltas_limit = self.revinfo.textlen * LIMIT_DELTA2TEXT
9a1239c362ae delta-find: split the "sparse" part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51335 diff changeset	990 # filter out delta base that will never produce good delta
9a1239c362ae delta-find: split the "sparse" part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51335 diff changeset	991 #
9a1239c362ae delta-find: split the "sparse" part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51335 diff changeset	992 # if the delta of that base is already bigger than the limit
9a1239c362ae delta-find: split the "sparse" part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51335 diff changeset	993 # for the delta chain size, doing a delta is hopeless.
9a1239c362ae delta-find: split the "sparse" part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51335 diff changeset	994 if deltas_limit < self.revlog.length(rev):
9a1239c362ae delta-find: split the "sparse" part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51335 diff changeset	995 return False
9a1239c362ae delta-find: split the "sparse" part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51335 diff changeset	996
9a1239c362ae delta-find: split the "sparse" part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51335 diff changeset	997 # If we reach here, we are about to build and test a delta.
9a1239c362ae delta-find: split the "sparse" part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51335 diff changeset	998 # The delta building process will compute the chaininfo in all
9a1239c362ae delta-find: split the "sparse" part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51335 diff changeset	999 # case, since that computation is cached, it is fine to access
9a1239c362ae delta-find: split the "sparse" part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51335 diff changeset	1000 # it here too.
9a1239c362ae delta-find: split the "sparse" part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51335 diff changeset	1001 chainlen, chainsize = self.revlog._chaininfo(rev)
9a1239c362ae delta-find: split the "sparse" part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51335 diff changeset	1002 # if chain will be too long, skip base
9a1239c362ae delta-find: split the "sparse" part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51335 diff changeset	1003 if (
9a1239c362ae delta-find: split the "sparse" part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51335 diff changeset	1004 self.revlog.delta_config.max_chain_len
9a1239c362ae delta-find: split the "sparse" part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51335 diff changeset	1005 and chainlen >= self.revlog.delta_config.max_chain_len
9a1239c362ae delta-find: split the "sparse" part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51335 diff changeset	1006 ):
9a1239c362ae delta-find: split the "sparse" part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51335 diff changeset	1007 return False
9a1239c362ae delta-find: split the "sparse" part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51335 diff changeset	1008 # if chain already have too much data, skip base
9a1239c362ae delta-find: split the "sparse" part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51335 diff changeset	1009 if deltas_limit < chainsize:
9a1239c362ae delta-find: split the "sparse" part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51335 diff changeset	1010 return False
51337 1ea56b10dd4a delta-find: split the delta-chain part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51336 diff changeset	1011 return True
51336 9a1239c362ae delta-find: split the "sparse" part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51335 diff changeset	1012
51337 1ea56b10dd4a delta-find: split the delta-chain part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51336 diff changeset	1013 def _pre_filter_rev(self, rev):
1ea56b10dd4a delta-find: split the delta-chain part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51336 diff changeset	1014 """return True if it seems okay to test a rev, False otherwise"""
1ea56b10dd4a delta-find: split the delta-chain part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51336 diff changeset	1015 if not self._pre_filter_rev_universal(rev):
1ea56b10dd4a delta-find: split the delta-chain part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51336 diff changeset	1016 return False
1ea56b10dd4a delta-find: split the delta-chain part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51336 diff changeset	1017 if not self._pre_filter_rev_delta_chain(rev):
1ea56b10dd4a delta-find: split the delta-chain part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51336 diff changeset	1018 return False
51336 9a1239c362ae delta-find: split the "sparse" part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51335 diff changeset	1019 if self.revlog.delta_config.sparse_revlog:
9a1239c362ae delta-find: split the "sparse" part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51335 diff changeset	1020 if not self._pre_filter_rev_sparse(rev):
9a1239c362ae delta-find: split the "sparse" part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51335 diff changeset	1021 return False
9a1239c362ae delta-find: split the "sparse" part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51335 diff changeset	1022
9a1239c362ae delta-find: split the "sparse" part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51335 diff changeset	1023 return True
9a1239c362ae delta-find: split the "sparse" part of `_pre_filter_rev` in a method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51335 diff changeset	1024
51350 5cc04a6da19d delta-find: move the emotion of parents in a dedicated method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51349 diff changeset	1025 def _iter_parents(self):
5cc04a6da19d delta-find: move the emotion of parents in a dedicated method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51349 diff changeset	1026 # exclude already lazy tested base if any
5cc04a6da19d delta-find: move the emotion of parents in a dedicated method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51349 diff changeset	1027 parents = [p for p in (self.p1, self.p2) if p != nullrev]
5cc04a6da19d delta-find: move the emotion of parents in a dedicated method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51349 diff changeset	1028
5cc04a6da19d delta-find: move the emotion of parents in a dedicated method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51349 diff changeset	1029 self.current_stage = _STAGE_PARENTS
5cc04a6da19d delta-find: move the emotion of parents in a dedicated method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51349 diff changeset	1030 if (
5cc04a6da19d delta-find: move the emotion of parents in a dedicated method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51349 diff changeset	1031 not self.revlog.delta_config.delta_both_parents
5cc04a6da19d delta-find: move the emotion of parents in a dedicated method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51349 diff changeset	1032 and len(parents) == 2
5cc04a6da19d delta-find: move the emotion of parents in a dedicated method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51349 diff changeset	1033 ):
5cc04a6da19d delta-find: move the emotion of parents in a dedicated method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51349 diff changeset	1034 parents.sort()
5cc04a6da19d delta-find: move the emotion of parents in a dedicated method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51349 diff changeset	1035 # To minimize the chance of having to build a fulltext,
5cc04a6da19d delta-find: move the emotion of parents in a dedicated method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51349 diff changeset	1036 # pick first whichever parent is closest to us (max rev)
5cc04a6da19d delta-find: move the emotion of parents in a dedicated method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51349 diff changeset	1037 yield (parents[1],)
5cc04a6da19d delta-find: move the emotion of parents in a dedicated method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51349 diff changeset	1038 # then the other one (min rev) if the first did not fit
5cc04a6da19d delta-find: move the emotion of parents in a dedicated method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51349 diff changeset	1039 yield (parents[0],)
5cc04a6da19d delta-find: move the emotion of parents in a dedicated method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51349 diff changeset	1040 elif len(parents) > 0:
5cc04a6da19d delta-find: move the emotion of parents in a dedicated method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51349 diff changeset	1041 # Test all parents (1 or 2), and keep the best candidate
5cc04a6da19d delta-find: move the emotion of parents in a dedicated method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51349 diff changeset	1042 yield parents
5cc04a6da19d delta-find: move the emotion of parents in a dedicated method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51349 diff changeset	1043
51351 94fe4474b053 delta-find: move the emotion of prev in a dedicated method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51350 diff changeset	1044 def _iter_prev(self):
94fe4474b053 delta-find: move the emotion of prev in a dedicated method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51350 diff changeset	1045 # other approach failed try against prev to hopefully save us a
94fe4474b053 delta-find: move the emotion of prev in a dedicated method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51350 diff changeset	1046 # fulltext.
94fe4474b053 delta-find: move the emotion of prev in a dedicated method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51350 diff changeset	1047 self.current_stage = _STAGE_PREV
94fe4474b053 delta-find: move the emotion of prev in a dedicated method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51350 diff changeset	1048 yield (self.target_rev - 1,)
94fe4474b053 delta-find: move the emotion of prev in a dedicated method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51350 diff changeset	1049
51354 d7e2acdd50ba delta-find: split the _DeltaSearch class in two Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	1050 def _iter_groups(self):
d7e2acdd50ba delta-find: split the _DeltaSearch class in two Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	1051 good = None
d7e2acdd50ba delta-find: split the _DeltaSearch class in two Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	1052 for group in self._iter_parents():
d7e2acdd50ba delta-find: split the _DeltaSearch class in two Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	1053 good = yield group
d7e2acdd50ba delta-find: split the _DeltaSearch class in two Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	1054 if good is not None:
d7e2acdd50ba delta-find: split the _DeltaSearch class in two Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	1055 break
d7e2acdd50ba delta-find: split the _DeltaSearch class in two Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	1056 else:
d7e2acdd50ba delta-find: split the _DeltaSearch class in two Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	1057 assert good is None
d7e2acdd50ba delta-find: split the _DeltaSearch class in two Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	1058 yield from self._iter_prev()
d7e2acdd50ba delta-find: split the _DeltaSearch class in two Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	1059 yield None
d7e2acdd50ba delta-find: split the _DeltaSearch class in two Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	1060
d7e2acdd50ba delta-find: split the _DeltaSearch class in two Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	1061
d7e2acdd50ba delta-find: split the _DeltaSearch class in two Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	1062 class _SparseDeltaSearch(_GeneralDeltaSearch):
d7e2acdd50ba delta-find: split the _DeltaSearch class in two Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	1063 """Delta search variants for sparse-revlog"""
d7e2acdd50ba delta-find: split the _DeltaSearch class in two Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	1064
51355 383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51354 diff changeset	1065 def is_good_delta_info(self, deltainfo):
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51354 diff changeset	1066 """Returns True if the given delta is good.
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51354 diff changeset	1067
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51354 diff changeset	1068 Good means that it is within the disk span, disk size, and chain length
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51354 diff changeset	1069 bounds that we know to be performant.
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51354 diff changeset	1070 """
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51354 diff changeset	1071 if not self._is_good_delta_info_universal(deltainfo):
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51354 diff changeset	1072 return False
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51354 diff changeset	1073 if not self._is_good_delta_info_chain_quality(deltainfo):
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51354 diff changeset	1074 return False
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51354 diff changeset	1075 if not self._is_good_delta_info_snapshot_constraints(deltainfo):
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51354 diff changeset	1076 return False
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51354 diff changeset	1077 return True
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51354 diff changeset	1078
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51354 diff changeset	1079 def _is_good_delta_info_snapshot_constraints(self, deltainfo):
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51354 diff changeset	1080 """Returns True if the chain associated with snapshots
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51354 diff changeset	1081
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51354 diff changeset	1082 This performs checks for format that use sparse-revlog and intermediate
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51354 diff changeset	1083 snapshots.
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51354 diff changeset	1084
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51354 diff changeset	1085 This is used by is_good_delta_info.
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51354 diff changeset	1086 """
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51354 diff changeset	1087 # if not a snapshot, this method has no filtering to do
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51354 diff changeset	1088 if deltainfo.snapshotdepth is None:
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51354 diff changeset	1089 return True
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51354 diff changeset	1090 # bad delta from intermediate snapshot size limit
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51354 diff changeset	1091 #
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51354 diff changeset	1092 # If an intermediate snapshot size is higher than the limit. The
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51354 diff changeset	1093 # limit exist to prevent endless chain of intermediate delta to be
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51354 diff changeset	1094 # created.
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51354 diff changeset	1095 if (
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51354 diff changeset	1096 self.revinfo.textlen >> deltainfo.snapshotdepth
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51354 diff changeset	1097 ) < deltainfo.deltalen:
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51354 diff changeset	1098 return False
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51354 diff changeset	1099
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51354 diff changeset	1100 # bad delta if new intermediate snapshot is larger than the previous
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51354 diff changeset	1101 # snapshot
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51354 diff changeset	1102 if self.revlog.length(deltainfo.base) < deltainfo.deltalen:
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51354 diff changeset	1103 return False
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51354 diff changeset	1104
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51354 diff changeset	1105 return True
383e99f6bc99 delta-find: move sparse-revlog delta checks in the associated class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51354 diff changeset	1106
51352 fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1107 def _iter_snapshots_base(self):
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1108 assert self.revlog.delta_config.sparse_revlog
51353 701caeabbee7 delta-find: finish reworking the snapshot logic and drop more layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51352 diff changeset	1109 assert self.current_stage == _STAGE_SNAPSHOT
51352 fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1110 prev = self.target_rev - 1
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1111 deltachain = lambda rev: self.revlog._deltachain(rev)[0]
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1112
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1113 parents = [p for p in (self.p1, self.p2) if p != nullrev]
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1114 if not parents:
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1115 return
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1116 # See if we can use an existing snapshot in the parent chains to
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1117 # use as a base for a new intermediate-snapshot
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1118 #
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1119 # search for snapshot in parents delta chain map: snapshot-level:
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1120 # snapshot-rev
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1121 parents_snaps = collections.defaultdict(set)
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1122 candidate_chains = [deltachain(p) for p in parents]
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1123 for chain in candidate_chains:
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1124 for idx, s in enumerate(chain):
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1125 if not self.revlog.issnapshot(s):
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1126 break
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1127 parents_snaps[idx].add(s)
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1128 snapfloor = min(parents_snaps[0]) + 1
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1129 self.snapshot_cache.update(self.revlog, snapfloor)
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1130 # search for the highest "unrelated" revision
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1131 #
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1132 # Adding snapshots used by "unrelated" revision increase the odd we
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1133 # reuse an independant, yet better snapshot chain.
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1134 #
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1135 # XXX instead of building a set of revisions, we could lazily
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1136 # enumerate over the chains. That would be more efficient, however
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1137 # we stick to simple code for now.
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1138 all_revs = set()
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1139 for chain in candidate_chains:
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1140 all_revs.update(chain)
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1141 other = None
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1142 for r in self.revlog.revs(prev, snapfloor):
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1143 if r not in all_revs:
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1144 other = r
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1145 break
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1146 if other is not None:
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1147 # To avoid unfair competition, we won't use unrelated
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1148 # intermediate snapshot that are deeper than the ones from the
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1149 # parent delta chain.
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1150 max_depth = max(parents_snaps.keys())
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1151 chain = deltachain(other)
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1152 for depth, s in enumerate(chain):
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1153 if s < snapfloor:
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1154 continue
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1155 if max_depth < depth:
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1156 break
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1157 if not self.revlog.issnapshot(s):
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1158 break
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1159 parents_snaps[depth].add(s)
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1160 # Test them as possible intermediate snapshot base We test them
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1161 # from highest to lowest level. High level one are more likely to
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1162 # result in small delta
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1163 floor = None
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1164 for idx, snaps in sorted(parents_snaps.items(), reverse=True):
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1165 siblings = set()
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1166 for s in snaps:
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1167 siblings.update(self.snapshot_cache.snapshots[s])
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1168 # Before considering making a new intermediate snapshot, we
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1169 # check if an existing snapshot, children of base we consider,
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1170 # would be suitable.
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1171 #
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1172 # It give a change to reuse a delta chain "unrelated" to the
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1173 # current revision instead of starting our own. Without such
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1174 # re-use, topological branches would keep reopening new chains.
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1175 # Creating more and more snapshot as the repository grow.
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1176
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1177 if floor is not None:
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1178 # We only do this for siblings created after the one in our
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1179 # parent's delta chain. Those created before has less
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1180 # chances to be valid base since our ancestors had to
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1181 # create a new snapshot.
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1182 siblings = [r for r in siblings if floor < r]
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1183 yield tuple(sorted(siblings))
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1184 # then test the base from our parent's delta chain.
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1185 yield tuple(sorted(snaps))
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1186 floor = min(snaps)
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1187 # No suitable base found in the parent chain, search if any full
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1188 # snapshots emitted since parent's base would be a suitable base
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1189 # for an intermediate snapshot.
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1190 #
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1191 # It give a chance to reuse a delta chain unrelated to the current
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1192 # revisions instead of starting our own. Without such re-use,
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1193 # topological branches would keep reopening new full chains.
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1194 # Creating more and more snapshot as the repository grow.
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1195 full = [
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1196 r for r in self.snapshot_cache.snapshots[nullrev] if snapfloor <= r
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1197 ]
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1198 yield tuple(sorted(full))
fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1199
51353 701caeabbee7 delta-find: finish reworking the snapshot logic and drop more layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51352 diff changeset	1200 def _iter_snapshots(self):
701caeabbee7 delta-find: finish reworking the snapshot logic and drop more layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51352 diff changeset	1201 assert self.revlog.delta_config.sparse_revlog
701caeabbee7 delta-find: finish reworking the snapshot logic and drop more layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51352 diff changeset	1202 self.current_stage = _STAGE_SNAPSHOT
51318 c83074405276 delta-find: move `_refinedgroups` on the `_DeltaSearch` object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51317 diff changeset	1203 good = None
51353 701caeabbee7 delta-find: finish reworking the snapshot logic and drop more layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51352 diff changeset	1204 groups = self._iter_snapshots_base()
51318 c83074405276 delta-find: move `_refinedgroups` on the `_DeltaSearch` object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51317 diff changeset	1205 for candidates in groups:
c83074405276 delta-find: move `_refinedgroups` on the `_DeltaSearch` object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51317 diff changeset	1206 good = yield candidates
c83074405276 delta-find: move `_refinedgroups` on the `_DeltaSearch` object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51317 diff changeset	1207 if good is not None:
c83074405276 delta-find: move `_refinedgroups` on the `_DeltaSearch` object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51317 diff changeset	1208 break
c83074405276 delta-find: move `_refinedgroups` on the `_DeltaSearch` object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51317 diff changeset	1209 # if we have a refinable value, try to refine it
c83074405276 delta-find: move `_refinedgroups` on the `_DeltaSearch` object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51317 diff changeset	1210 if (
c83074405276 delta-find: move `_refinedgroups` on the `_DeltaSearch` object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51317 diff changeset	1211 good is not None
c83074405276 delta-find: move `_refinedgroups` on the `_DeltaSearch` object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51317 diff changeset	1212 and good not in (self.p1, self.p2)
c83074405276 delta-find: move `_refinedgroups` on the `_DeltaSearch` object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51317 diff changeset	1213 and self.revlog.issnapshot(good)
c83074405276 delta-find: move `_refinedgroups` on the `_DeltaSearch` object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51317 diff changeset	1214 ):
51347 670e68729aa7 delta-find: stop using heuristic to determine if we are creating a snapshot Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51346 diff changeset	1215 assert self.current_stage == _STAGE_SNAPSHOT
51318 c83074405276 delta-find: move `_refinedgroups` on the `_DeltaSearch` object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51317 diff changeset	1216 # refine snapshot down
c83074405276 delta-find: move `_refinedgroups` on the `_DeltaSearch` object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51317 diff changeset	1217 previous = None
c83074405276 delta-find: move `_refinedgroups` on the `_DeltaSearch` object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51317 diff changeset	1218 while previous != good:
c83074405276 delta-find: move `_refinedgroups` on the `_DeltaSearch` object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51317 diff changeset	1219 previous = good
c83074405276 delta-find: move `_refinedgroups` on the `_DeltaSearch` object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51317 diff changeset	1220 base = self.revlog.deltaparent(good)
c83074405276 delta-find: move `_refinedgroups` on the `_DeltaSearch` object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51317 diff changeset	1221 if base == nullrev:
c83074405276 delta-find: move `_refinedgroups` on the `_DeltaSearch` object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51317 diff changeset	1222 break
c83074405276 delta-find: move `_refinedgroups` on the `_DeltaSearch` object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51317 diff changeset	1223 good = yield (base,)
c83074405276 delta-find: move `_refinedgroups` on the `_DeltaSearch` object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51317 diff changeset	1224 # refine snapshot up
c83074405276 delta-find: move `_refinedgroups` on the `_DeltaSearch` object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51317 diff changeset	1225 if not self.snapshot_cache.snapshots:
c83074405276 delta-find: move `_refinedgroups` on the `_DeltaSearch` object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51317 diff changeset	1226 self.snapshot_cache.update(self.revlog, good + 1)
c83074405276 delta-find: move `_refinedgroups` on the `_DeltaSearch` object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51317 diff changeset	1227 previous = None
c83074405276 delta-find: move `_refinedgroups` on the `_DeltaSearch` object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51317 diff changeset	1228 while good != previous:
c83074405276 delta-find: move `_refinedgroups` on the `_DeltaSearch` object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51317 diff changeset	1229 previous = good
c83074405276 delta-find: move `_refinedgroups` on the `_DeltaSearch` object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51317 diff changeset	1230 children = tuple(
c83074405276 delta-find: move `_refinedgroups` on the `_DeltaSearch` object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51317 diff changeset	1231 sorted(c for c in self.snapshot_cache.snapshots[good])
c83074405276 delta-find: move `_refinedgroups` on the `_DeltaSearch` object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51317 diff changeset	1232 )
c83074405276 delta-find: move `_refinedgroups` on the `_DeltaSearch` object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51317 diff changeset	1233 good = yield children
c83074405276 delta-find: move `_refinedgroups` on the `_DeltaSearch` object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51317 diff changeset	1234 yield None
39496 2f9f7889549b snapshot: introduce an intermediate `_refinedgroups` generator Boris Feld <boris.feld@octobus.net> parents: 39495 diff changeset	1235
51353 701caeabbee7 delta-find: finish reworking the snapshot logic and drop more layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51352 diff changeset	1236 def _iter_groups(self):
701caeabbee7 delta-find: finish reworking the snapshot logic and drop more layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51352 diff changeset	1237 good = None
701caeabbee7 delta-find: finish reworking the snapshot logic and drop more layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51352 diff changeset	1238 for group in self._iter_parents():
701caeabbee7 delta-find: finish reworking the snapshot logic and drop more layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51352 diff changeset	1239 good = yield group
701caeabbee7 delta-find: finish reworking the snapshot logic and drop more layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51352 diff changeset	1240 if good is not None:
701caeabbee7 delta-find: finish reworking the snapshot logic and drop more layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51352 diff changeset	1241 break
51352 fac6038b11f5 delta-find: move the base of the delta search in its own function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51351 diff changeset	1242 else:
51353 701caeabbee7 delta-find: finish reworking the snapshot logic and drop more layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51352 diff changeset	1243 assert good is None
51354 d7e2acdd50ba delta-find: split the _DeltaSearch class in two Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	1244 assert self.revlog.delta_config.sparse_revlog
d7e2acdd50ba delta-find: split the _DeltaSearch class in two Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	1245 # If sparse revlog is enabled, we can try to refine the
d7e2acdd50ba delta-find: split the _DeltaSearch class in two Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	1246 # available deltas
d7e2acdd50ba delta-find: split the _DeltaSearch class in two Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	1247 iter_snap = self._iter_snapshots()
d7e2acdd50ba delta-find: split the _DeltaSearch class in two Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	1248 group = iter_snap.send(None)
d7e2acdd50ba delta-find: split the _DeltaSearch class in two Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	1249 while group is not None:
d7e2acdd50ba delta-find: split the _DeltaSearch class in two Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	1250 good = yield group
d7e2acdd50ba delta-find: split the _DeltaSearch class in two Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	1251 group = iter_snap.send(good)
51353 701caeabbee7 delta-find: finish reworking the snapshot logic and drop more layer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51352 diff changeset	1252 yield None
39334 507f5b1dd7c8 revlogdeltas: extract _getcandidaterevs in a function Boris Feld <boris.feld@octobus.net> parents: 39333 diff changeset	1253
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1254
49678 efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1255 class SnapshotCache:
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1256 __slots__ = ('snapshots', '_start_rev', '_end_rev')
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1257
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1258 def __init__(self):
49679 b670eb3dd6c9 delta-find: use sets instead of list in the snapshot cache Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49678 diff changeset	1259 self.snapshots = collections.defaultdict(set)
49678 efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1260 self._start_rev = None
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1261 self._end_rev = None
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1262
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1263 def update(self, revlog, start_rev=0):
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1264 """find snapshots from start_rev to tip"""
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1265 nb_revs = len(revlog)
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1266 end_rev = nb_revs - 1
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1267 if start_rev > end_rev:
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1268 return # range is empty
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1269
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1270 if self._start_rev is None:
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1271 assert self._end_rev is None
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1272 self._update(revlog, start_rev, end_rev)
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1273 elif not (self._start_rev <= start_rev and end_rev <= self._end_rev):
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1274 if start_rev < self._start_rev:
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1275 self._update(revlog, start_rev, self._start_rev - 1)
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1276 if self._end_rev < end_rev:
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1277 self._update(revlog, self._end_rev + 1, end_rev)
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1278
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1279 if self._start_rev is None:
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1280 assert self._end_rev is None
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1281 self._end_rev = end_rev
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1282 self._start_rev = start_rev
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1283 else:
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1284 self._start_rev = min(self._start_rev, start_rev)
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1285 self._end_rev = max(self._end_rev, end_rev)
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1286 assert self._start_rev <= self._end_rev, (
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1287 self._start_rev,
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1288 self._end_rev,
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1289 )
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1290
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1291 def _update(self, revlog, start_rev, end_rev):
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1292 """internal method that actually do update content"""
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1293 assert self._start_rev is None or (
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1294 start_rev < self._start_rev or start_rev > self._end_rev
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1295 ), (self._start_rev, self._end_rev, start_rev, end_rev)
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1296 assert self._start_rev is None or (
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1297 end_rev < self._start_rev or end_rev > self._end_rev
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1298 ), (self._start_rev, self._end_rev, start_rev, end_rev)
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1299 cache = self.snapshots
50925 d718eddf01d9 safehasattr: drop usage in favor of hasattr Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50695 diff changeset	1300 if hasattr(revlog.index, 'findsnapshots'):
49678 efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1301 revlog.index.findsnapshots(cache, start_rev, end_rev)
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1302 else:
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1303 deltaparent = revlog.deltaparent
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1304 issnapshot = revlog.issnapshot
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1305 for rev in revlog.revs(start_rev, end_rev):
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1306 if issnapshot(rev):
49679 b670eb3dd6c9 delta-find: use sets instead of list in the snapshot cache Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49678 diff changeset	1307 cache[deltaparent(rev)].add(rev)
49678 efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1308
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1309
48946 642e31cb55f0 py3: use class X: instead of class X(object): Gregory Szorc <gregory.szorc@gmail.com> parents: 48875 diff changeset	1310 class deltacomputer:
51316 da7ecb4deaec delta-find: add a small docstring to deltacomputer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51088 diff changeset	1311 """object capable of computing delta and finding delta for multiple revision
da7ecb4deaec delta-find: add a small docstring to deltacomputer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51088 diff changeset	1312
da7ecb4deaec delta-find: add a small docstring to deltacomputer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51088 diff changeset	1313 This object is meant to compute and find multiple delta applied to the same
da7ecb4deaec delta-find: add a small docstring to deltacomputer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51088 diff changeset	1314 revlog.
da7ecb4deaec delta-find: add a small docstring to deltacomputer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51088 diff changeset	1315 """
da7ecb4deaec delta-find: add a small docstring to deltacomputer Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51088 diff changeset	1316
49610 35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	1317 def __init__(
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	1318 self,
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	1319 revlog,
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	1320 write_debug=None,
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	1321 debug_search=False,
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	1322 debug_info=None,
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	1323 ):
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1324 self.revlog = revlog
49226 e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	1325 self._write_debug = write_debug
50646 c84cc0ac77e4 delta-fine: use the `_debug_search` attribute directly Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50645 diff changeset	1326 if write_debug is None:
c84cc0ac77e4 delta-fine: use the `_debug_search` attribute directly Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50645 diff changeset	1327 self._debug_search = False
c84cc0ac77e4 delta-fine: use the `_debug_search` attribute directly Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50645 diff changeset	1328 else:
c84cc0ac77e4 delta-fine: use the `_debug_search` attribute directly Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50645 diff changeset	1329 self._debug_search = debug_search
49610 35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	1330 self._debug_info = debug_info
49681 c261a628e525 delta-find: use a single snapshot cache when applying a group to an object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49680 diff changeset	1331 self._snapshot_cache = SnapshotCache()
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1332
50645 5d210ff4b657 delta-find: move the `gather_debug` logic in a property Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50357 diff changeset	1333 @property
5d210ff4b657 delta-find: move the `gather_debug` logic in a property Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50357 diff changeset	1334 def _gather_debug(self):
5d210ff4b657 delta-find: move the `gather_debug` logic in a property Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50357 diff changeset	1335 return self._write_debug is not None or self._debug_info is not None
5d210ff4b657 delta-find: move the `gather_debug` logic in a property Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50357 diff changeset	1336
51017 509f0f7fc89e delta-computer: stop explicitly taking file handle Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50926 diff changeset	1337 def buildtext(self, revinfo):
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1338 """Builds a fulltext version of a revision
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1339
47399 34cc102c73f5 revlog: move `revisioninfo` in `revlogutils` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47253 diff changeset	1340 revinfo: revisioninfo instance that contains all needed info
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1341 """
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1342 btext = revinfo.btext
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1343 if btext[0] is not None:
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1344 return btext[0]
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1345
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1346 revlog = self.revlog
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1347 cachedelta = revinfo.cachedelta
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1348 baserev = cachedelta[0]
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1349 delta = cachedelta[1]
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1350
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1351 fulltext = btext[0] = _textfromdelta(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1352 revlog,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1353 baserev,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1354 delta,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1355 revinfo.p1,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1356 revinfo.p2,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1357 revinfo.flags,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1358 revinfo.node,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1359 )
39331 fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39330 diff changeset	1360 return fulltext
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1361
51017 509f0f7fc89e delta-computer: stop explicitly taking file handle Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50926 diff changeset	1362 def _builddeltadiff(self, base, revinfo):
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1363 revlog = self.revlog
51017 509f0f7fc89e delta-computer: stop explicitly taking file handle Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50926 diff changeset	1364 t = self.buildtext(revinfo)
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1365 if revlog.iscensored(base):
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1366 # deltas based on a censored revision must replace the
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1367 # full content in one patch, so delta works everywhere
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1368 header = mdiff.replacediffheader(revlog.rawsize(base), len(t))
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1369 delta = header + t
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1370 else:
51017 509f0f7fc89e delta-computer: stop explicitly taking file handle Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50926 diff changeset	1371 ptext = revlog.rawdata(base)
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1372 delta = mdiff.textdiff(ptext, t)
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1373
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1374 return delta
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1375
51347 670e68729aa7 delta-find: stop using heuristic to determine if we are creating a snapshot Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51346 diff changeset	1376 def _builddeltainfo(
670e68729aa7 delta-find: stop using heuristic to determine if we are creating a snapshot Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51346 diff changeset	1377 self, revinfo, base, target_rev=None, as_snapshot=False
670e68729aa7 delta-find: stop using heuristic to determine if we are creating a snapshot Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51346 diff changeset	1378 ):
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1379 # can we use the cached delta?
42465 6e9ba867a946 delta: move some delta chain related computation earlier in deltainfo Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42464 diff changeset	1380 revlog = self.revlog
6e9ba867a946 delta: move some delta chain related computation earlier in deltainfo Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42464 diff changeset	1381 chainbase = revlog.chainbase(base)
51043 f636103c4d67 revlog: remove legacy usage of `_generaldelta` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51042 diff changeset	1382 if revlog.delta_config.general_delta:
42465 6e9ba867a946 delta: move some delta chain related computation earlier in deltainfo Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42464 diff changeset	1383 deltabase = base
6e9ba867a946 delta: move some delta chain related computation earlier in deltainfo Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42464 diff changeset	1384 else:
50356 2a6949ab9d23 delta-find: add a simple safeguard to prevent bad non-general-delta Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50355 diff changeset	1385 if target_rev is not None and base != target_rev - 1:
2a6949ab9d23 delta-find: add a simple safeguard to prevent bad non-general-delta Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50355 diff changeset	1386 msg = (
2a6949ab9d23 delta-find: add a simple safeguard to prevent bad non-general-delta Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50355 diff changeset	1387 b'general delta cannot use delta for something else '
2a6949ab9d23 delta-find: add a simple safeguard to prevent bad non-general-delta Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50355 diff changeset	1388 b'than `prev`: %d<-%d'
2a6949ab9d23 delta-find: add a simple safeguard to prevent bad non-general-delta Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50355 diff changeset	1389 )
2a6949ab9d23 delta-find: add a simple safeguard to prevent bad non-general-delta Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50355 diff changeset	1390 msg %= (base, target_rev)
2a6949ab9d23 delta-find: add a simple safeguard to prevent bad non-general-delta Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50355 diff changeset	1391 raise error.ProgrammingError(msg)
42465 6e9ba867a946 delta: move some delta chain related computation earlier in deltainfo Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42464 diff changeset	1392 deltabase = chainbase
6e9ba867a946 delta: move some delta chain related computation earlier in deltainfo Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42464 diff changeset	1393 snapshotdepth = None
51057 f71f07a679b4 revlog: remove legacy usage of `_sparserevlog` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51055 diff changeset	1394 if revlog.delta_config.sparse_revlog and deltabase == nullrev:
42465 6e9ba867a946 delta: move some delta chain related computation earlier in deltainfo Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42464 diff changeset	1395 snapshotdepth = 0
51347 670e68729aa7 delta-find: stop using heuristic to determine if we are creating a snapshot Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51346 diff changeset	1396 elif revlog.delta_config.sparse_revlog and as_snapshot:
670e68729aa7 delta-find: stop using heuristic to determine if we are creating a snapshot Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51346 diff changeset	1397 assert revlog.issnapshot(deltabase)
42465 6e9ba867a946 delta: move some delta chain related computation earlier in deltainfo Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42464 diff changeset	1398 # A delta chain should always be one full snapshot,
6e9ba867a946 delta: move some delta chain related computation earlier in deltainfo Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42464 diff changeset	1399 # zero or more semi-snapshots, and zero or more deltas
6e9ba867a946 delta: move some delta chain related computation earlier in deltainfo Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42464 diff changeset	1400 p1, p2 = revlog.rev(revinfo.p1), revlog.rev(revinfo.p2)
6e9ba867a946 delta: move some delta chain related computation earlier in deltainfo Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42464 diff changeset	1401 if deltabase not in (p1, p2) and revlog.issnapshot(deltabase):
6e9ba867a946 delta: move some delta chain related computation earlier in deltainfo Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42464 diff changeset	1402 snapshotdepth = len(revlog._deltachain(deltabase)[0])
39595 a911932d5003 revlog: reuse cached delta for identical base revision (issue5975) Boris Feld <boris.feld@octobus.net> parents: 39594 diff changeset	1403 delta = None
a911932d5003 revlog: reuse cached delta for identical base revision (issue5975) Boris Feld <boris.feld@octobus.net> parents: 39594 diff changeset	1404 if revinfo.cachedelta:
49675 0fca63953810 find-delta: minor preparatory change Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49674 diff changeset	1405 cachebase = revinfo.cachedelta[0]
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1406 # check if the diff still apply
39595 a911932d5003 revlog: reuse cached delta for identical base revision (issue5975) Boris Feld <boris.feld@octobus.net> parents: 39594 diff changeset	1407 currentbase = cachebase
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1408 while (
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1409 currentbase != nullrev
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1410 and currentbase != base
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1411 and self.revlog.length(currentbase) == 0
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1412 ):
39595 a911932d5003 revlog: reuse cached delta for identical base revision (issue5975) Boris Feld <boris.feld@octobus.net> parents: 39594 diff changeset	1413 currentbase = self.revlog.deltaparent(currentbase)
51064 26dcdbe15024 revlog: remove legacy usage of `_lazydeltabase` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51060 diff changeset	1414 if self.revlog.delta_config.lazy_delta and currentbase == base:
39595 a911932d5003 revlog: reuse cached delta for identical base revision (issue5975) Boris Feld <boris.feld@octobus.net> parents: 39594 diff changeset	1415 delta = revinfo.cachedelta[1]
a911932d5003 revlog: reuse cached delta for identical base revision (issue5975) Boris Feld <boris.feld@octobus.net> parents: 39594 diff changeset	1416 if delta is None:
51017 509f0f7fc89e delta-computer: stop explicitly taking file handle Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50926 diff changeset	1417 delta = self._builddeltadiff(base, revinfo)
50646 c84cc0ac77e4 delta-fine: use the `_debug_search` attribute directly Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50645 diff changeset	1418 if self._debug_search:
49228 b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1419 msg = b"DBG-DELTAS-SEARCH: uncompressed-delta-size=%d\n"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1420 msg %= len(delta)
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1421 self._write_debug(msg)
42467 c1c1872d25d1 deltas: skip if projected compressed size does not match text size constraint Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42466 diff changeset	1422 # snapshotdept need to be neither None nor 0 level snapshot
51080 5645524c6b62 revlog: also migrates `revlog.upperboundcomp` to ConfigClass Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51064 diff changeset	1423 if revlog.delta_config.upper_bound_comp is not None and snapshotdepth:
5645524c6b62 revlog: also migrates `revlog.upperboundcomp` to ConfigClass Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51064 diff changeset	1424 lowestrealisticdeltalen = (
5645524c6b62 revlog: also migrates `revlog.upperboundcomp` to ConfigClass Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51064 diff changeset	1425 len(delta) // revlog.delta_config.upper_bound_comp
5645524c6b62 revlog: also migrates `revlog.upperboundcomp` to ConfigClass Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51064 diff changeset	1426 )
42467 c1c1872d25d1 deltas: skip if projected compressed size does not match text size constraint Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42466 diff changeset	1427 snapshotlimit = revinfo.textlen >> snapshotdepth
50646 c84cc0ac77e4 delta-fine: use the `_debug_search` attribute directly Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50645 diff changeset	1428 if self._debug_search:
49228 b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1429 msg = b"DBG-DELTAS-SEARCH: projected-lower-size=%d\n"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1430 msg %= lowestrealisticdeltalen
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1431 self._write_debug(msg)
42467 c1c1872d25d1 deltas: skip if projected compressed size does not match text size constraint Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42466 diff changeset	1432 if snapshotlimit < lowestrealisticdeltalen:
50646 c84cc0ac77e4 delta-fine: use the `_debug_search` attribute directly Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50645 diff changeset	1433 if self._debug_search:
49228 b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1434 msg = b"DBG-DELTAS-SEARCH: DISCARDED (snapshot limit)\n"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1435 self._write_debug(msg)
42467 c1c1872d25d1 deltas: skip if projected compressed size does not match text size constraint Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42466 diff changeset	1436 return None
42468 9b5fbe5ead89 deltas: skip if projected compressed size is bigger than previous snapshot Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42467 diff changeset	1437 if revlog.length(base) < lowestrealisticdeltalen:
50646 c84cc0ac77e4 delta-fine: use the `_debug_search` attribute directly Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50645 diff changeset	1438 if self._debug_search:
49228 b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1439 msg = b"DBG-DELTAS-SEARCH: DISCARDED (prev size)\n"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1440 self._write_debug(msg)
42468 9b5fbe5ead89 deltas: skip if projected compressed size is bigger than previous snapshot Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42467 diff changeset	1441 return None
51088 a82704902db8 revlog: move the compression/decompression logic on the inner object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51080 diff changeset	1442 header, data = revlog._inner.compress(delta)
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1443 deltalen = len(header) + len(data)
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1444 offset = revlog.end(len(revlog) - 1)
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1445 dist = deltalen + offset - revlog.start(chainbase)
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1446 chainlen, compresseddeltalen = revlog._chaininfo(base)
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1447 chainlen += 1
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1448 compresseddeltalen += deltalen
39154 e0da43e2f71f revlog: compute snapshot depth on delta info Boris Feld <boris.feld@octobus.net> parents: 39152 diff changeset	1449
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1450 return _deltainfo(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1451 dist,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1452 deltalen,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1453 (header, data),
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1454 deltabase,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1455 chainbase,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1456 chainlen,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1457 compresseddeltalen,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1458 snapshotdepth,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1459 )
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1460
51017 509f0f7fc89e delta-computer: stop explicitly taking file handle Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50926 diff changeset	1461 def _fullsnapshotinfo(self, revinfo, curr):
509f0f7fc89e delta-computer: stop explicitly taking file handle Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50926 diff changeset	1462 rawtext = self.buildtext(revinfo)
51088 a82704902db8 revlog: move the compression/decompression logic on the inner object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51080 diff changeset	1463 data = self.revlog._inner.compress(rawtext)
39333 5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39332 diff changeset	1464 compresseddeltalen = deltalen = dist = len(data[1]) + len(data[0])
5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39332 diff changeset	1465 deltabase = chainbase = curr
5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39332 diff changeset	1466 snapshotdepth = 0
5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39332 diff changeset	1467 chainlen = 1
5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39332 diff changeset	1468
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1469 return _deltainfo(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1470 dist,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1471 deltalen,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1472 data,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1473 deltabase,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1474 chainbase,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1475 chainlen,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1476 compresseddeltalen,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1477 snapshotdepth,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1478 )
39333 5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39332 diff changeset	1479
51017 509f0f7fc89e delta-computer: stop explicitly taking file handle Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50926 diff changeset	1480 def finddeltainfo(self, revinfo, excluded_bases=None, target_rev=None):
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1481 """Find an acceptable delta against a candidate revision
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1482
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1483 revinfo: information about the revision (instance of _revisioninfo)
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1484
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1485 Returns the first acceptable candidate revision, as ordered by
39334 507f5b1dd7c8 revlogdeltas: extract _getcandidaterevs in a function Boris Feld <boris.feld@octobus.net> parents: 39333 diff changeset	1486 _candidategroups
39333 5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39332 diff changeset	1487
5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39332 diff changeset	1488 If no suitable deltabase is found, we return delta info for a full
5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39332 diff changeset	1489 snapshot.
47401 1efe3cdef53a revlog: add a ways to blacklist some revision when searching for a delta Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47399 diff changeset	1490
1efe3cdef53a revlog: add a ways to blacklist some revision when searching for a delta Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47399 diff changeset	1491 `excluded_bases` is an optional set of revision that cannot be used as
1efe3cdef53a revlog: add a ways to blacklist some revision when searching for a delta Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47399 diff changeset	1492 a delta base. Use this to recompute delta suitable in censor or strip
1efe3cdef53a revlog: add a ways to blacklist some revision when searching for a delta Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47399 diff changeset	1493 context.
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1494 """
47456 93f4e183b3f5 deltas: at a `target_rev` parameter to finddeltainfo Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47452 diff changeset	1495 if target_rev is None:
47457 f8330a3fc39f censor: implement censoring for revlogv2 Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47456 diff changeset	1496 target_rev = len(self.revlog)
47456 93f4e183b3f5 deltas: at a `target_rev` parameter to finddeltainfo Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47452 diff changeset	1497
50650 f930b1b1d671 delta-find: initialize the debug information much sooner (when possible) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50649 diff changeset	1498 gather_debug = self._gather_debug
f930b1b1d671 delta-find: initialize the debug information much sooner (when possible) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50649 diff changeset	1499 cachedelta = revinfo.cachedelta
f930b1b1d671 delta-find: initialize the debug information much sooner (when possible) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50649 diff changeset	1500 revlog = self.revlog
50654 e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50652 diff changeset	1501 p1r = p2r = None
50650 f930b1b1d671 delta-find: initialize the debug information much sooner (when possible) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50649 diff changeset	1502
50654 e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50652 diff changeset	1503 if excluded_bases is None:
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50652 diff changeset	1504 excluded_bases = set()
50650 f930b1b1d671 delta-find: initialize the debug information much sooner (when possible) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50649 diff changeset	1505
f930b1b1d671 delta-find: initialize the debug information much sooner (when possible) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50649 diff changeset	1506 if gather_debug:
f930b1b1d671 delta-find: initialize the debug information much sooner (when possible) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50649 diff changeset	1507 start = util.timer()
f930b1b1d671 delta-find: initialize the debug information much sooner (when possible) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50649 diff changeset	1508 dbg = self._one_dbg_data()
f930b1b1d671 delta-find: initialize the debug information much sooner (when possible) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50649 diff changeset	1509 dbg['revision'] = target_rev
f930b1b1d671 delta-find: initialize the debug information much sooner (when possible) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50649 diff changeset	1510 p1r = revlog.rev(revinfo.p1)
f930b1b1d671 delta-find: initialize the debug information much sooner (when possible) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50649 diff changeset	1511 p2r = revlog.rev(revinfo.p2)
f930b1b1d671 delta-find: initialize the debug information much sooner (when possible) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50649 diff changeset	1512 if p1r != nullrev:
f930b1b1d671 delta-find: initialize the debug information much sooner (when possible) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50649 diff changeset	1513 p1_chain_len = revlog._chaininfo(p1r)[0]
f930b1b1d671 delta-find: initialize the debug information much sooner (when possible) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50649 diff changeset	1514 else:
f930b1b1d671 delta-find: initialize the debug information much sooner (when possible) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50649 diff changeset	1515 p1_chain_len = -1
f930b1b1d671 delta-find: initialize the debug information much sooner (when possible) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50649 diff changeset	1516 if p2r != nullrev:
f930b1b1d671 delta-find: initialize the debug information much sooner (when possible) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50649 diff changeset	1517 p2_chain_len = revlog._chaininfo(p2r)[0]
f930b1b1d671 delta-find: initialize the debug information much sooner (when possible) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50649 diff changeset	1518 else:
f930b1b1d671 delta-find: initialize the debug information much sooner (when possible) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50649 diff changeset	1519 p2_chain_len = -1
f930b1b1d671 delta-find: initialize the debug information much sooner (when possible) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50649 diff changeset	1520 dbg['p1-chain-len'] = p1_chain_len
f930b1b1d671 delta-find: initialize the debug information much sooner (when possible) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50649 diff changeset	1521 dbg['p2-chain-len'] = p2_chain_len
f930b1b1d671 delta-find: initialize the debug information much sooner (when possible) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50649 diff changeset	1522
50651 bfb6404089a5 delta-find: gather the condition to blindly use a full snapshot together Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50650 diff changeset	1523 # 1) if the revision is empty, no amount of delta can beat it
bfb6404089a5 delta-find: gather the condition to blindly use a full snapshot together Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50650 diff changeset	1524 #
bfb6404089a5 delta-find: gather the condition to blindly use a full snapshot together Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50650 diff changeset	1525 # 2) no delta for flag processor revision (see "candelta" for why)
bfb6404089a5 delta-find: gather the condition to blindly use a full snapshot together Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50650 diff changeset	1526 # not calling candelta since only one revision needs test, also to
bfb6404089a5 delta-find: gather the condition to blindly use a full snapshot together Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50650 diff changeset	1527 # avoid overhead fetching flags again.
bfb6404089a5 delta-find: gather the condition to blindly use a full snapshot together Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50650 diff changeset	1528 if not revinfo.textlen or revinfo.flags & REVIDX_RAWTEXT_CHANGING_FLAGS:
51017 509f0f7fc89e delta-computer: stop explicitly taking file handle Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50926 diff changeset	1529 deltainfo = self._fullsnapshotinfo(revinfo, target_rev)
50652 485c9410b75a deltafind: issue debug information when we fast-path rivial case too Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50651 diff changeset	1530 if gather_debug:
485c9410b75a deltafind: issue debug information when we fast-path rivial case too Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50651 diff changeset	1531 end = util.timer()
485c9410b75a deltafind: issue debug information when we fast-path rivial case too Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50651 diff changeset	1532 dbg['duration'] = end - start
485c9410b75a deltafind: issue debug information when we fast-path rivial case too Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50651 diff changeset	1533 dbg[
485c9410b75a deltafind: issue debug information when we fast-path rivial case too Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50651 diff changeset	1534 'delta-base'
485c9410b75a deltafind: issue debug information when we fast-path rivial case too Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50651 diff changeset	1535 ] = deltainfo.base # pytype: disable=attribute-error
485c9410b75a deltafind: issue debug information when we fast-path rivial case too Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50651 diff changeset	1536 dbg['search_round_count'] = 0
50654 e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50652 diff changeset	1537 dbg['using-cached-base'] = False
50652 485c9410b75a deltafind: issue debug information when we fast-path rivial case too Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50651 diff changeset	1538 dbg['delta_try_count'] = 0
485c9410b75a deltafind: issue debug information when we fast-path rivial case too Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50651 diff changeset	1539 dbg['type'] = b"full"
485c9410b75a deltafind: issue debug information when we fast-path rivial case too Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50651 diff changeset	1540 dbg['snapshot-depth'] = 0
485c9410b75a deltafind: issue debug information when we fast-path rivial case too Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50651 diff changeset	1541 self._dbg_process_data(dbg)
485c9410b75a deltafind: issue debug information when we fast-path rivial case too Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50651 diff changeset	1542 return deltainfo
39085 dbb3e9e44fce revlog: do not search for delta for empty content Boris Feld <boris.feld@octobus.net> parents: 39084 diff changeset	1543
50654 e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50652 diff changeset	1544 deltainfo = None
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50652 diff changeset	1545
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50652 diff changeset	1546 # If this source delta are to be forcibly reuse, let us comply early.
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50652 diff changeset	1547 if (
51043 f636103c4d67 revlog: remove legacy usage of `_generaldelta` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51042 diff changeset	1548 revlog.delta_config.general_delta
50654 e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50652 diff changeset	1549 and revinfo.cachedelta is not None
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50652 diff changeset	1550 and revinfo.cachedelta[2] == DELTA_BASE_REUSE_FORCE
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50652 diff changeset	1551 ):
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50652 diff changeset	1552 base = revinfo.cachedelta[0]
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50652 diff changeset	1553 if base == nullrev:
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50652 diff changeset	1554 dbg_type = b"full"
51017 509f0f7fc89e delta-computer: stop explicitly taking file handle Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50926 diff changeset	1555 deltainfo = self._fullsnapshotinfo(revinfo, target_rev)
50654 e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50652 diff changeset	1556 if gather_debug:
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50652 diff changeset	1557 snapshotdepth = 0
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50652 diff changeset	1558 elif base not in excluded_bases:
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50652 diff changeset	1559 delta = revinfo.cachedelta[1]
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50652 diff changeset	1560 header, data = revlog.compress(delta)
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50652 diff changeset	1561 deltalen = len(header) + len(data)
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50652 diff changeset	1562 if gather_debug:
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50652 diff changeset	1563 offset = revlog.end(len(revlog) - 1)
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50652 diff changeset	1564 chainbase = revlog.chainbase(base)
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50652 diff changeset	1565 distance = deltalen + offset - revlog.start(chainbase)
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50652 diff changeset	1566 chainlen, compresseddeltalen = revlog._chaininfo(base)
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50652 diff changeset	1567 chainlen += 1
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50652 diff changeset	1568 compresseddeltalen += deltalen
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50652 diff changeset	1569 if base == p1r or base == p2r:
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50652 diff changeset	1570 dbg_type = b"delta"
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50652 diff changeset	1571 snapshotdepth = None
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50652 diff changeset	1572 elif not revlog.issnapshot(base):
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50652 diff changeset	1573 snapshotdepth = None
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50652 diff changeset	1574 else:
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50652 diff changeset	1575 dbg_type = b"snapshot"
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50652 diff changeset	1576 snapshotdepth = revlog.snapshotdepth(base) + 1
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50652 diff changeset	1577 else:
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50652 diff changeset	1578 distance = None
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50652 diff changeset	1579 chainbase = None
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50652 diff changeset	1580 chainlen = None
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50652 diff changeset	1581 compresseddeltalen = None
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50652 diff changeset	1582 snapshotdepth = None
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50652 diff changeset	1583 deltainfo = _deltainfo(
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50652 diff changeset	1584 distance=distance,
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50652 diff changeset	1585 deltalen=deltalen,
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50652 diff changeset	1586 data=(header, data),
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50652 diff changeset	1587 base=base,
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50652 diff changeset	1588 chainbase=chainbase,
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50652 diff changeset	1589 chainlen=chainlen,
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50652 diff changeset	1590 compresseddeltalen=compresseddeltalen,
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50652 diff changeset	1591 snapshotdepth=snapshotdepth,
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50652 diff changeset	1592 )
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50652 diff changeset	1593
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50652 diff changeset	1594 if deltainfo is not None:
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50652 diff changeset	1595 if gather_debug:
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50652 diff changeset	1596 end = util.timer()
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50652 diff changeset	1597 dbg['duration'] = end - start
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50652 diff changeset	1598 dbg[
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50652 diff changeset	1599 'delta-base'
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50652 diff changeset	1600 ] = deltainfo.base # pytype: disable=attribute-error
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50652 diff changeset	1601 dbg['search_round_count'] = 0
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50652 diff changeset	1602 dbg['using-cached-base'] = True
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50652 diff changeset	1603 dbg['delta_try_count'] = 0
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50652 diff changeset	1604 dbg['type'] = b"full"
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50652 diff changeset	1605 if snapshotdepth is None:
51330 0d92d62ecde0 delta-find: use "-1" as depth snapshot-dept for non snapshot in debug Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51329 diff changeset	1606 dbg['snapshot-depth'] = -1
50654 e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50652 diff changeset	1607 else:
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50652 diff changeset	1608 dbg['snapshot-depth'] = snapshotdepth
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50652 diff changeset	1609 self._dbg_process_data(dbg)
e77ca247b85b delta-find: fix pulled-delta-reuse-policy=forced behavior Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50652 diff changeset	1610 return deltainfo
47401 1efe3cdef53a revlog: add a ways to blacklist some revision when searching for a delta Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47399 diff changeset	1611
49226 e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	1612 # count the number of different delta we tried (for debug purpose)
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	1613 dbg_try_count = 0
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	1614 # count the number of "search round" we did. (for debug purpose)
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	1615 dbg_try_rounds = 0
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	1616 dbg_type = b'unknown'
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	1617
50650 f930b1b1d671 delta-find: initialize the debug information much sooner (when possible) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50649 diff changeset	1618 if p1r is None:
f930b1b1d671 delta-find: initialize the debug information much sooner (when possible) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50649 diff changeset	1619 p1r = revlog.rev(revinfo.p1)
f930b1b1d671 delta-find: initialize the debug information much sooner (when possible) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50649 diff changeset	1620 p2r = revlog.rev(revinfo.p2)
49226 e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	1621
50646 c84cc0ac77e4 delta-fine: use the `_debug_search` attribute directly Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50645 diff changeset	1622 if self._debug_search:
49228 b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1623 msg = b"DBG-DELTAS-SEARCH: SEARCH rev=%d\n"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1624 msg %= target_rev
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1625 self._write_debug(msg)
49226 e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	1626
51344 555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51343 diff changeset	1627 # should we try to build a delta?
555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51343 diff changeset	1628 if not (len(self.revlog) and self.revlog._storedeltachains):
555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51343 diff changeset	1629 search_cls = _NoDeltaSearch
51354 d7e2acdd50ba delta-find: split the _DeltaSearch class in two Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	1630 elif self.revlog.delta_config.sparse_revlog:
d7e2acdd50ba delta-find: split the _DeltaSearch class in two Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	1631 search_cls = _SparseDeltaSearch
d7e2acdd50ba delta-find: split the _DeltaSearch class in two Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	1632 elif self.revlog.delta_config.general_delta:
d7e2acdd50ba delta-find: split the _DeltaSearch class in two Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	1633 search_cls = _GeneralDeltaSearch
d7e2acdd50ba delta-find: split the _DeltaSearch class in two Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51353 diff changeset	1634 else:
51344 555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51343 diff changeset	1635 # before general delta, there is only one possible delta base
555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51343 diff changeset	1636 search_cls = _PrevDeltaSearch
555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51343 diff changeset	1637
555826073625 delta-find: introduce and use specialized _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51343 diff changeset	1638 search = search_cls(
49611 2afee2176775 delta-find: expand a function definition and call before extendin it Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49610 diff changeset	1639 self.revlog,
51327 49401b7dec0c delta-find: feed revinfo to _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51325 diff changeset	1640 revinfo,
49611 2afee2176775 delta-find: expand a function definition and call before extendin it Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49610 diff changeset	1641 p1r,
2afee2176775 delta-find: expand a function definition and call before extendin it Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49610 diff changeset	1642 p2r,
2afee2176775 delta-find: expand a function definition and call before extendin it Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49610 diff changeset	1643 cachedelta,
49612 e706bb41fdb3 delta-find: move pre-filtering with other pre-filtering logic Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49611 diff changeset	1644 excluded_bases,
e706bb41fdb3 delta-find: move pre-filtering with other pre-filtering logic Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49611 diff changeset	1645 target_rev,
49681 c261a628e525 delta-find: use a single snapshot cache when applying a group to an object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49680 diff changeset	1646 snapshot_cache=self._snapshot_cache,
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1647 )
51317 c82e03b102a6 delta-find: introduce a _DeltaSearch object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51316 diff changeset	1648
51331 d0d869fccd20 delta-find: move away from the generator API for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51330 diff changeset	1649 while not search.done:
d0d869fccd20 delta-find: move away from the generator API for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51330 diff changeset	1650 current_group = search.current_group
d0d869fccd20 delta-find: move away from the generator API for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51330 diff changeset	1651 # current_group can be `None`, but not is search.done is False
d0d869fccd20 delta-find: move away from the generator API for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51330 diff changeset	1652 # We add this assert to help pytype
d0d869fccd20 delta-find: move away from the generator API for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51330 diff changeset	1653 assert current_group is not None
d0d869fccd20 delta-find: move away from the generator API for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51330 diff changeset	1654 candidaterevs = current_group
49226 e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	1655 dbg_try_rounds += 1
50646 c84cc0ac77e4 delta-fine: use the `_debug_search` attribute directly Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50645 diff changeset	1656 if self._debug_search:
49228 b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1657 prev = None
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1658 if deltainfo is not None:
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1659 prev = deltainfo.base
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1660
49608 78ba41878f2e delta-find: add debug information about reuse of cached data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49228 diff changeset	1661 if (
78ba41878f2e delta-find: add debug information about reuse of cached data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49228 diff changeset	1662 cachedelta is not None
78ba41878f2e delta-find: add debug information about reuse of cached data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49228 diff changeset	1663 and len(candidaterevs) == 1
78ba41878f2e delta-find: add debug information about reuse of cached data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49228 diff changeset	1664 and cachedelta[0] in candidaterevs
78ba41878f2e delta-find: add debug information about reuse of cached data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49228 diff changeset	1665 ):
78ba41878f2e delta-find: add debug information about reuse of cached data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49228 diff changeset	1666 round_type = b"cached-delta"
50649 ebb292ffdf4d delta-find: fix `parents` round detection Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50648 diff changeset	1667 elif p1r in candidaterevs or p2r in candidaterevs:
49228 b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1668 round_type = b"parents"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1669 elif prev is not None and all(c < prev for c in candidaterevs):
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1670 round_type = b"refine-down"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1671 elif prev is not None and all(c > prev for c in candidaterevs):
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1672 round_type = b"refine-up"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1673 else:
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1674 round_type = b"search-down"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1675 msg = b"DBG-DELTAS-SEARCH: ROUND #%d - %d candidates - %s\n"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1676 msg %= (dbg_try_rounds, len(candidaterevs), round_type)
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1677 self._write_debug(msg)
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1678 nominateddeltas = []
39498 04b75f3a3f2a snapshot: add refining logic at the findeltainfo level Boris Feld <boris.feld@octobus.net> parents: 39497 diff changeset	1679 if deltainfo is not None:
50646 c84cc0ac77e4 delta-fine: use the `_debug_search` attribute directly Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50645 diff changeset	1680 if self._debug_search:
49228 b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1681 msg = (
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1682 b"DBG-DELTAS-SEARCH: CONTENDER: rev=%d - length=%d\n"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1683 )
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1684 msg %= (deltainfo.base, deltainfo.deltalen)
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1685 self._write_debug(msg)
39498 04b75f3a3f2a snapshot: add refining logic at the findeltainfo level Boris Feld <boris.feld@octobus.net> parents: 39497 diff changeset	1686 # if we already found a good delta,
04b75f3a3f2a snapshot: add refining logic at the findeltainfo level Boris Feld <boris.feld@octobus.net> parents: 39497 diff changeset	1687 # challenge it against refined candidates
04b75f3a3f2a snapshot: add refining logic at the findeltainfo level Boris Feld <boris.feld@octobus.net> parents: 39497 diff changeset	1688 nominateddeltas.append(deltainfo)
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1689 for candidaterev in candidaterevs:
50646 c84cc0ac77e4 delta-fine: use the `_debug_search` attribute directly Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50645 diff changeset	1690 if self._debug_search:
49228 b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1691 msg = b"DBG-DELTAS-SEARCH: CANDIDATE: rev=%d\n"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1692 msg %= candidaterev
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1693 self._write_debug(msg)
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1694 candidate_type = None
50649 ebb292ffdf4d delta-find: fix `parents` round detection Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50648 diff changeset	1695 if candidaterev == p1r:
49228 b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1696 candidate_type = b"p1"
50649 ebb292ffdf4d delta-find: fix `parents` round detection Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50648 diff changeset	1697 elif candidaterev == p2r:
49228 b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1698 candidate_type = b"p2"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1699 elif self.revlog.issnapshot(candidaterev):
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1700 candidate_type = b"snapshot-%d"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1701 candidate_type %= self.revlog.snapshotdepth(
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1702 candidaterev
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1703 )
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1704
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1705 if candidate_type is not None:
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1706 msg = b"DBG-DELTAS-SEARCH: type=%s\n"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1707 msg %= candidate_type
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1708 self._write_debug(msg)
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1709 msg = b"DBG-DELTAS-SEARCH: size=%d\n"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1710 msg %= self.revlog.length(candidaterev)
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1711 self._write_debug(msg)
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1712 msg = b"DBG-DELTAS-SEARCH: base=%d\n"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1713 msg %= self.revlog.deltaparent(candidaterev)
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1714 self._write_debug(msg)
49612 e706bb41fdb3 delta-find: move pre-filtering with other pre-filtering logic Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49611 diff changeset	1715
49226 e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	1716 dbg_try_count += 1
49228 b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1717
50646 c84cc0ac77e4 delta-fine: use the `_debug_search` attribute directly Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50645 diff changeset	1718 if self._debug_search:
49228 b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1719 delta_start = util.timer()
50356 2a6949ab9d23 delta-find: add a simple safeguard to prevent bad non-general-delta Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50355 diff changeset	1720 candidatedelta = self._builddeltainfo(
2a6949ab9d23 delta-find: add a simple safeguard to prevent bad non-general-delta Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50355 diff changeset	1721 revinfo,
2a6949ab9d23 delta-find: add a simple safeguard to prevent bad non-general-delta Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50355 diff changeset	1722 candidaterev,
2a6949ab9d23 delta-find: add a simple safeguard to prevent bad non-general-delta Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50355 diff changeset	1723 target_rev=target_rev,
51347 670e68729aa7 delta-find: stop using heuristic to determine if we are creating a snapshot Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51346 diff changeset	1724 as_snapshot=search.current_stage == _STAGE_SNAPSHOT,
50356 2a6949ab9d23 delta-find: add a simple safeguard to prevent bad non-general-delta Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50355 diff changeset	1725 )
50646 c84cc0ac77e4 delta-fine: use the `_debug_search` attribute directly Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50645 diff changeset	1726 if self._debug_search:
49228 b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1727 delta_end = util.timer()
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1728 msg = b"DBG-DELTAS-SEARCH: delta-search-time=%f\n"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1729 msg %= delta_end - delta_start
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1730 self._write_debug(msg)
42466 465f2d0df9ae deltas: accept and skip None return for delta info Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42465 diff changeset	1731 if candidatedelta is not None:
51328 7455cae67260 delta-find: move is_good_delta_info on the _DeltaSearch class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51327 diff changeset	1732 if search.is_good_delta_info(candidatedelta):
50646 c84cc0ac77e4 delta-fine: use the `_debug_search` attribute directly Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50645 diff changeset	1733 if self._debug_search:
49228 b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1734 msg = b"DBG-DELTAS-SEARCH: DELTA: length=%d (GOOD)\n"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1735 msg %= candidatedelta.deltalen
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1736 self._write_debug(msg)
42466 465f2d0df9ae deltas: accept and skip None return for delta info Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42465 diff changeset	1737 nominateddeltas.append(candidatedelta)
50646 c84cc0ac77e4 delta-fine: use the `_debug_search` attribute directly Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50645 diff changeset	1738 elif self._debug_search:
49228 b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1739 msg = b"DBG-DELTAS-SEARCH: DELTA: length=%d (BAD)\n"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1740 msg %= candidatedelta.deltalen
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1741 self._write_debug(msg)
50646 c84cc0ac77e4 delta-fine: use the `_debug_search` attribute directly Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50645 diff changeset	1742 elif self._debug_search:
49228 b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1743 msg = b"DBG-DELTAS-SEARCH: NO-DELTA\n"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1744 self._write_debug(msg)
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1745 if nominateddeltas:
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1746 deltainfo = min(nominateddeltas, key=lambda x: x.deltalen)
51331 d0d869fccd20 delta-find: move away from the generator API for _DeltaSearch Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51330 diff changeset	1747 search.next_group(deltainfo)
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1748
39333 5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39332 diff changeset	1749 if deltainfo is None:
49226 e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	1750 dbg_type = b"full"
51017 509f0f7fc89e delta-computer: stop explicitly taking file handle Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50926 diff changeset	1751 deltainfo = self._fullsnapshotinfo(revinfo, target_rev)
49228 b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1752 elif deltainfo.snapshotdepth: # pytype: disable=attribute-error
49226 e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	1753 dbg_type = b"snapshot"
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	1754 else:
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	1755 dbg_type = b"delta"
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	1756
49610 35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	1757 if gather_debug:
49226 e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	1758 end = util.timer()
49775 d57b966cdeb1 delta-find: properly report full snapshot used from cache as such Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49768 diff changeset	1759 if dbg_type == b'full':
d57b966cdeb1 delta-find: properly report full snapshot used from cache as such Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49768 diff changeset	1760 used_cached = (
d57b966cdeb1 delta-find: properly report full snapshot used from cache as such Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49768 diff changeset	1761 cachedelta is not None
d57b966cdeb1 delta-find: properly report full snapshot used from cache as such Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49768 diff changeset	1762 and dbg_try_rounds == 0
d57b966cdeb1 delta-find: properly report full snapshot used from cache as such Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49768 diff changeset	1763 and dbg_try_count == 0
d57b966cdeb1 delta-find: properly report full snapshot used from cache as such Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49768 diff changeset	1764 and cachedelta[0] == nullrev
d57b966cdeb1 delta-find: properly report full snapshot used from cache as such Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49768 diff changeset	1765 )
d57b966cdeb1 delta-find: properly report full snapshot used from cache as such Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49768 diff changeset	1766 else:
d57b966cdeb1 delta-find: properly report full snapshot used from cache as such Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49768 diff changeset	1767 used_cached = (
d57b966cdeb1 delta-find: properly report full snapshot used from cache as such Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49768 diff changeset	1768 cachedelta is not None
d57b966cdeb1 delta-find: properly report full snapshot used from cache as such Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49768 diff changeset	1769 and dbg_try_rounds == 1
d57b966cdeb1 delta-find: properly report full snapshot used from cache as such Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49768 diff changeset	1770 and dbg_try_count == 1
d57b966cdeb1 delta-find: properly report full snapshot used from cache as such Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49768 diff changeset	1771 and deltainfo.base == cachedelta[0]
d57b966cdeb1 delta-find: properly report full snapshot used from cache as such Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49768 diff changeset	1772 )
50648 d1dc4a03125e delta-find: intrduce a `_one_dbg_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50647 diff changeset	1773 dbg['duration'] = end - start
d1dc4a03125e delta-find: intrduce a `_one_dbg_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50647 diff changeset	1774 dbg[
d1dc4a03125e delta-find: intrduce a `_one_dbg_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50647 diff changeset	1775 'delta-base'
d1dc4a03125e delta-find: intrduce a `_one_dbg_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50647 diff changeset	1776 ] = deltainfo.base # pytype: disable=attribute-error
d1dc4a03125e delta-find: intrduce a `_one_dbg_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50647 diff changeset	1777 dbg['search_round_count'] = dbg_try_rounds
d1dc4a03125e delta-find: intrduce a `_one_dbg_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50647 diff changeset	1778 dbg['using-cached-base'] = used_cached
d1dc4a03125e delta-find: intrduce a `_one_dbg_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50647 diff changeset	1779 dbg['delta_try_count'] = dbg_try_count
d1dc4a03125e delta-find: intrduce a `_one_dbg_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50647 diff changeset	1780 dbg['type'] = dbg_type
49228 b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1781 if (
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1782 deltainfo.snapshotdepth # pytype: disable=attribute-error
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1783 is not None
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1784 ):
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1785 dbg[
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1786 'snapshot-depth'
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1787 ] = deltainfo.snapshotdepth # pytype: disable=attribute-error
49226 e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	1788 else:
51330 0d92d62ecde0 delta-find: use "-1" as depth snapshot-dept for non snapshot in debug Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51329 diff changeset	1789 dbg['snapshot-depth'] = -1
50647 4d84b6d52e93 delta-find: move final debug processing in a `_dbg_process_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50646 diff changeset	1790 self._dbg_process_data(dbg)
4d84b6d52e93 delta-find: move final debug processing in a `_dbg_process_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50646 diff changeset	1791 return deltainfo
49226 e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	1792
50648 d1dc4a03125e delta-find: intrduce a `_one_dbg_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50647 diff changeset	1793 def _one_dbg_data(self):
51325 46415df4eb1c delta-find: move filing of some debug data in `_one_dbg_data` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51324 diff changeset	1794 dbg = {
50648 d1dc4a03125e delta-find: intrduce a `_one_dbg_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50647 diff changeset	1795 'duration': None,
d1dc4a03125e delta-find: intrduce a `_one_dbg_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50647 diff changeset	1796 'revision': None,
d1dc4a03125e delta-find: intrduce a `_one_dbg_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50647 diff changeset	1797 'delta-base': None,
d1dc4a03125e delta-find: intrduce a `_one_dbg_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50647 diff changeset	1798 'search_round_count': None,
d1dc4a03125e delta-find: intrduce a `_one_dbg_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50647 diff changeset	1799 'using-cached-base': None,
d1dc4a03125e delta-find: intrduce a `_one_dbg_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50647 diff changeset	1800 'delta_try_count': None,
d1dc4a03125e delta-find: intrduce a `_one_dbg_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50647 diff changeset	1801 'type': None,
d1dc4a03125e delta-find: intrduce a `_one_dbg_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50647 diff changeset	1802 'p1-chain-len': None,
d1dc4a03125e delta-find: intrduce a `_one_dbg_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50647 diff changeset	1803 'p2-chain-len': None,
d1dc4a03125e delta-find: intrduce a `_one_dbg_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50647 diff changeset	1804 'snapshot-depth': None,
d1dc4a03125e delta-find: intrduce a `_one_dbg_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50647 diff changeset	1805 'target-revlog': None,
d1dc4a03125e delta-find: intrduce a `_one_dbg_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50647 diff changeset	1806 }
51325 46415df4eb1c delta-find: move filing of some debug data in `_one_dbg_data` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51324 diff changeset	1807 target_revlog = b"UNKNOWN"
46415df4eb1c delta-find: move filing of some debug data in `_one_dbg_data` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51324 diff changeset	1808 target_type = self.revlog.target[0]
46415df4eb1c delta-find: move filing of some debug data in `_one_dbg_data` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51324 diff changeset	1809 target_key = self.revlog.target[1]
46415df4eb1c delta-find: move filing of some debug data in `_one_dbg_data` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51324 diff changeset	1810 if target_type == KIND_CHANGELOG:
46415df4eb1c delta-find: move filing of some debug data in `_one_dbg_data` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51324 diff changeset	1811 target_revlog = b'CHANGELOG:'
46415df4eb1c delta-find: move filing of some debug data in `_one_dbg_data` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51324 diff changeset	1812 elif target_type == KIND_MANIFESTLOG:
46415df4eb1c delta-find: move filing of some debug data in `_one_dbg_data` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51324 diff changeset	1813 target_revlog = b'MANIFESTLOG:'
46415df4eb1c delta-find: move filing of some debug data in `_one_dbg_data` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51324 diff changeset	1814 if target_key:
46415df4eb1c delta-find: move filing of some debug data in `_one_dbg_data` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51324 diff changeset	1815 target_revlog += b'%s:' % target_key
46415df4eb1c delta-find: move filing of some debug data in `_one_dbg_data` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51324 diff changeset	1816 elif target_type == KIND_FILELOG:
46415df4eb1c delta-find: move filing of some debug data in `_one_dbg_data` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51324 diff changeset	1817 target_revlog = b'FILELOG:'
46415df4eb1c delta-find: move filing of some debug data in `_one_dbg_data` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51324 diff changeset	1818 if target_key:
46415df4eb1c delta-find: move filing of some debug data in `_one_dbg_data` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51324 diff changeset	1819 target_revlog += b'%s:' % target_key
46415df4eb1c delta-find: move filing of some debug data in `_one_dbg_data` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51324 diff changeset	1820 dbg['target-revlog'] = target_revlog
46415df4eb1c delta-find: move filing of some debug data in `_one_dbg_data` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51324 diff changeset	1821 return dbg
50648 d1dc4a03125e delta-find: intrduce a `_one_dbg_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50647 diff changeset	1822
50647 4d84b6d52e93 delta-find: move final debug processing in a `_dbg_process_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50646 diff changeset	1823 def _dbg_process_data(self, dbg):
4d84b6d52e93 delta-find: move final debug processing in a `_dbg_process_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50646 diff changeset	1824 if self._debug_info is not None:
4d84b6d52e93 delta-find: move final debug processing in a `_dbg_process_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50646 diff changeset	1825 self._debug_info.append(dbg)
49610 35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	1826
50647 4d84b6d52e93 delta-find: move final debug processing in a `_dbg_process_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50646 diff changeset	1827 if self._write_debug is not None:
4d84b6d52e93 delta-find: move final debug processing in a `_dbg_process_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50646 diff changeset	1828 msg = (
4d84b6d52e93 delta-find: move final debug processing in a `_dbg_process_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50646 diff changeset	1829 b"DBG-DELTAS:"
4d84b6d52e93 delta-find: move final debug processing in a `_dbg_process_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50646 diff changeset	1830 b" %-12s"
4d84b6d52e93 delta-find: move final debug processing in a `_dbg_process_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50646 diff changeset	1831 b" rev=%d:"
4d84b6d52e93 delta-find: move final debug processing in a `_dbg_process_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50646 diff changeset	1832 b" delta-base=%d"
4d84b6d52e93 delta-find: move final debug processing in a `_dbg_process_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50646 diff changeset	1833 b" is-cached=%d"
4d84b6d52e93 delta-find: move final debug processing in a `_dbg_process_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50646 diff changeset	1834 b" - search-rounds=%d"
4d84b6d52e93 delta-find: move final debug processing in a `_dbg_process_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50646 diff changeset	1835 b" try-count=%d"
4d84b6d52e93 delta-find: move final debug processing in a `_dbg_process_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50646 diff changeset	1836 b" - delta-type=%-6s"
4d84b6d52e93 delta-find: move final debug processing in a `_dbg_process_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50646 diff changeset	1837 b" snap-depth=%d"
4d84b6d52e93 delta-find: move final debug processing in a `_dbg_process_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50646 diff changeset	1838 b" - p1-chain-length=%d"
4d84b6d52e93 delta-find: move final debug processing in a `_dbg_process_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50646 diff changeset	1839 b" p2-chain-length=%d"
4d84b6d52e93 delta-find: move final debug processing in a `_dbg_process_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50646 diff changeset	1840 b" - duration=%f"
4d84b6d52e93 delta-find: move final debug processing in a `_dbg_process_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50646 diff changeset	1841 b"\n"
4d84b6d52e93 delta-find: move final debug processing in a `_dbg_process_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50646 diff changeset	1842 )
4d84b6d52e93 delta-find: move final debug processing in a `_dbg_process_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50646 diff changeset	1843 msg %= (
4d84b6d52e93 delta-find: move final debug processing in a `_dbg_process_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50646 diff changeset	1844 dbg["target-revlog"],
4d84b6d52e93 delta-find: move final debug processing in a `_dbg_process_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50646 diff changeset	1845 dbg["revision"],
4d84b6d52e93 delta-find: move final debug processing in a `_dbg_process_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50646 diff changeset	1846 dbg["delta-base"],
4d84b6d52e93 delta-find: move final debug processing in a `_dbg_process_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50646 diff changeset	1847 dbg["using-cached-base"],
4d84b6d52e93 delta-find: move final debug processing in a `_dbg_process_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50646 diff changeset	1848 dbg["search_round_count"],
4d84b6d52e93 delta-find: move final debug processing in a `_dbg_process_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50646 diff changeset	1849 dbg["delta_try_count"],
4d84b6d52e93 delta-find: move final debug processing in a `_dbg_process_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50646 diff changeset	1850 dbg["type"],
4d84b6d52e93 delta-find: move final debug processing in a `_dbg_process_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50646 diff changeset	1851 dbg["snapshot-depth"],
4d84b6d52e93 delta-find: move final debug processing in a `_dbg_process_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50646 diff changeset	1852 dbg["p1-chain-len"],
4d84b6d52e93 delta-find: move final debug processing in a `_dbg_process_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50646 diff changeset	1853 dbg["p2-chain-len"],
4d84b6d52e93 delta-find: move final debug processing in a `_dbg_process_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50646 diff changeset	1854 dbg["duration"],
4d84b6d52e93 delta-find: move final debug processing in a `_dbg_process_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50646 diff changeset	1855 )
4d84b6d52e93 delta-find: move final debug processing in a `_dbg_process_data` method Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50646 diff changeset	1856 self._write_debug(msg)
47452 c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47401 diff changeset	1857
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47401 diff changeset	1858
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47401 diff changeset	1859 def delta_compression(default_compression_header, deltainfo):
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47401 diff changeset	1860 """return (COMPRESSION_MODE, deltainfo)
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47401 diff changeset	1861
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47401 diff changeset	1862 used by revlog v2+ format to dispatch between PLAIN and DEFAULT
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47401 diff changeset	1863 compression.
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47401 diff changeset	1864 """
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47401 diff changeset	1865 h, d = deltainfo.data
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47401 diff changeset	1866 compression_mode = COMP_MODE_INLINE
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47401 diff changeset	1867 if not h and not d:
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47401 diff changeset	1868 # not data to store at all... declare them uncompressed
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47401 diff changeset	1869 compression_mode = COMP_MODE_PLAIN
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47401 diff changeset	1870 elif not h:
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47401 diff changeset	1871 t = d[0:1]
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47401 diff changeset	1872 if t == b'\0':
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47401 diff changeset	1873 compression_mode = COMP_MODE_PLAIN
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47401 diff changeset	1874 elif t == default_compression_header:
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47401 diff changeset	1875 compression_mode = COMP_MODE_DEFAULT
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47401 diff changeset	1876 elif h == b'u':
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47401 diff changeset	1877 # we have a more efficient way to declare uncompressed
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47401 diff changeset	1878 h = b''
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47401 diff changeset	1879 compression_mode = COMP_MODE_PLAIN
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47401 diff changeset	1880 deltainfo = drop_u_compression(deltainfo)
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47401 diff changeset	1881 return compression_mode, deltainfo

Mercurial > public > mercurial-scm > hg

annotate mercurial/revlogutils/deltas.py @ 51355:383e99f6bc99