mercurial-scm/hg: mercurial/revlogutils/deltas.py annotate

annotate mercurial/revlogutils/deltas.py @ 49681:c261a628e525

delta-find: use a single snapshot cache when applying a group to an object This will avoid walking the revlog over and over again in some situations. The difference is hard to show in our current benchmark suite, as the gain is lower than their overall instability.

author	Pierre-Yves David <pierre-yves.david@octobus.net>
date	Sun, 06 Nov 2022 17:53:17 -0500
parents	40e24d82b513
children	bcae90c53def

rev	line source
39330 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	1 # revlogdeltas.py - Logic around delta computation for revlog
8226 8b2cd04a6e97 put license and copyright info into comment blocks Martin Geisler <mg@lazybytes.net> parents: 8225 diff changeset	2 #
46819 d4ba4d51f85f contributor: change mentions of mpm to olivia Rapha?l Gom?s <rgomes@octobus.net> parents: 43089 diff changeset	3 # Copyright 2005-2007 Olivia Mackall <olivia@selenic.com>
39330 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	4 # Copyright 2018 Octobus <contact@octobus.net>
8226 8b2cd04a6e97 put license and copyright info into comment blocks Martin Geisler <mg@lazybytes.net> parents: 8225 diff changeset	5 #
8b2cd04a6e97 put license and copyright info into comment blocks Martin Geisler <mg@lazybytes.net> parents: 8225 diff changeset	6 # This software may be used and distributed according to the terms of the
10263 25e572394f5c Update license to GPLv2+ Matt Mackall <mpm@selenic.com> parents: 10047 diff changeset	7 # GNU General Public License version 2 or any later version.
39330 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	8 """Helper class to compute deltas stored inside revlogs"""
8227 0a9542703300 turn some comments back into module docstrings Martin Geisler <mg@lazybytes.net> parents: 8226 diff changeset	9
27361 29f50344fa83 revlog: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27251 diff changeset	10
39493 3ca144f1c8dd snapshot: search for unrelated but reusable full-snapshot Boris Feld <boris.feld@octobus.net> parents: 39492 diff changeset	11 import collections
27361 29f50344fa83 revlog: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27251 diff changeset	12 import struct
29f50344fa83 revlog: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27251 diff changeset	13
29f50344fa83 revlog: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27251 diff changeset	14 # import stuff from node for others to import from revlog
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	15 from ..node import nullrev
39330 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	16 from ..i18n import _
43089 c59eb1560c44 py3: manually import getattr where it is needed Gregory Szorc <gregory.szorc@gmail.com> parents: 43077 diff changeset	17 from ..pycompat import getattr
39330 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	18
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	19 from .constants import (
47452 c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47401 diff changeset	20 COMP_MODE_DEFAULT,
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47401 diff changeset	21 COMP_MODE_INLINE,
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47401 diff changeset	22 COMP_MODE_PLAIN,
49677 05db41701ece find-delta: pass the cache-delta usage policy alongside the cache-delta Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49675 diff changeset	23 DELTA_BASE_REUSE_NO,
49226 e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	24 KIND_CHANGELOG,
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	25 KIND_FILELOG,
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	26 KIND_MANIFESTLOG,
39329 729082bb9938 revlog: split constants into a new `revlogutils.constants` module Boris Feld <boris.feld@octobus.net> parents: 39232 diff changeset	27 REVIDX_ISCENSORED,
729082bb9938 revlog: split constants into a new `revlogutils.constants` module Boris Feld <boris.feld@octobus.net> parents: 39232 diff changeset	28 REVIDX_RAWTEXT_CHANGING_FLAGS,
729082bb9938 revlog: split constants into a new `revlogutils.constants` module Boris Feld <boris.feld@octobus.net> parents: 39232 diff changeset	29 )
39330 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	30
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	31 from ..thirdparty import attr
39330 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	32
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	33 from .. import (
27361 29f50344fa83 revlog: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27251 diff changeset	34 error,
29f50344fa83 revlog: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27251 diff changeset	35 mdiff,
41108 38e88450138c delta: have a native implementation of _findsnapshot Boris Feld <boris.feld@octobus.net> parents: 41079 diff changeset	36 util,
27361 29f50344fa83 revlog: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27251 diff changeset	37 )
10913 f2ecc5733c89 revlog: factor out _maxinline global. Greg Ward <greg-hg@gerg.ca> parents: 10404 diff changeset	38
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	39 from . import flagutil
42992 dff95420480f flagprocessors: make `processflagsraw` a module level function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42878 diff changeset	40
39330 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	41 # maximum <delta-chain-data>/<revision-text-length> ratio
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	42 LIMIT_DELTA2TEXT = 2
1091 d62130f99a73 Move hash function back to revlog from node mpm@selenic.com parents: 1089 diff changeset	43
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	44
48946 642e31cb55f0 py3: use class X: instead of class X(object): Gregory Szorc <gregory.szorc@gmail.com> parents: 48875 diff changeset	45 class _testrevlog:
38637 e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	46 """minimalist fake revlog to use in doctests"""
e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	47
40641 85b14f0dc334 doctest: add a `issnapshot` method to _testrevlog Boris Feld <boris.feld@octobus.net> parents: 40608 diff changeset	48 def __init__(self, data, density=0.5, mingap=0, snapshot=()):
38637 e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	49 """data is an list of revision payload boundaries"""
e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	50 self._data = data
e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	51 self._srdensitythreshold = density
e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	52 self._srmingapsize = mingap
40641 85b14f0dc334 doctest: add a `issnapshot` method to _testrevlog Boris Feld <boris.feld@octobus.net> parents: 40608 diff changeset	53 self._snapshot = set(snapshot)
40709 39d29542fe40 sparse-revlog: put the native implementation of slicechunktodensity to use Boris Feld <boris.feld@octobus.net> parents: 40657 diff changeset	54 self.index = None
38637 e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	55
e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	56 def start(self, rev):
41079 88d813cd9acd revlog: fix pure python slicing test when chain contains nullrev Boris Feld <boris.feld@octobus.net> parents: 41033 diff changeset	57 if rev == nullrev:
88d813cd9acd revlog: fix pure python slicing test when chain contains nullrev Boris Feld <boris.feld@octobus.net> parents: 41033 diff changeset	58 return 0
38637 e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	59 if rev == 0:
e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	60 return 0
e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	61 return self._data[rev - 1]
e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	62
e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	63 def end(self, rev):
41079 88d813cd9acd revlog: fix pure python slicing test when chain contains nullrev Boris Feld <boris.feld@octobus.net> parents: 41033 diff changeset	64 if rev == nullrev:
88d813cd9acd revlog: fix pure python slicing test when chain contains nullrev Boris Feld <boris.feld@octobus.net> parents: 41033 diff changeset	65 return 0
38637 e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	66 return self._data[rev]
e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	67
e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	68 def length(self, rev):
e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	69 return self.end(rev) - self.start(rev)
e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	70
38718 f8762ea73e0d sparse-revlog: implement algorithm to write sparse delta chains (issue5480) Paul Morelle <paul.morelle@octobus.net> parents: 38717 diff changeset	71 def __len__(self):
f8762ea73e0d sparse-revlog: implement algorithm to write sparse delta chains (issue5480) Paul Morelle <paul.morelle@octobus.net> parents: 38717 diff changeset	72 return len(self._data)
f8762ea73e0d sparse-revlog: implement algorithm to write sparse delta chains (issue5480) Paul Morelle <paul.morelle@octobus.net> parents: 38717 diff changeset	73
40641 85b14f0dc334 doctest: add a `issnapshot` method to _testrevlog Boris Feld <boris.feld@octobus.net> parents: 40608 diff changeset	74 def issnapshot(self, rev):
41079 88d813cd9acd revlog: fix pure python slicing test when chain contains nullrev Boris Feld <boris.feld@octobus.net> parents: 41033 diff changeset	75 if rev == nullrev:
88d813cd9acd revlog: fix pure python slicing test when chain contains nullrev Boris Feld <boris.feld@octobus.net> parents: 41033 diff changeset	76 return True
40641 85b14f0dc334 doctest: add a `issnapshot` method to _testrevlog Boris Feld <boris.feld@octobus.net> parents: 40608 diff changeset	77 return rev in self._snapshot
85b14f0dc334 doctest: add a `issnapshot` method to _testrevlog Boris Feld <boris.feld@octobus.net> parents: 40608 diff changeset	78
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	79
40604 3ac23dad6364 sparse-revlog: drop unused deltainfo parameter from _slicechunktodensity Boris Feld <boris.feld@octobus.net> parents: 40603 diff changeset	80 def slicechunk(revlog, revs, targetsize=None):
34824 e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	81 """slice revs to reduce the amount of unrelated data to be read from disk.
e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	82
e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	83 ``revs`` is sliced into groups that should be read in one time.
e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	84 Assume that revs are sorted.
38640 f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	85
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	86 The initial chunk is sliced until the overall density (payload/chunks-span
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	87 ratio) is above `revlog._srdensitythreshold`. No gap smaller than
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	88 `revlog._srmingapsize` is skipped.
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	89
38643 967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size Boris Feld <boris.feld@octobus.net> parents: 38642 diff changeset	90 If `targetsize` is set, no chunk larger than `targetsize` will be yield.
967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size Boris Feld <boris.feld@octobus.net> parents: 38642 diff changeset	91 For consistency with other slicing choice, this limit won't go lower than
967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size Boris Feld <boris.feld@octobus.net> parents: 38642 diff changeset	92 `revlog._srmingapsize`.
967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size Boris Feld <boris.feld@octobus.net> parents: 38642 diff changeset	93
967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size Boris Feld <boris.feld@octobus.net> parents: 38642 diff changeset	94 If individual revisions chunk are larger than this limit, they will still
967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size Boris Feld <boris.feld@octobus.net> parents: 38642 diff changeset	95 be raised individually.
967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size Boris Feld <boris.feld@octobus.net> parents: 38642 diff changeset	96
40642 9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	97 >>> data = [
38640 f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	98 ... 5, #00 (5)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	99 ... 10, #01 (5)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	100 ... 12, #02 (2)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	101 ... 12, #03 (empty)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	102 ... 27, #04 (15)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	103 ... 31, #05 (4)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	104 ... 31, #06 (empty)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	105 ... 42, #07 (11)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	106 ... 47, #08 (5)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	107 ... 47, #09 (empty)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	108 ... 48, #10 (1)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	109 ... 51, #11 (3)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	110 ... 74, #12 (23)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	111 ... 85, #13 (11)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	112 ... 86, #14 (1)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	113 ... 91, #15 (5)
40642 9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	114 ... ]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	115 >>> revlog = _testrevlog(data, snapshot=range(16))
38640 f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	116
39330 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	117 >>> list(slicechunk(revlog, list(range(16))))
38640 f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	118 [[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]]
39330 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	119 >>> list(slicechunk(revlog, [0, 15]))
38640 f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	120 [[0], [15]]
39330 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	121 >>> list(slicechunk(revlog, [0, 11, 15]))
38640 f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	122 [[0], [11], [15]]
39330 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	123 >>> list(slicechunk(revlog, [0, 11, 13, 15]))
38640 f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	124 [[0], [11, 13, 15]]
39330 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	125 >>> list(slicechunk(revlog, [1, 2, 3, 5, 8, 10, 11, 14]))
38640 f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	126 [[1, 2], [5, 8, 10, 11], [14]]
38643 967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size Boris Feld <boris.feld@octobus.net> parents: 38642 diff changeset	127
967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size Boris Feld <boris.feld@octobus.net> parents: 38642 diff changeset	128 Slicing with a maximum chunk size
39330 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	129 >>> list(slicechunk(revlog, [0, 11, 13, 15], targetsize=15))
38643 967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size Boris Feld <boris.feld@octobus.net> parents: 38642 diff changeset	130 [[0], [11], [13], [15]]
39330 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	131 >>> list(slicechunk(revlog, [0, 11, 13, 15], targetsize=20))
38643 967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size Boris Feld <boris.feld@octobus.net> parents: 38642 diff changeset	132 [[0], [11], [13, 15]]
41079 88d813cd9acd revlog: fix pure python slicing test when chain contains nullrev Boris Feld <boris.feld@octobus.net> parents: 41033 diff changeset	133
88d813cd9acd revlog: fix pure python slicing test when chain contains nullrev Boris Feld <boris.feld@octobus.net> parents: 41033 diff changeset	134 Slicing involving nullrev
88d813cd9acd revlog: fix pure python slicing test when chain contains nullrev Boris Feld <boris.feld@octobus.net> parents: 41033 diff changeset	135 >>> list(slicechunk(revlog, [-1, 0, 11, 13, 15], targetsize=20))
88d813cd9acd revlog: fix pure python slicing test when chain contains nullrev Boris Feld <boris.feld@octobus.net> parents: 41033 diff changeset	136 [[-1, 0], [11], [13, 15]]
88d813cd9acd revlog: fix pure python slicing test when chain contains nullrev Boris Feld <boris.feld@octobus.net> parents: 41033 diff changeset	137 >>> list(slicechunk(revlog, [-1, 13, 15], targetsize=5))
88d813cd9acd revlog: fix pure python slicing test when chain contains nullrev Boris Feld <boris.feld@octobus.net> parents: 41033 diff changeset	138 [[-1], [13], [15]]
34824 e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	139 """
38643 967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size Boris Feld <boris.feld@octobus.net> parents: 38642 diff changeset	140 if targetsize is not None:
967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size Boris Feld <boris.feld@octobus.net> parents: 38642 diff changeset	141 targetsize = max(targetsize, revlog._srmingapsize)
38718 f8762ea73e0d sparse-revlog: implement algorithm to write sparse delta chains (issue5480) Paul Morelle <paul.morelle@octobus.net> parents: 38717 diff changeset	142 # targetsize should not be specified when evaluating delta candidates:
f8762ea73e0d sparse-revlog: implement algorithm to write sparse delta chains (issue5480) Paul Morelle <paul.morelle@octobus.net> parents: 38717 diff changeset	143 # * targetsize is used to ensure we stay within specification when reading,
40709 39d29542fe40 sparse-revlog: put the native implementation of slicechunktodensity to use Boris Feld <boris.feld@octobus.net> parents: 40657 diff changeset	144 densityslicing = getattr(revlog.index, 'slicechunktodensity', None)
39d29542fe40 sparse-revlog: put the native implementation of slicechunktodensity to use Boris Feld <boris.feld@octobus.net> parents: 40657 diff changeset	145 if densityslicing is None:
39d29542fe40 sparse-revlog: put the native implementation of slicechunktodensity to use Boris Feld <boris.feld@octobus.net> parents: 40657 diff changeset	146 densityslicing = lambda x, y, z: _slicechunktodensity(revlog, x, y, z)
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	147 for chunk in densityslicing(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	148 revs, revlog._srdensitythreshold, revlog._srmingapsize
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	149 ):
38643 967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size Boris Feld <boris.feld@octobus.net> parents: 38642 diff changeset	150 for subchunk in _slicechunktosize(revlog, chunk, targetsize):
967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size Boris Feld <boris.feld@octobus.net> parents: 38642 diff changeset	151 yield subchunk
38641 feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	152
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	153
38718 f8762ea73e0d sparse-revlog: implement algorithm to write sparse delta chains (issue5480) Paul Morelle <paul.morelle@octobus.net> parents: 38717 diff changeset	154 def _slicechunktosize(revlog, revs, targetsize=None):
38642 e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	155 """slice revs to match the target size
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	156
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	157 This is intended to be used on chunk that density slicing selected by that
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	158 are still too large compared to the read garantee of revlog. This might
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	159 happens when "minimal gap size" interrupted the slicing or when chain are
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	160 built in a way that create large blocks next to each other.
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	161
40642 9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	162 >>> data = [
38642 e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	163 ... 3, #0 (3)
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	164 ... 5, #1 (2)
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	165 ... 6, #2 (1)
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	166 ... 8, #3 (2)
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	167 ... 8, #4 (empty)
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	168 ... 11, #5 (3)
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	169 ... 12, #6 (1)
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	170 ... 13, #7 (1)
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	171 ... 14, #8 (1)
40642 9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	172 ... ]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	173
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	174 == All snapshots cases ==
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	175 >>> revlog = _testrevlog(data, snapshot=range(9))
38642 e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	176
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	177 Cases where chunk is already small enough
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	178 >>> list(_slicechunktosize(revlog, [0], 3))
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	179 [[0]]
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	180 >>> list(_slicechunktosize(revlog, [6, 7], 3))
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	181 [[6, 7]]
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	182 >>> list(_slicechunktosize(revlog, [0], None))
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	183 [[0]]
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	184 >>> list(_slicechunktosize(revlog, [6, 7], None))
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	185 [[6, 7]]
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	186
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	187 cases where we need actual slicing
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	188 >>> list(_slicechunktosize(revlog, [0, 1], 3))
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	189 [[0], [1]]
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	190 >>> list(_slicechunktosize(revlog, [1, 3], 3))
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	191 [[1], [3]]
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	192 >>> list(_slicechunktosize(revlog, [1, 2, 3], 3))
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	193 [[1, 2], [3]]
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	194 >>> list(_slicechunktosize(revlog, [3, 5], 3))
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	195 [[3], [5]]
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	196 >>> list(_slicechunktosize(revlog, [3, 4, 5], 3))
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	197 [[3], [5]]
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	198 >>> list(_slicechunktosize(revlog, [5, 6, 7, 8], 3))
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	199 [[5], [6, 7, 8]]
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	200 >>> list(_slicechunktosize(revlog, [0, 1, 2, 3, 4, 5, 6, 7, 8], 3))
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	201 [[0], [1, 2], [3], [5], [6, 7, 8]]
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	202
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	203 Case with too large individual chunk (must return valid chunk)
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	204 >>> list(_slicechunktosize(revlog, [0, 1], 2))
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	205 [[0], [1]]
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	206 >>> list(_slicechunktosize(revlog, [1, 3], 1))
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	207 [[1], [3]]
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	208 >>> list(_slicechunktosize(revlog, [3, 4, 5], 2))
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	209 [[3], [5]]
40642 9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	210
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	211 == No Snapshot cases ==
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	212 >>> revlog = _testrevlog(data)
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	213
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	214 Cases where chunk is already small enough
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	215 >>> list(_slicechunktosize(revlog, [0], 3))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	216 [[0]]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	217 >>> list(_slicechunktosize(revlog, [6, 7], 3))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	218 [[6, 7]]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	219 >>> list(_slicechunktosize(revlog, [0], None))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	220 [[0]]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	221 >>> list(_slicechunktosize(revlog, [6, 7], None))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	222 [[6, 7]]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	223
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	224 cases where we need actual slicing
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	225 >>> list(_slicechunktosize(revlog, [0, 1], 3))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	226 [[0], [1]]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	227 >>> list(_slicechunktosize(revlog, [1, 3], 3))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	228 [[1], [3]]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	229 >>> list(_slicechunktosize(revlog, [1, 2, 3], 3))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	230 [[1], [2, 3]]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	231 >>> list(_slicechunktosize(revlog, [3, 5], 3))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	232 [[3], [5]]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	233 >>> list(_slicechunktosize(revlog, [3, 4, 5], 3))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	234 [[3], [4, 5]]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	235 >>> list(_slicechunktosize(revlog, [5, 6, 7, 8], 3))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	236 [[5], [6, 7, 8]]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	237 >>> list(_slicechunktosize(revlog, [0, 1, 2, 3, 4, 5, 6, 7, 8], 3))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	238 [[0], [1, 2], [3], [5], [6, 7, 8]]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	239
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	240 Case with too large individual chunk (must return valid chunk)
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	241 >>> list(_slicechunktosize(revlog, [0, 1], 2))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	242 [[0], [1]]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	243 >>> list(_slicechunktosize(revlog, [1, 3], 1))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	244 [[1], [3]]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	245 >>> list(_slicechunktosize(revlog, [3, 4, 5], 2))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	246 [[3], [5]]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	247
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	248 == mixed case ==
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	249 >>> revlog = _testrevlog(data, snapshot=[0, 1, 2])
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	250 >>> list(_slicechunktosize(revlog, list(range(9)), 5))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	251 [[0, 1], [2], [3, 4, 5], [6, 7, 8]]
38642 e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	252 """
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	253 assert targetsize is None or 0 <= targetsize
40642 9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	254 startdata = revlog.start(revs[0])
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	255 enddata = revlog.end(revs[-1])
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	256 fullspan = enddata - startdata
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	257 if targetsize is None or fullspan <= targetsize:
38642 e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	258 yield revs
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	259 return
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	260
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	261 startrevidx = 0
40657 2eb48aa0acce sparse-revlog: align endrevidx usages in the _slicechunktosize Boris Feld <boris.feld@octobus.net> parents: 40654 diff changeset	262 endrevidx = 1
38642 e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	263 iterrevs = enumerate(revs)
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	264 next(iterrevs) # skip first rev.
40642 9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	265 # first step: get snapshots out of the way
38642 e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	266 for idx, r in iterrevs:
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	267 span = revlog.end(r) - startdata
40642 9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	268 snapshot = revlog.issnapshot(r)
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	269 if span <= targetsize and snapshot:
40657 2eb48aa0acce sparse-revlog: align endrevidx usages in the _slicechunktosize Boris Feld <boris.feld@octobus.net> parents: 40654 diff changeset	270 endrevidx = idx + 1
38642 e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	271 else:
40657 2eb48aa0acce sparse-revlog: align endrevidx usages in the _slicechunktosize Boris Feld <boris.feld@octobus.net> parents: 40654 diff changeset	272 chunk = _trimchunk(revlog, revs, startrevidx, endrevidx)
38642 e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	273 if chunk:
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	274 yield chunk
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	275 startrevidx = idx
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	276 startdata = revlog.start(r)
40657 2eb48aa0acce sparse-revlog: align endrevidx usages in the _slicechunktosize Boris Feld <boris.feld@octobus.net> parents: 40654 diff changeset	277 endrevidx = idx + 1
40642 9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	278 if not snapshot:
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	279 break
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	280
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	281 # for the others, we use binary slicing to quickly converge toward valid
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	282 # chunks (otherwise, we might end up looking for start/end of many
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	283 # revisions). This logic is not looking for the perfect slicing point, it
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	284 # focuses on quickly converging toward valid chunks.
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	285 nbitem = len(revs)
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	286 while (enddata - startdata) > targetsize:
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	287 endrevidx = nbitem
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	288 if nbitem - startrevidx <= 1:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	289 break # protect against individual chunk larger than limit
40642 9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	290 localenddata = revlog.end(revs[endrevidx - 1])
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	291 span = localenddata - startdata
40654 fd1d41ccbe38 sparse-revlog: use `span` variable as intended Boris Feld <boris.feld@octobus.net> parents: 40642 diff changeset	292 while span > targetsize:
40642 9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	293 if endrevidx - startrevidx <= 1:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	294 break # protect against individual chunk larger than limit
40642 9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	295 endrevidx -= (endrevidx - startrevidx) // 2
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	296 localenddata = revlog.end(revs[endrevidx - 1])
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	297 span = localenddata - startdata
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	298 chunk = _trimchunk(revlog, revs, startrevidx, endrevidx)
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	299 if chunk:
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	300 yield chunk
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	301 startrevidx = endrevidx
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	302 startdata = revlog.start(revs[startrevidx])
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	303
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	304 chunk = _trimchunk(revlog, revs, startrevidx)
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	305 if chunk:
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40641 diff changeset	306 yield chunk
38642 e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	307
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	308
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	309 def _slicechunktodensity(revlog, revs, targetdensity=0.5, mingapsize=0):
38641 feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	310 """slice revs to reduce the amount of unrelated data to be read from disk.
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	311
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	312 ``revs`` is sliced into groups that should be read in one time.
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	313 Assume that revs are sorted.
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	314
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	315 The initial chunk is sliced until the overall density (payload/chunks-span
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	316 ratio) is above `targetdensity`. No gap smaller than `mingapsize` is
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	317 skipped.
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	318
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	319 >>> revlog = _testrevlog([
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	320 ... 5, #00 (5)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	321 ... 10, #01 (5)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	322 ... 12, #02 (2)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	323 ... 12, #03 (empty)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	324 ... 27, #04 (15)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	325 ... 31, #05 (4)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	326 ... 31, #06 (empty)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	327 ... 42, #07 (11)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	328 ... 47, #08 (5)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	329 ... 47, #09 (empty)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	330 ... 48, #10 (1)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	331 ... 51, #11 (3)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	332 ... 74, #12 (23)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	333 ... 85, #13 (11)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	334 ... 86, #14 (1)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	335 ... 91, #15 (5)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	336 ... ])
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	337
38655 cd1c484e31e8 revlog: adjust doctest examples to be portable to Python 3 Augie Fackler <augie@google.com> parents: 38644 diff changeset	338 >>> list(_slicechunktodensity(revlog, list(range(16))))
38641 feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	339 [[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]]
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	340 >>> list(_slicechunktodensity(revlog, [0, 15]))
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	341 [[0], [15]]
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	342 >>> list(_slicechunktodensity(revlog, [0, 11, 15]))
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	343 [[0], [11], [15]]
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	344 >>> list(_slicechunktodensity(revlog, [0, 11, 13, 15]))
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	345 [[0], [11, 13, 15]]
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	346 >>> list(_slicechunktodensity(revlog, [1, 2, 3, 5, 8, 10, 11, 14]))
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	347 [[1, 2], [5, 8, 10, 11], [14]]
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	348 >>> list(_slicechunktodensity(revlog, [1, 2, 3, 5, 8, 10, 11, 14],
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	349 ... mingapsize=20))
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	350 [[1, 2, 3, 5, 8, 10, 11], [14]]
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	351 >>> list(_slicechunktodensity(revlog, [1, 2, 3, 5, 8, 10, 11, 14],
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	352 ... targetdensity=0.95))
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	353 [[1, 2], [5], [8, 10, 11], [14]]
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	354 >>> list(_slicechunktodensity(revlog, [1, 2, 3, 5, 8, 10, 11, 14],
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	355 ... targetdensity=0.95, mingapsize=12))
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	356 [[1, 2], [5, 8, 10, 11], [14]]
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	357 """
34824 e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	358 start = revlog.start
e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	359 length = revlog.length
e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	360
34880 9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	361 if len(revs) <= 1:
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	362 yield revs
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	363 return
34824 e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	364
40604 3ac23dad6364 sparse-revlog: drop unused deltainfo parameter from _slicechunktodensity Boris Feld <boris.feld@octobus.net> parents: 40603 diff changeset	365 deltachainspan = segmentspan(revlog, revs)
34880 9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	366
38641 feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	367 if deltachainspan < mingapsize:
38635 d083ae26c325 revlog: early return in _slicechunk when span is already small enough Boris Feld <boris.feld@octobus.net> parents: 38634 diff changeset	368 yield revs
d083ae26c325 revlog: early return in _slicechunk when span is already small enough Boris Feld <boris.feld@octobus.net> parents: 38634 diff changeset	369 return
d083ae26c325 revlog: early return in _slicechunk when span is already small enough Boris Feld <boris.feld@octobus.net> parents: 38634 diff changeset	370
38718 f8762ea73e0d sparse-revlog: implement algorithm to write sparse delta chains (issue5480) Paul Morelle <paul.morelle@octobus.net> parents: 38717 diff changeset	371 readdata = deltachainspan
40606 bfbfd15d65bd sparse-revlog: fast-path before computing payload size Boris Feld <boris.feld@octobus.net> parents: 40605 diff changeset	372 chainpayload = sum(length(r) for r in revs)
34824 e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	373
34880 9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	374 if deltachainspan:
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	375 density = chainpayload / float(deltachainspan)
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	376 else:
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	377 density = 1.0
34824 e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	378
38641 feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	379 if density >= targetdensity:
38634 f0ea8b847831 revlog: early return in _slicechunk when density is already good Paul Morelle <paul.morelle@octobus.net> parents: 38632 diff changeset	380 yield revs
f0ea8b847831 revlog: early return in _slicechunk when density is already good Paul Morelle <paul.morelle@octobus.net> parents: 38632 diff changeset	381 return
f0ea8b847831 revlog: early return in _slicechunk when density is already good Paul Morelle <paul.morelle@octobus.net> parents: 38632 diff changeset	382
34880 9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	383 # Store the gaps in a heap to have them sorted by decreasing size
40607 54de23400b2a sparse-revlog: stop using a heap to track gaps Boris Feld <boris.feld@octobus.net> parents: 40606 diff changeset	384 gaps = []
34880 9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	385 prevend = None
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	386 for i, rev in enumerate(revs):
40604 3ac23dad6364 sparse-revlog: drop unused deltainfo parameter from _slicechunktodensity Boris Feld <boris.feld@octobus.net> parents: 40603 diff changeset	387 revstart = start(rev)
3ac23dad6364 sparse-revlog: drop unused deltainfo parameter from _slicechunktodensity Boris Feld <boris.feld@octobus.net> parents: 40603 diff changeset	388 revlen = length(rev)
34824 e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	389
34898 1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk Paul Morelle <paul.morelle@octobus.net> parents: 34881 diff changeset	390 # Skip empty revisions to form larger holes
1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk Paul Morelle <paul.morelle@octobus.net> parents: 34881 diff changeset	391 if revlen == 0:
1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk Paul Morelle <paul.morelle@octobus.net> parents: 34881 diff changeset	392 continue
1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk Paul Morelle <paul.morelle@octobus.net> parents: 34881 diff changeset	393
34880 9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	394 if prevend is not None:
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	395 gapsize = revstart - prevend
34881 8c9b08a0c48c sparse-read: skip gaps too small to be worth splitting Paul Morelle <paul.morelle@octobus.net> parents: 34880 diff changeset	396 # only consider holes that are large enough
38641 feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	397 if gapsize > mingapsize:
40607 54de23400b2a sparse-revlog: stop using a heap to track gaps Boris Feld <boris.feld@octobus.net> parents: 40606 diff changeset	398 gaps.append((gapsize, i))
34880 9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	399
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	400 prevend = revstart + revlen
40607 54de23400b2a sparse-revlog: stop using a heap to track gaps Boris Feld <boris.feld@octobus.net> parents: 40606 diff changeset	401 # sort the gaps to pop them from largest to small
54de23400b2a sparse-revlog: stop using a heap to track gaps Boris Feld <boris.feld@octobus.net> parents: 40606 diff changeset	402 gaps.sort()
34880 9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	403
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	404 # Collect the indices of the largest holes until the density is acceptable
40608 526ee887c4d5 sparse-revlog: stop using a heap to track selected gap Boris Feld <boris.feld@octobus.net> parents: 40607 diff changeset	405 selected = []
40607 54de23400b2a sparse-revlog: stop using a heap to track gaps Boris Feld <boris.feld@octobus.net> parents: 40606 diff changeset	406 while gaps and density < targetdensity:
54de23400b2a sparse-revlog: stop using a heap to track gaps Boris Feld <boris.feld@octobus.net> parents: 40606 diff changeset	407 gapsize, gapidx = gaps.pop()
34880 9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	408
40608 526ee887c4d5 sparse-revlog: stop using a heap to track selected gap Boris Feld <boris.feld@octobus.net> parents: 40607 diff changeset	409 selected.append(gapidx)
34824 e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	410
34880 9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	411 # the gap sizes are stored as negatives to be sorted decreasingly
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	412 # by the heap
40607 54de23400b2a sparse-revlog: stop using a heap to track gaps Boris Feld <boris.feld@octobus.net> parents: 40606 diff changeset	413 readdata -= gapsize
34880 9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	414 if readdata > 0:
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	415 density = chainpayload / float(readdata)
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	416 else:
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	417 density = 1.0
40608 526ee887c4d5 sparse-revlog: stop using a heap to track selected gap Boris Feld <boris.feld@octobus.net> parents: 40607 diff changeset	418 selected.sort()
34824 e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	419
34880 9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	420 # Cut the revs at collected indices
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	421 previdx = 0
40608 526ee887c4d5 sparse-revlog: stop using a heap to track selected gap Boris Feld <boris.feld@octobus.net> parents: 40607 diff changeset	422 for idx in selected:
34898 1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk Paul Morelle <paul.morelle@octobus.net> parents: 34881 diff changeset	423
1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk Paul Morelle <paul.morelle@octobus.net> parents: 34881 diff changeset	424 chunk = _trimchunk(revlog, revs, previdx, idx)
1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk Paul Morelle <paul.morelle@octobus.net> parents: 34881 diff changeset	425 if chunk:
1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk Paul Morelle <paul.morelle@octobus.net> parents: 34881 diff changeset	426 yield chunk
1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk Paul Morelle <paul.morelle@octobus.net> parents: 34881 diff changeset	427
34880 9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	428 previdx = idx
34898 1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk Paul Morelle <paul.morelle@octobus.net> parents: 34881 diff changeset	429
1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk Paul Morelle <paul.morelle@octobus.net> parents: 34881 diff changeset	430 chunk = _trimchunk(revlog, revs, previdx)
1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk Paul Morelle <paul.morelle@octobus.net> parents: 34881 diff changeset	431 if chunk:
1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk Paul Morelle <paul.morelle@octobus.net> parents: 34881 diff changeset	432 yield chunk
34824 e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	433
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	434
39330 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	435 def _trimchunk(revlog, revs, startidx, endidx=None):
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	436 """returns revs[startidx:endidx] without empty trailing revs
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	437
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	438 Doctest Setup
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	439 >>> revlog = _testrevlog([
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	440 ... 5, #0
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	441 ... 10, #1
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	442 ... 12, #2
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	443 ... 12, #3 (empty)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	444 ... 17, #4
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	445 ... 21, #5
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	446 ... 21, #6 (empty)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	447 ... ])
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	448
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	449 Contiguous cases:
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	450 >>> _trimchunk(revlog, [0, 1, 2, 3, 4, 5, 6], 0)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	451 [0, 1, 2, 3, 4, 5]
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	452 >>> _trimchunk(revlog, [0, 1, 2, 3, 4, 5, 6], 0, 5)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	453 [0, 1, 2, 3, 4]
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	454 >>> _trimchunk(revlog, [0, 1, 2, 3, 4, 5, 6], 0, 4)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	455 [0, 1, 2]
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	456 >>> _trimchunk(revlog, [0, 1, 2, 3, 4, 5, 6], 2, 4)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	457 [2]
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	458 >>> _trimchunk(revlog, [0, 1, 2, 3, 4, 5, 6], 3)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	459 [3, 4, 5]
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	460 >>> _trimchunk(revlog, [0, 1, 2, 3, 4, 5, 6], 3, 5)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	461 [3, 4]
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	462
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	463 Discontiguous cases:
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	464 >>> _trimchunk(revlog, [1, 3, 5, 6], 0)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	465 [1, 3, 5]
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	466 >>> _trimchunk(revlog, [1, 3, 5, 6], 0, 2)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	467 [1]
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	468 >>> _trimchunk(revlog, [1, 3, 5, 6], 1, 3)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	469 [3, 5]
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	470 >>> _trimchunk(revlog, [1, 3, 5, 6], 1)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	471 [3, 5]
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	472 """
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	473 length = revlog.length
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	474
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	475 if endidx is None:
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	476 endidx = len(revs)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	477
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	478 # If we have a non-emtpy delta candidate, there are nothing to trim
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	479 if revs[endidx - 1] < len(revlog):
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	480 # Trim empty revs at the end, except the very first revision of a chain
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	481 while (
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	482 endidx > 1 and endidx > startidx and length(revs[endidx - 1]) == 0
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	483 ):
39330 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	484 endidx -= 1
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	485
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	486 return revs[startidx:endidx]
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	487
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	488
40605 a32ccd32982b sparse-revlog: drop unused deltainfo parameter from segmentspan Boris Feld <boris.feld@octobus.net> parents: 40604 diff changeset	489 def segmentspan(revlog, revs):
39330 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	490 """Get the byte span of a segment of revisions
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	491
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	492 revs is a sorted array of revision numbers
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	493
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	494 >>> revlog = _testrevlog([
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	495 ... 5, #0
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	496 ... 10, #1
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	497 ... 12, #2
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	498 ... 12, #3 (empty)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	499 ... 17, #4
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	500 ... ])
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	501
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	502 >>> segmentspan(revlog, [0, 1, 2, 3, 4])
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	503 17
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	504 >>> segmentspan(revlog, [0, 4])
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	505 17
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	506 >>> segmentspan(revlog, [3, 4])
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	507 5
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	508 >>> segmentspan(revlog, [1, 2, 3,])
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	509 7
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	510 >>> segmentspan(revlog, [1, 3])
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	511 7
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	512 """
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	513 if not revs:
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	514 return 0
40605 a32ccd32982b sparse-revlog: drop unused deltainfo parameter from segmentspan Boris Feld <boris.feld@octobus.net> parents: 40604 diff changeset	515 end = revlog.end(revs[-1])
39330 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	516 return end - revlog.start(revs[0])
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	517
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	518
39331 fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39330 diff changeset	519 def _textfromdelta(fh, revlog, baserev, delta, p1, p2, flags, expectednode):
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39330 diff changeset	520 """build full text from a (base, delta) pair and other metadata"""
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39330 diff changeset	521 # special case deltas which replace entire base; no need to decode
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39330 diff changeset	522 # base revision. this neatly avoids censored bases, which throw when
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39330 diff changeset	523 # they're decoded.
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	524 hlen = struct.calcsize(b">lll")
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	525 if delta[:hlen] == mdiff.replacediffheader(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	526 revlog.rawsize(baserev), len(delta) - hlen
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	527 ):
39331 fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39330 diff changeset	528 fulltext = delta[hlen:]
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39330 diff changeset	529 else:
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39330 diff changeset	530 # deltabase is rawtext before changed by flag processors, which is
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39330 diff changeset	531 # equivalent to non-raw text
48529 c514936d92b4 revlog: remove deprecated APIs Rapha?l Gom?s <rgomes@octobus.net> parents: 47457 diff changeset	532 basetext = revlog.revision(baserev, _df=fh)
39331 fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39330 diff changeset	533 fulltext = mdiff.patch(basetext, delta)
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39330 diff changeset	534
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39330 diff changeset	535 try:
42992 dff95420480f flagprocessors: make `processflagsraw` a module level function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42878 diff changeset	536 validatehash = flagutil.processflagsraw(revlog, fulltext, flags)
39331 fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39330 diff changeset	537 if validatehash:
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39330 diff changeset	538 revlog.checkhash(fulltext, expectednode, p1=p1, p2=p2)
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39330 diff changeset	539 if flags & REVIDX_ISCENSORED:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	540 raise error.StorageError(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	541 _(b'node %s is not censored') % expectednode
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	542 )
39774 4a2466b2a434 revlog: drop some more error aliases (API) Gregory Szorc <gregory.szorc@gmail.com> parents: 39773 diff changeset	543 except error.CensoredNodeError:
39331 fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39330 diff changeset	544 # must pass the censored index flag to add censored revisions
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39330 diff changeset	545 if not flags & REVIDX_ISCENSORED:
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39330 diff changeset	546 raise
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39330 diff changeset	547 return fulltext
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39330 diff changeset	548
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	549
35638 edc9330acac1 revlog: introduce 'deltainfo' to distinguish from 'delta' Paul Morelle <paul.morelle@octobus.net> parents: 35637 diff changeset	550 @attr.s(slots=True, frozen=True)
48946 642e31cb55f0 py3: use class X: instead of class X(object): Gregory Szorc <gregory.szorc@gmail.com> parents: 48875 diff changeset	551 class _deltainfo:
35638 edc9330acac1 revlog: introduce 'deltainfo' to distinguish from 'delta' Paul Morelle <paul.morelle@octobus.net> parents: 35637 diff changeset	552 distance = attr.ib()
edc9330acac1 revlog: introduce 'deltainfo' to distinguish from 'delta' Paul Morelle <paul.morelle@octobus.net> parents: 35637 diff changeset	553 deltalen = attr.ib()
edc9330acac1 revlog: introduce 'deltainfo' to distinguish from 'delta' Paul Morelle <paul.morelle@octobus.net> parents: 35637 diff changeset	554 data = attr.ib()
edc9330acac1 revlog: introduce 'deltainfo' to distinguish from 'delta' Paul Morelle <paul.morelle@octobus.net> parents: 35637 diff changeset	555 base = attr.ib()
edc9330acac1 revlog: introduce 'deltainfo' to distinguish from 'delta' Paul Morelle <paul.morelle@octobus.net> parents: 35637 diff changeset	556 chainbase = attr.ib()
edc9330acac1 revlog: introduce 'deltainfo' to distinguish from 'delta' Paul Morelle <paul.morelle@octobus.net> parents: 35637 diff changeset	557 chainlen = attr.ib()
edc9330acac1 revlog: introduce 'deltainfo' to distinguish from 'delta' Paul Morelle <paul.morelle@octobus.net> parents: 35637 diff changeset	558 compresseddeltalen = attr.ib()
39154 e0da43e2f71f revlog: compute snapshot depth on delta info Boris Feld <boris.feld@octobus.net> parents: 39152 diff changeset	559 snapshotdepth = attr.ib()
35638 edc9330acac1 revlog: introduce 'deltainfo' to distinguish from 'delta' Paul Morelle <paul.morelle@octobus.net> parents: 35637 diff changeset	560
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	561
47253 b876f0bf7366 revlog: introduce a plain compression mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46819 diff changeset	562 def drop_u_compression(delta):
b876f0bf7366 revlog: introduce a plain compression mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46819 diff changeset	563 """turn into a "u" (no-compression) into no-compression without header
b876f0bf7366 revlog: introduce a plain compression mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46819 diff changeset	564
b876f0bf7366 revlog: introduce a plain compression mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46819 diff changeset	565 This is useful for revlog format that has better compression method.
b876f0bf7366 revlog: introduce a plain compression mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46819 diff changeset	566 """
b876f0bf7366 revlog: introduce a plain compression mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46819 diff changeset	567 assert delta.data[0] == b'u', delta.data[0]
b876f0bf7366 revlog: introduce a plain compression mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46819 diff changeset	568 return _deltainfo(
b876f0bf7366 revlog: introduce a plain compression mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46819 diff changeset	569 delta.distance,
b876f0bf7366 revlog: introduce a plain compression mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46819 diff changeset	570 delta.deltalen - 1,
b876f0bf7366 revlog: introduce a plain compression mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46819 diff changeset	571 (b'', delta.data[1]),
b876f0bf7366 revlog: introduce a plain compression mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46819 diff changeset	572 delta.base,
b876f0bf7366 revlog: introduce a plain compression mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46819 diff changeset	573 delta.chainbase,
b876f0bf7366 revlog: introduce a plain compression mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46819 diff changeset	574 delta.chainlen,
b876f0bf7366 revlog: introduce a plain compression mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46819 diff changeset	575 delta.compresseddeltalen,
b876f0bf7366 revlog: introduce a plain compression mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46819 diff changeset	576 delta.snapshotdepth,
b876f0bf7366 revlog: introduce a plain compression mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46819 diff changeset	577 )
b876f0bf7366 revlog: introduce a plain compression mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46819 diff changeset	578
b876f0bf7366 revlog: introduce a plain compression mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46819 diff changeset	579
49674 5af4a0a73e4c find-delta: rename _isgooddeltainfo Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49657 diff changeset	580 def is_good_delta_info(revlog, deltainfo, revinfo):
39330 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	581 """Returns True if the given delta is good. Good means that it is within
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	582 the disk span, disk size, and chain length bounds that we know to be
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	583 performant."""
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	584 if deltainfo is None:
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	585 return False
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	586
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	587 # - 'deltainfo.distance' is the distance from the base revision --
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	588 # bounding it limits the amount of I/O we need to do.
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	589 # - 'deltainfo.compresseddeltalen' is the sum of the total size of
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	590 # deltas we need to apply -- bounding it limits the amount of CPU
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	591 # we consume.
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	592
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	593 textlen = revinfo.textlen
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	594 defaultmax = textlen * 4
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	595 maxdist = revlog._maxdeltachainspan
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	596 if not maxdist:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	597 maxdist = deltainfo.distance # ensure the conditional pass
39330 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	598 maxdist = max(maxdist, defaultmax)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	599
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	600 # Bad delta from read span:
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	601 #
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	602 # If the span of data read is larger than the maximum allowed.
40603 2f7e531ef3e7 sparse-revlog: skip the span check in the sparse-revlog case Boris Feld <boris.feld@octobus.net> parents: 40451 diff changeset	603 #
2f7e531ef3e7 sparse-revlog: skip the span check in the sparse-revlog case Boris Feld <boris.feld@octobus.net> parents: 40451 diff changeset	604 # In the sparse-revlog case, we rely on the associated "sparse reading"
2f7e531ef3e7 sparse-revlog: skip the span check in the sparse-revlog case Boris Feld <boris.feld@octobus.net> parents: 40451 diff changeset	605 # to avoid issue related to the span of data. In theory, it would be
2f7e531ef3e7 sparse-revlog: skip the span check in the sparse-revlog case Boris Feld <boris.feld@octobus.net> parents: 40451 diff changeset	606 # possible to build pathological revlog where delta pattern would lead
2f7e531ef3e7 sparse-revlog: skip the span check in the sparse-revlog case Boris Feld <boris.feld@octobus.net> parents: 40451 diff changeset	607 # to too many reads. However, they do not happen in practice at all. So
2f7e531ef3e7 sparse-revlog: skip the span check in the sparse-revlog case Boris Feld <boris.feld@octobus.net> parents: 40451 diff changeset	608 # we skip the span check entirely.
2f7e531ef3e7 sparse-revlog: skip the span check in the sparse-revlog case Boris Feld <boris.feld@octobus.net> parents: 40451 diff changeset	609 if not revlog._sparserevlog and maxdist < deltainfo.distance:
39330 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	610 return False
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	611
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	612 # Bad delta from new delta size:
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	613 #
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	614 # If the delta size is larger than the target text, storing the
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	615 # delta will be inefficient.
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	616 if textlen < deltainfo.deltalen:
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	617 return False
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	618
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	619 # Bad delta from cumulated payload size:
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	620 #
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	621 # If the sum of delta get larger than K * target text length.
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	622 if textlen * LIMIT_DELTA2TEXT < deltainfo.compresseddeltalen:
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	623 return False
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	624
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	625 # Bad delta from chain length:
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	626 #
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	627 # If the number of delta in the chain gets too high.
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	628 if revlog._maxchainlen and revlog._maxchainlen < deltainfo.chainlen:
39330 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	629 return False
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	630
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	631 # bad delta from intermediate snapshot size limit
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	632 #
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	633 # If an intermediate snapshot size is higher than the limit. The
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	634 # limit exist to prevent endless chain of intermediate delta to be
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	635 # created.
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	636 if (
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	637 deltainfo.snapshotdepth is not None
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	638 and (textlen >> deltainfo.snapshotdepth) < deltainfo.deltalen
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	639 ):
39330 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	640 return False
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	641
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	642 # bad delta if new intermediate snapshot is larger than the previous
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	643 # snapshot
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	644 if (
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	645 deltainfo.snapshotdepth
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	646 and revlog.length(deltainfo.base) < deltainfo.deltalen
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	647 ):
39330 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	648 return False
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	649
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	650 return True
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39329 diff changeset	651
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	652
40978 42f59d3f714d delta: exclude base candidate much smaller than the target Boris Feld <boris.feld@octobus.net> parents: 40957 diff changeset	653 # If a revision's full text is that much bigger than a base candidate full
42f59d3f714d delta: exclude base candidate much smaller than the target Boris Feld <boris.feld@octobus.net> parents: 40957 diff changeset	654 # text's, it is very unlikely that it will produce a valid delta. We no longer
42f59d3f714d delta: exclude base candidate much smaller than the target Boris Feld <boris.feld@octobus.net> parents: 40957 diff changeset	655 # consider these candidates.
41033 b373477948df revlog: limit base to rev size ratio to 500 instead of 50 Boris Feld <boris.feld@octobus.net> parents: 40979 diff changeset	656 LIMIT_BASE2TEXT = 500
40978 42f59d3f714d delta: exclude base candidate much smaller than the target Boris Feld <boris.feld@octobus.net> parents: 40957 diff changeset	657
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	658
49611 2afee2176775 delta-find: expand a function definition and call before extendin it Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49610 diff changeset	659 def _candidategroups(
2afee2176775 delta-find: expand a function definition and call before extendin it Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49610 diff changeset	660 revlog,
2afee2176775 delta-find: expand a function definition and call before extendin it Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49610 diff changeset	661 textlen,
2afee2176775 delta-find: expand a function definition and call before extendin it Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49610 diff changeset	662 p1,
2afee2176775 delta-find: expand a function definition and call before extendin it Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49610 diff changeset	663 p2,
2afee2176775 delta-find: expand a function definition and call before extendin it Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49610 diff changeset	664 cachedelta,
49612 e706bb41fdb3 delta-find: move pre-filtering with other pre-filtering logic Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49611 diff changeset	665 excluded_bases=None,
e706bb41fdb3 delta-find: move pre-filtering with other pre-filtering logic Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49611 diff changeset	666 target_rev=None,
49681 c261a628e525 delta-find: use a single snapshot cache when applying a group to an object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49680 diff changeset	667 snapshot_cache=None,
49611 2afee2176775 delta-find: expand a function definition and call before extendin it Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49610 diff changeset	668 ):
39336 1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39335 diff changeset	669 """Provides group of revision to be tested as delta base
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39335 diff changeset	670
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39335 diff changeset	671 This top level function focus on emitting groups with unique and worthwhile
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39335 diff changeset	672 content. See _raw_candidate_groups for details about the group order.
39334 507f5b1dd7c8 revlogdeltas: extract _getcandidaterevs in a function Boris Feld <boris.feld@octobus.net> parents: 39333 diff changeset	673 """
39336 1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39335 diff changeset	674 # should we try to build a delta?
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39335 diff changeset	675 if not (len(revlog) and revlog._storedeltachains):
39497 5b308a4e6d03 snapshot: use None as a stop value when looking for a good delta Boris Feld <boris.feld@octobus.net> parents: 39496 diff changeset	676 yield None
39336 1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39335 diff changeset	677 return
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39335 diff changeset	678
39337 37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups Boris Feld <boris.feld@octobus.net> parents: 39336 diff changeset	679 deltalength = revlog.length
37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups Boris Feld <boris.feld@octobus.net> parents: 39336 diff changeset	680 deltaparent = revlog.deltaparent
40978 42f59d3f714d delta: exclude base candidate much smaller than the target Boris Feld <boris.feld@octobus.net> parents: 40957 diff changeset	681 sparse = revlog._sparserevlog
39498 04b75f3a3f2a snapshot: add refining logic at the findeltainfo level Boris Feld <boris.feld@octobus.net> parents: 39497 diff changeset	682 good = None
39337 37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups Boris Feld <boris.feld@octobus.net> parents: 39336 diff changeset	683
37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups Boris Feld <boris.feld@octobus.net> parents: 39336 diff changeset	684 deltas_limit = textlen * LIMIT_DELTA2TEXT
49657 f5f113f1b011 delta-find: add a way to control the number of bases tested at the same time Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49615 diff changeset	685 group_chunk_size = revlog._candidate_group_chunk_size
39337 37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups Boris Feld <boris.feld@octobus.net> parents: 39336 diff changeset	686
42057 566daffc607d cleanup: use set literals where possible Martin von Zweigbergk <martinvonz@google.com> parents: 41819 diff changeset	687 tested = {nullrev}
49610 35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	688 candidates = _refinedgroups(
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	689 revlog,
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	690 p1,
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	691 p2,
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	692 cachedelta,
49681 c261a628e525 delta-find: use a single snapshot cache when applying a group to an object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49680 diff changeset	693 snapshot_cache=snapshot_cache,
49610 35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	694 )
39499 51cec7fb672e snapshot: also use None as a stop value for `_refinegroup` Boris Feld <boris.feld@octobus.net> parents: 39498 diff changeset	695 while True:
39500 cc85ebb68ff9 snapshot: turn _refinedgroups into a coroutine Boris Feld <boris.feld@octobus.net> parents: 39499 diff changeset	696 temptative = candidates.send(good)
39499 51cec7fb672e snapshot: also use None as a stop value for `_refinegroup` Boris Feld <boris.feld@octobus.net> parents: 39498 diff changeset	697 if temptative is None:
51cec7fb672e snapshot: also use None as a stop value for `_refinegroup` Boris Feld <boris.feld@octobus.net> parents: 39498 diff changeset	698 break
39337 37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups Boris Feld <boris.feld@octobus.net> parents: 39336 diff changeset	699 group = []
37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups Boris Feld <boris.feld@octobus.net> parents: 39336 diff changeset	700 for rev in temptative:
37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups Boris Feld <boris.feld@octobus.net> parents: 39336 diff changeset	701 # skip over empty delta (no need to include them in a chain)
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	702 while revlog._generaldelta and not (
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	703 rev == nullrev or rev in tested or deltalength(rev)
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	704 ):
39594 bdb41eaa8b59 snapshot: fix line order when skipping over empty deltas Boris Feld <boris.feld@octobus.net> parents: 39505 diff changeset	705 tested.add(rev)
39337 37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups Boris Feld <boris.feld@octobus.net> parents: 39336 diff changeset	706 rev = deltaparent(rev)
40957 f960c51eebf3 delta: filter nullrev out first Boris Feld <boris.feld@octobus.net> parents: 40709 diff changeset	707 # no need to try a delta against nullrev, this will be done as a
f960c51eebf3 delta: filter nullrev out first Boris Feld <boris.feld@octobus.net> parents: 40709 diff changeset	708 # last resort.
f960c51eebf3 delta: filter nullrev out first Boris Feld <boris.feld@octobus.net> parents: 40709 diff changeset	709 if rev == nullrev:
f960c51eebf3 delta: filter nullrev out first Boris Feld <boris.feld@octobus.net> parents: 40709 diff changeset	710 continue
39337 37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups Boris Feld <boris.feld@octobus.net> parents: 39336 diff changeset	711 # filter out revision we tested already
37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups Boris Feld <boris.feld@octobus.net> parents: 39336 diff changeset	712 if rev in tested:
37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups Boris Feld <boris.feld@octobus.net> parents: 39336 diff changeset	713 continue
49612 e706bb41fdb3 delta-find: move pre-filtering with other pre-filtering logic Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49611 diff changeset	714 # an higher authority deamed the base unworthy (e.g. censored)
e706bb41fdb3 delta-find: move pre-filtering with other pre-filtering logic Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49611 diff changeset	715 if excluded_bases is not None and rev in excluded_bases:
49615 4956942c0416 delta-find: adjust the moment when we mark something as "tested" Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49614 diff changeset	716 tested.add(rev)
49612 e706bb41fdb3 delta-find: move pre-filtering with other pre-filtering logic Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49611 diff changeset	717 continue
e706bb41fdb3 delta-find: move pre-filtering with other pre-filtering logic Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49611 diff changeset	718 # We are in some recomputation cases and that rev is too high in
e706bb41fdb3 delta-find: move pre-filtering with other pre-filtering logic Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49611 diff changeset	719 # the revlog
e706bb41fdb3 delta-find: move pre-filtering with other pre-filtering logic Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49611 diff changeset	720 if target_rev is not None and rev >= target_rev:
49615 4956942c0416 delta-find: adjust the moment when we mark something as "tested" Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49614 diff changeset	721 tested.add(rev)
49612 e706bb41fdb3 delta-find: move pre-filtering with other pre-filtering logic Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49611 diff changeset	722 continue
39337 37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups Boris Feld <boris.feld@octobus.net> parents: 39336 diff changeset	723 # filter out delta base that will never produce good delta
37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups Boris Feld <boris.feld@octobus.net> parents: 39336 diff changeset	724 if deltas_limit < revlog.length(rev):
49615 4956942c0416 delta-find: adjust the moment when we mark something as "tested" Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49614 diff changeset	725 tested.add(rev)
39337 37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups Boris Feld <boris.feld@octobus.net> parents: 39336 diff changeset	726 continue
40978 42f59d3f714d delta: exclude base candidate much smaller than the target Boris Feld <boris.feld@octobus.net> parents: 40957 diff changeset	727 if sparse and revlog.rawsize(rev) < (textlen // LIMIT_BASE2TEXT):
49615 4956942c0416 delta-find: adjust the moment when we mark something as "tested" Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49614 diff changeset	728 tested.add(rev)
40978 42f59d3f714d delta: exclude base candidate much smaller than the target Boris Feld <boris.feld@octobus.net> parents: 40957 diff changeset	729 continue
39337 37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups Boris Feld <boris.feld@octobus.net> parents: 39336 diff changeset	730 # no delta for rawtext-changing revs (see "candelta" for why)
37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups Boris Feld <boris.feld@octobus.net> parents: 39336 diff changeset	731 if revlog.flags(rev) & REVIDX_RAWTEXT_CHANGING_FLAGS:
49615 4956942c0416 delta-find: adjust the moment when we mark something as "tested" Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49614 diff changeset	732 tested.add(rev)
39337 37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups Boris Feld <boris.feld@octobus.net> parents: 39336 diff changeset	733 continue
49610 35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	734
40979 ba09db267cb6 delta: ignore base whose chains already don't match expectations Boris Feld <boris.feld@octobus.net> parents: 40978 diff changeset	735 # If we reach here, we are about to build and test a delta.
ba09db267cb6 delta: ignore base whose chains already don't match expectations Boris Feld <boris.feld@octobus.net> parents: 40978 diff changeset	736 # The delta building process will compute the chaininfo in all
ba09db267cb6 delta: ignore base whose chains already don't match expectations Boris Feld <boris.feld@octobus.net> parents: 40978 diff changeset	737 # case, since that computation is cached, it is fine to access it
ba09db267cb6 delta: ignore base whose chains already don't match expectations Boris Feld <boris.feld@octobus.net> parents: 40978 diff changeset	738 # here too.
ba09db267cb6 delta: ignore base whose chains already don't match expectations Boris Feld <boris.feld@octobus.net> parents: 40978 diff changeset	739 chainlen, chainsize = revlog._chaininfo(rev)
ba09db267cb6 delta: ignore base whose chains already don't match expectations Boris Feld <boris.feld@octobus.net> parents: 40978 diff changeset	740 # if chain will be too long, skip base
ba09db267cb6 delta: ignore base whose chains already don't match expectations Boris Feld <boris.feld@octobus.net> parents: 40978 diff changeset	741 if revlog._maxchainlen and chainlen >= revlog._maxchainlen:
49615 4956942c0416 delta-find: adjust the moment when we mark something as "tested" Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49614 diff changeset	742 tested.add(rev)
40979 ba09db267cb6 delta: ignore base whose chains already don't match expectations Boris Feld <boris.feld@octobus.net> parents: 40978 diff changeset	743 continue
ba09db267cb6 delta: ignore base whose chains already don't match expectations Boris Feld <boris.feld@octobus.net> parents: 40978 diff changeset	744 # if chain already have too much data, skip base
ba09db267cb6 delta: ignore base whose chains already don't match expectations Boris Feld <boris.feld@octobus.net> parents: 40978 diff changeset	745 if deltas_limit < chainsize:
49615 4956942c0416 delta-find: adjust the moment when we mark something as "tested" Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49614 diff changeset	746 tested.add(rev)
40979 ba09db267cb6 delta: ignore base whose chains already don't match expectations Boris Feld <boris.feld@octobus.net> parents: 40978 diff changeset	747 continue
42463 a0b26fc8fbba deltas: skip if projected delta size does not match text size constraint Valentin Gatien-Baron <vgatien-baron@janestreet.com>, Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42057 diff changeset	748 if sparse and revlog.upperboundcomp is not None:
a0b26fc8fbba deltas: skip if projected delta size does not match text size constraint Valentin Gatien-Baron <vgatien-baron@janestreet.com>, Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42057 diff changeset	749 maxcomp = revlog.upperboundcomp
a0b26fc8fbba deltas: skip if projected delta size does not match text size constraint Valentin Gatien-Baron <vgatien-baron@janestreet.com>, Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42057 diff changeset	750 basenotsnap = (p1, p2, nullrev)
a0b26fc8fbba deltas: skip if projected delta size does not match text size constraint Valentin Gatien-Baron <vgatien-baron@janestreet.com>, Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42057 diff changeset	751 if rev not in basenotsnap and revlog.issnapshot(rev):
a0b26fc8fbba deltas: skip if projected delta size does not match text size constraint Valentin Gatien-Baron <vgatien-baron@janestreet.com>, Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42057 diff changeset	752 snapshotdepth = revlog.snapshotdepth(rev)
a0b26fc8fbba deltas: skip if projected delta size does not match text size constraint Valentin Gatien-Baron <vgatien-baron@janestreet.com>, Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42057 diff changeset	753 # If text is significantly larger than the base, we can
a0b26fc8fbba deltas: skip if projected delta size does not match text size constraint Valentin Gatien-Baron <vgatien-baron@janestreet.com>, Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42057 diff changeset	754 # expect the resulting delta to be proportional to the size
a0b26fc8fbba deltas: skip if projected delta size does not match text size constraint Valentin Gatien-Baron <vgatien-baron@janestreet.com>, Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42057 diff changeset	755 # difference
a0b26fc8fbba deltas: skip if projected delta size does not match text size constraint Valentin Gatien-Baron <vgatien-baron@janestreet.com>, Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42057 diff changeset	756 revsize = revlog.rawsize(rev)
a0b26fc8fbba deltas: skip if projected delta size does not match text size constraint Valentin Gatien-Baron <vgatien-baron@janestreet.com>, Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42057 diff changeset	757 rawsizedistance = max(textlen - revsize, 0)
a0b26fc8fbba deltas: skip if projected delta size does not match text size constraint Valentin Gatien-Baron <vgatien-baron@janestreet.com>, Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42057 diff changeset	758 # use an estimate of the compression upper bound.
a0b26fc8fbba deltas: skip if projected delta size does not match text size constraint Valentin Gatien-Baron <vgatien-baron@janestreet.com>, Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42057 diff changeset	759 lowestrealisticdeltalen = rawsizedistance // maxcomp
a0b26fc8fbba deltas: skip if projected delta size does not match text size constraint Valentin Gatien-Baron <vgatien-baron@janestreet.com>, Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42057 diff changeset	760
a0b26fc8fbba deltas: skip if projected delta size does not match text size constraint Valentin Gatien-Baron <vgatien-baron@janestreet.com>, Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42057 diff changeset	761 # check the absolute constraint on the delta size
a0b26fc8fbba deltas: skip if projected delta size does not match text size constraint Valentin Gatien-Baron <vgatien-baron@janestreet.com>, Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42057 diff changeset	762 snapshotlimit = textlen >> snapshotdepth
a0b26fc8fbba deltas: skip if projected delta size does not match text size constraint Valentin Gatien-Baron <vgatien-baron@janestreet.com>, Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42057 diff changeset	763 if snapshotlimit < lowestrealisticdeltalen:
a0b26fc8fbba deltas: skip if projected delta size does not match text size constraint Valentin Gatien-Baron <vgatien-baron@janestreet.com>, Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42057 diff changeset	764 # delta lower bound is larger than accepted upper bound
49615 4956942c0416 delta-find: adjust the moment when we mark something as "tested" Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49614 diff changeset	765 tested.add(rev)
42463 a0b26fc8fbba deltas: skip if projected delta size does not match text size constraint Valentin Gatien-Baron <vgatien-baron@janestreet.com>, Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42057 diff changeset	766 continue
a0b26fc8fbba deltas: skip if projected delta size does not match text size constraint Valentin Gatien-Baron <vgatien-baron@janestreet.com>, Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42057 diff changeset	767
42464 66c27df1be84 deltas: skip if projected delta size is bigger than previous snapshot Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42463 diff changeset	768 # check the relative constraint on the delta size
66c27df1be84 deltas: skip if projected delta size is bigger than previous snapshot Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42463 diff changeset	769 revlength = revlog.length(rev)
66c27df1be84 deltas: skip if projected delta size is bigger than previous snapshot Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42463 diff changeset	770 if revlength < lowestrealisticdeltalen:
66c27df1be84 deltas: skip if projected delta size is bigger than previous snapshot Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42463 diff changeset	771 # delta probable lower bound is larger than target base
49615 4956942c0416 delta-find: adjust the moment when we mark something as "tested" Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49614 diff changeset	772 tested.add(rev)
42464 66c27df1be84 deltas: skip if projected delta size is bigger than previous snapshot Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42463 diff changeset	773 continue
66c27df1be84 deltas: skip if projected delta size is bigger than previous snapshot Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42463 diff changeset	774
39337 37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups Boris Feld <boris.feld@octobus.net> parents: 39336 diff changeset	775 group.append(rev)
39336 1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39335 diff changeset	776 if group:
49657 f5f113f1b011 delta-find: add a way to control the number of bases tested at the same time Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49615 diff changeset	777 # When the size of the candidate group is big, it can result in a
f5f113f1b011 delta-find: add a way to control the number of bases tested at the same time Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49615 diff changeset	778 # quite significant performance impact. To reduce this, we can send
f5f113f1b011 delta-find: add a way to control the number of bases tested at the same time Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49615 diff changeset	779 # them in smaller batches until the new batch does not provide any
f5f113f1b011 delta-find: add a way to control the number of bases tested at the same time Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49615 diff changeset	780 # improvements.
f5f113f1b011 delta-find: add a way to control the number of bases tested at the same time Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49615 diff changeset	781 #
f5f113f1b011 delta-find: add a way to control the number of bases tested at the same time Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49615 diff changeset	782 # This might reduce the overall efficiency of the compression in
f5f113f1b011 delta-find: add a way to control the number of bases tested at the same time Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49615 diff changeset	783 # some corner cases, but that should also prevent very pathological
f5f113f1b011 delta-find: add a way to control the number of bases tested at the same time Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49615 diff changeset	784 # cases from being an issue. (eg. 20 000 candidates).
f5f113f1b011 delta-find: add a way to control the number of bases tested at the same time Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49615 diff changeset	785 #
f5f113f1b011 delta-find: add a way to control the number of bases tested at the same time Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49615 diff changeset	786 # XXX note that the ordering of the group becomes important as it
f5f113f1b011 delta-find: add a way to control the number of bases tested at the same time Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49615 diff changeset	787 # now impacts the final result. The current order is unprocessed
f5f113f1b011 delta-find: add a way to control the number of bases tested at the same time Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49615 diff changeset	788 # and can be improved.
f5f113f1b011 delta-find: add a way to control the number of bases tested at the same time Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49615 diff changeset	789 if group_chunk_size == 0:
f5f113f1b011 delta-find: add a way to control the number of bases tested at the same time Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49615 diff changeset	790 tested.update(group)
f5f113f1b011 delta-find: add a way to control the number of bases tested at the same time Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49615 diff changeset	791 good = yield tuple(group)
f5f113f1b011 delta-find: add a way to control the number of bases tested at the same time Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49615 diff changeset	792 else:
f5f113f1b011 delta-find: add a way to control the number of bases tested at the same time Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49615 diff changeset	793 prev_good = good
f5f113f1b011 delta-find: add a way to control the number of bases tested at the same time Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49615 diff changeset	794 for start in range(0, len(group), group_chunk_size):
f5f113f1b011 delta-find: add a way to control the number of bases tested at the same time Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49615 diff changeset	795 sub_group = group[start : start + group_chunk_size]
f5f113f1b011 delta-find: add a way to control the number of bases tested at the same time Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49615 diff changeset	796 tested.update(sub_group)
f5f113f1b011 delta-find: add a way to control the number of bases tested at the same time Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49615 diff changeset	797 good = yield tuple(sub_group)
f5f113f1b011 delta-find: add a way to control the number of bases tested at the same time Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49615 diff changeset	798 if prev_good == good:
f5f113f1b011 delta-find: add a way to control the number of bases tested at the same time Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49615 diff changeset	799 break
f5f113f1b011 delta-find: add a way to control the number of bases tested at the same time Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49615 diff changeset	800
39497 5b308a4e6d03 snapshot: use None as a stop value when looking for a good delta Boris Feld <boris.feld@octobus.net> parents: 39496 diff changeset	801 yield None
39336 1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39335 diff changeset	802
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	803
49681 c261a628e525 delta-find: use a single snapshot cache when applying a group to an object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49680 diff changeset	804 def _refinedgroups(revlog, p1, p2, cachedelta, snapshot_cache=None):
39496 2f9f7889549b snapshot: introduce an intermediate `_refinedgroups` generator Boris Feld <boris.feld@octobus.net> parents: 39495 diff changeset	805 good = None
39501 993d7e2c8b79 snapshot: make sure we'll never refine delta base from a reused source Boris Feld <boris.feld@octobus.net> parents: 39500 diff changeset	806 # First we try to reuse a the delta contained in the bundle.
993d7e2c8b79 snapshot: make sure we'll never refine delta base from a reused source Boris Feld <boris.feld@octobus.net> parents: 39500 diff changeset	807 # (or from the source revlog)
993d7e2c8b79 snapshot: make sure we'll never refine delta base from a reused source Boris Feld <boris.feld@octobus.net> parents: 39500 diff changeset	808 #
993d7e2c8b79 snapshot: make sure we'll never refine delta base from a reused source Boris Feld <boris.feld@octobus.net> parents: 39500 diff changeset	809 # This logic only applies to general delta repositories and can be disabled
993d7e2c8b79 snapshot: make sure we'll never refine delta base from a reused source Boris Feld <boris.feld@octobus.net> parents: 39500 diff changeset	810 # through configuration. Disabling reuse source delta is useful when
993d7e2c8b79 snapshot: make sure we'll never refine delta base from a reused source Boris Feld <boris.feld@octobus.net> parents: 39500 diff changeset	811 # we want to make sure we recomputed "optimal" deltas.
49610 35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	812 debug_info = None
49677 05db41701ece find-delta: pass the cache-delta usage policy alongside the cache-delta Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49675 diff changeset	813 if cachedelta is not None and cachedelta[2] > DELTA_BASE_REUSE_NO:
39501 993d7e2c8b79 snapshot: make sure we'll never refine delta base from a reused source Boris Feld <boris.feld@octobus.net> parents: 39500 diff changeset	814 # Assume what we received from the server is a good choice
993d7e2c8b79 snapshot: make sure we'll never refine delta base from a reused source Boris Feld <boris.feld@octobus.net> parents: 39500 diff changeset	815 # build delta will reuse the cache
49610 35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	816 if debug_info is not None:
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	817 debug_info['cached-delta.tested'] += 1
39501 993d7e2c8b79 snapshot: make sure we'll never refine delta base from a reused source Boris Feld <boris.feld@octobus.net> parents: 39500 diff changeset	818 good = yield (cachedelta[0],)
993d7e2c8b79 snapshot: make sure we'll never refine delta base from a reused source Boris Feld <boris.feld@octobus.net> parents: 39500 diff changeset	819 if good is not None:
49610 35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	820 if debug_info is not None:
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	821 debug_info['cached-delta.accepted'] += 1
39501 993d7e2c8b79 snapshot: make sure we'll never refine delta base from a reused source Boris Feld <boris.feld@octobus.net> parents: 39500 diff changeset	822 yield None
993d7e2c8b79 snapshot: make sure we'll never refine delta base from a reused source Boris Feld <boris.feld@octobus.net> parents: 39500 diff changeset	823 return
49681 c261a628e525 delta-find: use a single snapshot cache when applying a group to an object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49680 diff changeset	824 if snapshot_cache is None:
c261a628e525 delta-find: use a single snapshot cache when applying a group to an object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49680 diff changeset	825 snapshot_cache = SnapshotCache()
49610 35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	826 groups = _rawgroups(
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	827 revlog,
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	828 p1,
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	829 p2,
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	830 cachedelta,
49678 efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	831 snapshot_cache,
49610 35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	832 )
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	833 for candidates in groups:
39496 2f9f7889549b snapshot: introduce an intermediate `_refinedgroups` generator Boris Feld <boris.feld@octobus.net> parents: 39495 diff changeset	834 good = yield candidates
2f9f7889549b snapshot: introduce an intermediate `_refinedgroups` generator Boris Feld <boris.feld@octobus.net> parents: 39495 diff changeset	835 if good is not None:
2f9f7889549b snapshot: introduce an intermediate `_refinedgroups` generator Boris Feld <boris.feld@octobus.net> parents: 39495 diff changeset	836 break
39502 e4d4361d0bcd snapshot: try to refine new snapshot base down the chain Boris Feld <boris.feld@octobus.net> parents: 39501 diff changeset	837
40428 bafa1c4bb7a8 sparse-revlog: only refine delta candidates in the sparse case (issue6006) Boris Feld <boris.feld@octobus.net> parents: 39777 diff changeset	838 # If sparse revlog is enabled, we can try to refine the available deltas
bafa1c4bb7a8 sparse-revlog: only refine delta candidates in the sparse case (issue6006) Boris Feld <boris.feld@octobus.net> parents: 39777 diff changeset	839 if not revlog._sparserevlog:
bafa1c4bb7a8 sparse-revlog: only refine delta candidates in the sparse case (issue6006) Boris Feld <boris.feld@octobus.net> parents: 39777 diff changeset	840 yield None
bafa1c4bb7a8 sparse-revlog: only refine delta candidates in the sparse case (issue6006) Boris Feld <boris.feld@octobus.net> parents: 39777 diff changeset	841 return
bafa1c4bb7a8 sparse-revlog: only refine delta candidates in the sparse case (issue6006) Boris Feld <boris.feld@octobus.net> parents: 39777 diff changeset	842
39502 e4d4361d0bcd snapshot: try to refine new snapshot base down the chain Boris Feld <boris.feld@octobus.net> parents: 39501 diff changeset	843 # if we have a refinable value, try to refine it
e4d4361d0bcd snapshot: try to refine new snapshot base down the chain Boris Feld <boris.feld@octobus.net> parents: 39501 diff changeset	844 if good is not None and good not in (p1, p2) and revlog.issnapshot(good):
e4d4361d0bcd snapshot: try to refine new snapshot base down the chain Boris Feld <boris.feld@octobus.net> parents: 39501 diff changeset	845 # refine snapshot down
e4d4361d0bcd snapshot: try to refine new snapshot base down the chain Boris Feld <boris.feld@octobus.net> parents: 39501 diff changeset	846 previous = None
e4d4361d0bcd snapshot: try to refine new snapshot base down the chain Boris Feld <boris.feld@octobus.net> parents: 39501 diff changeset	847 while previous != good:
e4d4361d0bcd snapshot: try to refine new snapshot base down the chain Boris Feld <boris.feld@octobus.net> parents: 39501 diff changeset	848 previous = good
e4d4361d0bcd snapshot: try to refine new snapshot base down the chain Boris Feld <boris.feld@octobus.net> parents: 39501 diff changeset	849 base = revlog.deltaparent(good)
e4d4361d0bcd snapshot: try to refine new snapshot base down the chain Boris Feld <boris.feld@octobus.net> parents: 39501 diff changeset	850 if base == nullrev:
e4d4361d0bcd snapshot: try to refine new snapshot base down the chain Boris Feld <boris.feld@octobus.net> parents: 39501 diff changeset	851 break
e4d4361d0bcd snapshot: try to refine new snapshot base down the chain Boris Feld <boris.feld@octobus.net> parents: 39501 diff changeset	852 good = yield (base,)
39503 5aef5afa8654 snapshot: refine candidate snapshot base upward Boris Feld <boris.feld@octobus.net> parents: 39502 diff changeset	853 # refine snapshot up
49678 efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	854 if not snapshot_cache.snapshots:
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	855 snapshot_cache.update(revlog, good + 1)
39503 5aef5afa8654 snapshot: refine candidate snapshot base upward Boris Feld <boris.feld@octobus.net> parents: 39502 diff changeset	856 previous = None
5aef5afa8654 snapshot: refine candidate snapshot base upward Boris Feld <boris.feld@octobus.net> parents: 39502 diff changeset	857 while good != previous:
5aef5afa8654 snapshot: refine candidate snapshot base upward Boris Feld <boris.feld@octobus.net> parents: 39502 diff changeset	858 previous = good
49678 efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	859 children = tuple(sorted(c for c in snapshot_cache.snapshots[good]))
39503 5aef5afa8654 snapshot: refine candidate snapshot base upward Boris Feld <boris.feld@octobus.net> parents: 39502 diff changeset	860 good = yield children
5aef5afa8654 snapshot: refine candidate snapshot base upward Boris Feld <boris.feld@octobus.net> parents: 39502 diff changeset	861
49610 35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	862 if debug_info is not None:
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	863 if good is None:
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	864 debug_info['no-solution'] += 1
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	865
39499 51cec7fb672e snapshot: also use None as a stop value for `_refinegroup` Boris Feld <boris.feld@octobus.net> parents: 39498 diff changeset	866 yield None
39496 2f9f7889549b snapshot: introduce an intermediate `_refinedgroups` generator Boris Feld <boris.feld@octobus.net> parents: 39495 diff changeset	867
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	868
49678 efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	869 def _rawgroups(revlog, p1, p2, cachedelta, snapshot_cache=None):
39336 1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39335 diff changeset	870 """Provides group of revision to be tested as delta base
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39335 diff changeset	871
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39335 diff changeset	872 This lower level function focus on emitting delta theorically interresting
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39335 diff changeset	873 without looking it any practical details.
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39335 diff changeset	874
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39335 diff changeset	875 The group order aims at providing fast or small candidates first.
39334 507f5b1dd7c8 revlogdeltas: extract _getcandidaterevs in a function Boris Feld <boris.feld@octobus.net> parents: 39333 diff changeset	876 """
507f5b1dd7c8 revlogdeltas: extract _getcandidaterevs in a function Boris Feld <boris.feld@octobus.net> parents: 39333 diff changeset	877 gdelta = revlog._generaldelta
41447 189e06b2d719 revlog: make sure we never use sparserevlog without general delta (issue6056) Boris Feld <boris.feld@octobus.net> parents: 41109 diff changeset	878 # gate sparse behind general-delta because of issue6056
189e06b2d719 revlog: make sure we never use sparserevlog without general delta (issue6056) Boris Feld <boris.feld@octobus.net> parents: 41109 diff changeset	879 sparse = gdelta and revlog._sparserevlog
39334 507f5b1dd7c8 revlogdeltas: extract _getcandidaterevs in a function Boris Feld <boris.feld@octobus.net> parents: 39333 diff changeset	880 curr = len(revlog)
507f5b1dd7c8 revlogdeltas: extract _getcandidaterevs in a function Boris Feld <boris.feld@octobus.net> parents: 39333 diff changeset	881 prev = curr - 1
39492 a33f394b2bfd snapshot: try intermediate snapshot against parents' base Boris Feld <boris.feld@octobus.net> parents: 39488 diff changeset	882 deltachain = lambda rev: revlog._deltachain(rev)[0]
39334 507f5b1dd7c8 revlogdeltas: extract _getcandidaterevs in a function Boris Feld <boris.feld@octobus.net> parents: 39333 diff changeset	883
39336 1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39335 diff changeset	884 if gdelta:
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39335 diff changeset	885 # exclude already lazy tested base if any
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39335 diff changeset	886 parents = [p for p in (p1, p2) if p != nullrev]
39334 507f5b1dd7c8 revlogdeltas: extract _getcandidaterevs in a function Boris Feld <boris.feld@octobus.net> parents: 39333 diff changeset	887
39336 1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39335 diff changeset	888 if not revlog._deltabothparents and len(parents) == 2:
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39335 diff changeset	889 parents.sort()
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39335 diff changeset	890 # To minimize the chance of having to build a fulltext,
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39335 diff changeset	891 # pick first whichever parent is closest to us (max rev)
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39335 diff changeset	892 yield (parents[1],)
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39335 diff changeset	893 # then the other one (min rev) if the first did not fit
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39335 diff changeset	894 yield (parents[0],)
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39335 diff changeset	895 elif len(parents) > 0:
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39335 diff changeset	896 # Test all parents (1 or 2), and keep the best candidate
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39335 diff changeset	897 yield parents
39334 507f5b1dd7c8 revlogdeltas: extract _getcandidaterevs in a function Boris Feld <boris.feld@octobus.net> parents: 39333 diff changeset	898
39492 a33f394b2bfd snapshot: try intermediate snapshot against parents' base Boris Feld <boris.feld@octobus.net> parents: 39488 diff changeset	899 if sparse and parents:
49678 efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	900 if snapshot_cache is None:
49613 5447c1507c86 delta-find: small documentation update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49612 diff changeset	901 # map: base-rev: [snapshot-revs]
49678 efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	902 snapshot_cache = SnapshotCache()
39492 a33f394b2bfd snapshot: try intermediate snapshot against parents' base Boris Feld <boris.feld@octobus.net> parents: 39488 diff changeset	903 # See if we can use an existing snapshot in the parent chains to use as
a33f394b2bfd snapshot: try intermediate snapshot against parents' base Boris Feld <boris.feld@octobus.net> parents: 39488 diff changeset	904 # a base for a new intermediate-snapshot
39494 e72130f58f5d snapshot: consider all snapshots in the parents' chains Boris Feld <boris.feld@octobus.net> parents: 39493 diff changeset	905 #
e72130f58f5d snapshot: consider all snapshots in the parents' chains Boris Feld <boris.feld@octobus.net> parents: 39493 diff changeset	906 # search for snapshot in parents delta chain
e72130f58f5d snapshot: consider all snapshots in the parents' chains Boris Feld <boris.feld@octobus.net> parents: 39493 diff changeset	907 # map: snapshot-level: snapshot-rev
e72130f58f5d snapshot: consider all snapshots in the parents' chains Boris Feld <boris.feld@octobus.net> parents: 39493 diff changeset	908 parents_snaps = collections.defaultdict(set)
39504 05a165dc4f55 snapshot: extract parent chain computation Boris Feld <boris.feld@octobus.net> parents: 39503 diff changeset	909 candidate_chains = [deltachain(p) for p in parents]
05a165dc4f55 snapshot: extract parent chain computation Boris Feld <boris.feld@octobus.net> parents: 39503 diff changeset	910 for chain in candidate_chains:
05a165dc4f55 snapshot: extract parent chain computation Boris Feld <boris.feld@octobus.net> parents: 39503 diff changeset	911 for idx, s in enumerate(chain):
39494 e72130f58f5d snapshot: consider all snapshots in the parents' chains Boris Feld <boris.feld@octobus.net> parents: 39493 diff changeset	912 if not revlog.issnapshot(s):
e72130f58f5d snapshot: consider all snapshots in the parents' chains Boris Feld <boris.feld@octobus.net> parents: 39493 diff changeset	913 break
e72130f58f5d snapshot: consider all snapshots in the parents' chains Boris Feld <boris.feld@octobus.net> parents: 39493 diff changeset	914 parents_snaps[idx].add(s)
39495 6a53842727c1 snapshot: consider unrelated snapshots at a similar level first Boris Feld <boris.feld@octobus.net> parents: 39494 diff changeset	915 snapfloor = min(parents_snaps[0]) + 1
49678 efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	916 snapshot_cache.update(revlog, snapfloor)
39505 c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39504 diff changeset	917 # search for the highest "unrelated" revision
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39504 diff changeset	918 #
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39504 diff changeset	919 # Adding snapshots used by "unrelated" revision increase the odd we
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39504 diff changeset	920 # reuse an independant, yet better snapshot chain.
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39504 diff changeset	921 #
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39504 diff changeset	922 # XXX instead of building a set of revisions, we could lazily enumerate
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39504 diff changeset	923 # over the chains. That would be more efficient, however we stick to
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39504 diff changeset	924 # simple code for now.
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39504 diff changeset	925 all_revs = set()
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39504 diff changeset	926 for chain in candidate_chains:
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39504 diff changeset	927 all_revs.update(chain)
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39504 diff changeset	928 other = None
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39504 diff changeset	929 for r in revlog.revs(prev, snapfloor):
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39504 diff changeset	930 if r not in all_revs:
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39504 diff changeset	931 other = r
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39504 diff changeset	932 break
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39504 diff changeset	933 if other is not None:
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39504 diff changeset	934 # To avoid unfair competition, we won't use unrelated intermediate
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39504 diff changeset	935 # snapshot that are deeper than the ones from the parent delta
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39504 diff changeset	936 # chain.
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39504 diff changeset	937 max_depth = max(parents_snaps.keys())
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39504 diff changeset	938 chain = deltachain(other)
49614 01ccb45b7393 delta-find: rename a variable for clarity Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49613 diff changeset	939 for depth, s in enumerate(chain):
39505 c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39504 diff changeset	940 if s < snapfloor:
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39504 diff changeset	941 continue
49614 01ccb45b7393 delta-find: rename a variable for clarity Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49613 diff changeset	942 if max_depth < depth:
39505 c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39504 diff changeset	943 break
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39504 diff changeset	944 if not revlog.issnapshot(s):
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39504 diff changeset	945 break
49614 01ccb45b7393 delta-find: rename a variable for clarity Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49613 diff changeset	946 parents_snaps[depth].add(s)
39494 e72130f58f5d snapshot: consider all snapshots in the parents' chains Boris Feld <boris.feld@octobus.net> parents: 39493 diff changeset	947 # Test them as possible intermediate snapshot base
e72130f58f5d snapshot: consider all snapshots in the parents' chains Boris Feld <boris.feld@octobus.net> parents: 39493 diff changeset	948 # We test them from highest to lowest level. High level one are more
e72130f58f5d snapshot: consider all snapshots in the parents' chains Boris Feld <boris.feld@octobus.net> parents: 39493 diff changeset	949 # likely to result in small delta
39495 6a53842727c1 snapshot: consider unrelated snapshots at a similar level first Boris Feld <boris.feld@octobus.net> parents: 39494 diff changeset	950 floor = None
39494 e72130f58f5d snapshot: consider all snapshots in the parents' chains Boris Feld <boris.feld@octobus.net> parents: 39493 diff changeset	951 for idx, snaps in sorted(parents_snaps.items(), reverse=True):
39495 6a53842727c1 snapshot: consider unrelated snapshots at a similar level first Boris Feld <boris.feld@octobus.net> parents: 39494 diff changeset	952 siblings = set()
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first Boris Feld <boris.feld@octobus.net> parents: 39494 diff changeset	953 for s in snaps:
49678 efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	954 siblings.update(snapshot_cache.snapshots[s])
39495 6a53842727c1 snapshot: consider unrelated snapshots at a similar level first Boris Feld <boris.feld@octobus.net> parents: 39494 diff changeset	955 # Before considering making a new intermediate snapshot, we check
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first Boris Feld <boris.feld@octobus.net> parents: 39494 diff changeset	956 # if an existing snapshot, children of base we consider, would be
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first Boris Feld <boris.feld@octobus.net> parents: 39494 diff changeset	957 # suitable.
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first Boris Feld <boris.feld@octobus.net> parents: 39494 diff changeset	958 #
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first Boris Feld <boris.feld@octobus.net> parents: 39494 diff changeset	959 # It give a change to reuse a delta chain "unrelated" to the
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first Boris Feld <boris.feld@octobus.net> parents: 39494 diff changeset	960 # current revision instead of starting our own. Without such
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first Boris Feld <boris.feld@octobus.net> parents: 39494 diff changeset	961 # re-use, topological branches would keep reopening new chains.
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first Boris Feld <boris.feld@octobus.net> parents: 39494 diff changeset	962 # Creating more and more snapshot as the repository grow.
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first Boris Feld <boris.feld@octobus.net> parents: 39494 diff changeset	963
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first Boris Feld <boris.feld@octobus.net> parents: 39494 diff changeset	964 if floor is not None:
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first Boris Feld <boris.feld@octobus.net> parents: 39494 diff changeset	965 # We only do this for siblings created after the one in our
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first Boris Feld <boris.feld@octobus.net> parents: 39494 diff changeset	966 # parent's delta chain. Those created before has less chances
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first Boris Feld <boris.feld@octobus.net> parents: 39494 diff changeset	967 # to be valid base since our ancestors had to create a new
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first Boris Feld <boris.feld@octobus.net> parents: 39494 diff changeset	968 # snapshot.
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first Boris Feld <boris.feld@octobus.net> parents: 39494 diff changeset	969 siblings = [r for r in siblings if floor < r]
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first Boris Feld <boris.feld@octobus.net> parents: 39494 diff changeset	970 yield tuple(sorted(siblings))
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first Boris Feld <boris.feld@octobus.net> parents: 39494 diff changeset	971 # then test the base from our parent's delta chain.
39494 e72130f58f5d snapshot: consider all snapshots in the parents' chains Boris Feld <boris.feld@octobus.net> parents: 39493 diff changeset	972 yield tuple(sorted(snaps))
39495 6a53842727c1 snapshot: consider unrelated snapshots at a similar level first Boris Feld <boris.feld@octobus.net> parents: 39494 diff changeset	973 floor = min(snaps)
39493 3ca144f1c8dd snapshot: search for unrelated but reusable full-snapshot Boris Feld <boris.feld@octobus.net> parents: 39492 diff changeset	974 # No suitable base found in the parent chain, search if any full
3ca144f1c8dd snapshot: search for unrelated but reusable full-snapshot Boris Feld <boris.feld@octobus.net> parents: 39492 diff changeset	975 # snapshots emitted since parent's base would be a suitable base for an
3ca144f1c8dd snapshot: search for unrelated but reusable full-snapshot Boris Feld <boris.feld@octobus.net> parents: 39492 diff changeset	976 # intermediate snapshot.
3ca144f1c8dd snapshot: search for unrelated but reusable full-snapshot Boris Feld <boris.feld@octobus.net> parents: 39492 diff changeset	977 #
3ca144f1c8dd snapshot: search for unrelated but reusable full-snapshot Boris Feld <boris.feld@octobus.net> parents: 39492 diff changeset	978 # It give a chance to reuse a delta chain unrelated to the current
3ca144f1c8dd snapshot: search for unrelated but reusable full-snapshot Boris Feld <boris.feld@octobus.net> parents: 39492 diff changeset	979 # revisions instead of starting our own. Without such re-use,
3ca144f1c8dd snapshot: search for unrelated but reusable full-snapshot Boris Feld <boris.feld@octobus.net> parents: 39492 diff changeset	980 # topological branches would keep reopening new full chains. Creating
3ca144f1c8dd snapshot: search for unrelated but reusable full-snapshot Boris Feld <boris.feld@octobus.net> parents: 39492 diff changeset	981 # more and more snapshot as the repository grow.
49680 40e24d82b513 delta-find: make sure we only use newer full snapshot as candidate Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49679 diff changeset	982 full = [r for r in snapshot_cache.snapshots[nullrev] if snapfloor <= r]
40e24d82b513 delta-find: make sure we only use newer full snapshot as candidate Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49679 diff changeset	983 yield tuple(sorted(full))
39492 a33f394b2bfd snapshot: try intermediate snapshot against parents' base Boris Feld <boris.feld@octobus.net> parents: 39488 diff changeset	984
39505 c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39504 diff changeset	985 if not sparse:
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39504 diff changeset	986 # other approach failed try against prev to hopefully save us a
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39504 diff changeset	987 # fulltext.
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39504 diff changeset	988 yield (prev,)
39334 507f5b1dd7c8 revlogdeltas: extract _getcandidaterevs in a function Boris Feld <boris.feld@octobus.net> parents: 39333 diff changeset	989
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	990
49678 efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	991 class SnapshotCache:
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	992 __slots__ = ('snapshots', '_start_rev', '_end_rev')
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	993
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	994 def __init__(self):
49679 b670eb3dd6c9 delta-find: use sets instead of list in the snapshot cache Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49678 diff changeset	995 self.snapshots = collections.defaultdict(set)
49678 efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	996 self._start_rev = None
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	997 self._end_rev = None
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	998
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	999 def update(self, revlog, start_rev=0):
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1000 """find snapshots from start_rev to tip"""
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1001 nb_revs = len(revlog)
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1002 end_rev = nb_revs - 1
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1003 if start_rev > end_rev:
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1004 return # range is empty
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1005
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1006 if self._start_rev is None:
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1007 assert self._end_rev is None
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1008 self._update(revlog, start_rev, end_rev)
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1009 elif not (self._start_rev <= start_rev and end_rev <= self._end_rev):
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1010 if start_rev < self._start_rev:
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1011 self._update(revlog, start_rev, self._start_rev - 1)
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1012 if self._end_rev < end_rev:
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1013 self._update(revlog, self._end_rev + 1, end_rev)
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1014
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1015 if self._start_rev is None:
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1016 assert self._end_rev is None
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1017 self._end_rev = end_rev
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1018 self._start_rev = start_rev
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1019 else:
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1020 self._start_rev = min(self._start_rev, start_rev)
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1021 self._end_rev = max(self._end_rev, end_rev)
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1022 assert self._start_rev <= self._end_rev, (
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1023 self._start_rev,
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1024 self._end_rev,
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1025 )
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1026
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1027 def _update(self, revlog, start_rev, end_rev):
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1028 """internal method that actually do update content"""
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1029 assert self._start_rev is None or (
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1030 start_rev < self._start_rev or start_rev > self._end_rev
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1031 ), (self._start_rev, self._end_rev, start_rev, end_rev)
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1032 assert self._start_rev is None or (
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1033 end_rev < self._start_rev or end_rev > self._end_rev
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1034 ), (self._start_rev, self._end_rev, start_rev, end_rev)
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1035 cache = self.snapshots
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1036 if util.safehasattr(revlog.index, b'findsnapshots'):
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1037 revlog.index.findsnapshots(cache, start_rev, end_rev)
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1038 else:
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1039 deltaparent = revlog.deltaparent
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1040 issnapshot = revlog.issnapshot
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1041 for rev in revlog.revs(start_rev, end_rev):
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1042 if issnapshot(rev):
49679 b670eb3dd6c9 delta-find: use sets instead of list in the snapshot cache Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49678 diff changeset	1043 cache[deltaparent(rev)].add(rev)
49678 efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1044
efbbc2f9121e delta-find: use a smarter object for snapshot caching Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49677 diff changeset	1045
48946 642e31cb55f0 py3: use class X: instead of class X(object): Gregory Szorc <gregory.szorc@gmail.com> parents: 48875 diff changeset	1046 class deltacomputer:
49610 35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	1047 def __init__(
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	1048 self,
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	1049 revlog,
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	1050 write_debug=None,
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	1051 debug_search=False,
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	1052 debug_info=None,
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	1053 ):
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1054 self.revlog = revlog
49226 e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	1055 self._write_debug = write_debug
49228 b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1056 self._debug_search = debug_search
49610 35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	1057 self._debug_info = debug_info
49681 c261a628e525 delta-find: use a single snapshot cache when applying a group to an object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49680 diff changeset	1058 self._snapshot_cache = SnapshotCache()
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1059
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1060 def buildtext(self, revinfo, fh):
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1061 """Builds a fulltext version of a revision
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1062
47399 34cc102c73f5 revlog: move `revisioninfo` in `revlogutils` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47253 diff changeset	1063 revinfo: revisioninfo instance that contains all needed info
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1064 fh: file handle to either the .i or the .d revlog file,
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1065 depending on whether it is inlined or not
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1066 """
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1067 btext = revinfo.btext
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1068 if btext[0] is not None:
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1069 return btext[0]
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1070
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1071 revlog = self.revlog
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1072 cachedelta = revinfo.cachedelta
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1073 baserev = cachedelta[0]
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1074 delta = cachedelta[1]
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1075
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1076 fulltext = btext[0] = _textfromdelta(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1077 fh,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1078 revlog,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1079 baserev,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1080 delta,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1081 revinfo.p1,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1082 revinfo.p2,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1083 revinfo.flags,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1084 revinfo.node,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1085 )
39331 fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39330 diff changeset	1086 return fulltext
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1087
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1088 def _builddeltadiff(self, base, revinfo, fh):
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1089 revlog = self.revlog
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1090 t = self.buildtext(revinfo, fh)
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1091 if revlog.iscensored(base):
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1092 # deltas based on a censored revision must replace the
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1093 # full content in one patch, so delta works everywhere
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1094 header = mdiff.replacediffheader(revlog.rawsize(base), len(t))
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1095 delta = header + t
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1096 else:
42780 7a89b044eea4 rawdata: update callers in delta utils Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42468 diff changeset	1097 ptext = revlog.rawdata(base, _df=fh)
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1098 delta = mdiff.textdiff(ptext, t)
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1099
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1100 return delta
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1101
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1102 def _builddeltainfo(self, revinfo, base, fh):
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1103 # can we use the cached delta?
42465 6e9ba867a946 delta: move some delta chain related computation earlier in deltainfo Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42464 diff changeset	1104 revlog = self.revlog
49228 b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1105 debug_search = self._write_debug is not None and self._debug_search
42465 6e9ba867a946 delta: move some delta chain related computation earlier in deltainfo Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42464 diff changeset	1106 chainbase = revlog.chainbase(base)
6e9ba867a946 delta: move some delta chain related computation earlier in deltainfo Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42464 diff changeset	1107 if revlog._generaldelta:
6e9ba867a946 delta: move some delta chain related computation earlier in deltainfo Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42464 diff changeset	1108 deltabase = base
6e9ba867a946 delta: move some delta chain related computation earlier in deltainfo Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42464 diff changeset	1109 else:
6e9ba867a946 delta: move some delta chain related computation earlier in deltainfo Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42464 diff changeset	1110 deltabase = chainbase
6e9ba867a946 delta: move some delta chain related computation earlier in deltainfo Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42464 diff changeset	1111 snapshotdepth = None
6e9ba867a946 delta: move some delta chain related computation earlier in deltainfo Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42464 diff changeset	1112 if revlog._sparserevlog and deltabase == nullrev:
6e9ba867a946 delta: move some delta chain related computation earlier in deltainfo Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42464 diff changeset	1113 snapshotdepth = 0
6e9ba867a946 delta: move some delta chain related computation earlier in deltainfo Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42464 diff changeset	1114 elif revlog._sparserevlog and revlog.issnapshot(deltabase):
6e9ba867a946 delta: move some delta chain related computation earlier in deltainfo Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42464 diff changeset	1115 # A delta chain should always be one full snapshot,
6e9ba867a946 delta: move some delta chain related computation earlier in deltainfo Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42464 diff changeset	1116 # zero or more semi-snapshots, and zero or more deltas
6e9ba867a946 delta: move some delta chain related computation earlier in deltainfo Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42464 diff changeset	1117 p1, p2 = revlog.rev(revinfo.p1), revlog.rev(revinfo.p2)
6e9ba867a946 delta: move some delta chain related computation earlier in deltainfo Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42464 diff changeset	1118 if deltabase not in (p1, p2) and revlog.issnapshot(deltabase):
6e9ba867a946 delta: move some delta chain related computation earlier in deltainfo Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42464 diff changeset	1119 snapshotdepth = len(revlog._deltachain(deltabase)[0])
39595 a911932d5003 revlog: reuse cached delta for identical base revision (issue5975) Boris Feld <boris.feld@octobus.net> parents: 39594 diff changeset	1120 delta = None
a911932d5003 revlog: reuse cached delta for identical base revision (issue5975) Boris Feld <boris.feld@octobus.net> parents: 39594 diff changeset	1121 if revinfo.cachedelta:
49675 0fca63953810 find-delta: minor preparatory change Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49674 diff changeset	1122 cachebase = revinfo.cachedelta[0]
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1123 # check if the diff still apply
39595 a911932d5003 revlog: reuse cached delta for identical base revision (issue5975) Boris Feld <boris.feld@octobus.net> parents: 39594 diff changeset	1124 currentbase = cachebase
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1125 while (
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1126 currentbase != nullrev
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1127 and currentbase != base
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1128 and self.revlog.length(currentbase) == 0
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1129 ):
39595 a911932d5003 revlog: reuse cached delta for identical base revision (issue5975) Boris Feld <boris.feld@octobus.net> parents: 39594 diff changeset	1130 currentbase = self.revlog.deltaparent(currentbase)
41819 688fc33e105d storage: introduce a `revlog.reuse-external-delta` config Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41447 diff changeset	1131 if self.revlog._lazydelta and currentbase == base:
39595 a911932d5003 revlog: reuse cached delta for identical base revision (issue5975) Boris Feld <boris.feld@octobus.net> parents: 39594 diff changeset	1132 delta = revinfo.cachedelta[1]
a911932d5003 revlog: reuse cached delta for identical base revision (issue5975) Boris Feld <boris.feld@octobus.net> parents: 39594 diff changeset	1133 if delta is None:
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1134 delta = self._builddeltadiff(base, revinfo, fh)
49228 b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1135 if debug_search:
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1136 msg = b"DBG-DELTAS-SEARCH: uncompressed-delta-size=%d\n"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1137 msg %= len(delta)
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1138 self._write_debug(msg)
42467 c1c1872d25d1 deltas: skip if projected compressed size does not match text size constraint Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42466 diff changeset	1139 # snapshotdept need to be neither None nor 0 level snapshot
c1c1872d25d1 deltas: skip if projected compressed size does not match text size constraint Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42466 diff changeset	1140 if revlog.upperboundcomp is not None and snapshotdepth:
c1c1872d25d1 deltas: skip if projected compressed size does not match text size constraint Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42466 diff changeset	1141 lowestrealisticdeltalen = len(delta) // revlog.upperboundcomp
c1c1872d25d1 deltas: skip if projected compressed size does not match text size constraint Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42466 diff changeset	1142 snapshotlimit = revinfo.textlen >> snapshotdepth
49228 b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1143 if debug_search:
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1144 msg = b"DBG-DELTAS-SEARCH: projected-lower-size=%d\n"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1145 msg %= lowestrealisticdeltalen
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1146 self._write_debug(msg)
42467 c1c1872d25d1 deltas: skip if projected compressed size does not match text size constraint Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42466 diff changeset	1147 if snapshotlimit < lowestrealisticdeltalen:
49228 b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1148 if debug_search:
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1149 msg = b"DBG-DELTAS-SEARCH: DISCARDED (snapshot limit)\n"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1150 self._write_debug(msg)
42467 c1c1872d25d1 deltas: skip if projected compressed size does not match text size constraint Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42466 diff changeset	1151 return None
42468 9b5fbe5ead89 deltas: skip if projected compressed size is bigger than previous snapshot Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42467 diff changeset	1152 if revlog.length(base) < lowestrealisticdeltalen:
49228 b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1153 if debug_search:
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1154 msg = b"DBG-DELTAS-SEARCH: DISCARDED (prev size)\n"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1155 self._write_debug(msg)
42468 9b5fbe5ead89 deltas: skip if projected compressed size is bigger than previous snapshot Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42467 diff changeset	1156 return None
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1157 header, data = revlog.compress(delta)
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1158 deltalen = len(header) + len(data)
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1159 offset = revlog.end(len(revlog) - 1)
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1160 dist = deltalen + offset - revlog.start(chainbase)
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1161 chainlen, compresseddeltalen = revlog._chaininfo(base)
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1162 chainlen += 1
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1163 compresseddeltalen += deltalen
39154 e0da43e2f71f revlog: compute snapshot depth on delta info Boris Feld <boris.feld@octobus.net> parents: 39152 diff changeset	1164
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1165 return _deltainfo(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1166 dist,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1167 deltalen,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1168 (header, data),
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1169 deltabase,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1170 chainbase,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1171 chainlen,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1172 compresseddeltalen,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1173 snapshotdepth,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1174 )
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1175
47456 93f4e183b3f5 deltas: at a `target_rev` parameter to finddeltainfo Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47452 diff changeset	1176 def _fullsnapshotinfo(self, fh, revinfo, curr):
39333 5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39332 diff changeset	1177 rawtext = self.buildtext(revinfo, fh)
5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39332 diff changeset	1178 data = self.revlog.compress(rawtext)
5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39332 diff changeset	1179 compresseddeltalen = deltalen = dist = len(data[1]) + len(data[0])
5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39332 diff changeset	1180 deltabase = chainbase = curr
5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39332 diff changeset	1181 snapshotdepth = 0
5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39332 diff changeset	1182 chainlen = 1
5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39332 diff changeset	1183
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1184 return _deltainfo(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1185 dist,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1186 deltalen,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1187 data,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1188 deltabase,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1189 chainbase,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1190 chainlen,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1191 compresseddeltalen,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1192 snapshotdepth,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1193 )
39333 5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39332 diff changeset	1194
47456 93f4e183b3f5 deltas: at a `target_rev` parameter to finddeltainfo Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47452 diff changeset	1195 def finddeltainfo(self, revinfo, fh, excluded_bases=None, target_rev=None):
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1196 """Find an acceptable delta against a candidate revision
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1197
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1198 revinfo: information about the revision (instance of _revisioninfo)
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1199 fh: file handle to either the .i or the .d revlog file,
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1200 depending on whether it is inlined or not
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1201
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1202 Returns the first acceptable candidate revision, as ordered by
39334 507f5b1dd7c8 revlogdeltas: extract _getcandidaterevs in a function Boris Feld <boris.feld@octobus.net> parents: 39333 diff changeset	1203 _candidategroups
39333 5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39332 diff changeset	1204
5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39332 diff changeset	1205 If no suitable deltabase is found, we return delta info for a full
5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39332 diff changeset	1206 snapshot.
47401 1efe3cdef53a revlog: add a ways to blacklist some revision when searching for a delta Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47399 diff changeset	1207
1efe3cdef53a revlog: add a ways to blacklist some revision when searching for a delta Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47399 diff changeset	1208 `excluded_bases` is an optional set of revision that cannot be used as
1efe3cdef53a revlog: add a ways to blacklist some revision when searching for a delta Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47399 diff changeset	1209 a delta base. Use this to recompute delta suitable in censor or strip
1efe3cdef53a revlog: add a ways to blacklist some revision when searching for a delta Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47399 diff changeset	1210 context.
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1211 """
47456 93f4e183b3f5 deltas: at a `target_rev` parameter to finddeltainfo Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47452 diff changeset	1212 if target_rev is None:
47457 f8330a3fc39f censor: implement censoring for revlogv2 Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47456 diff changeset	1213 target_rev = len(self.revlog)
47456 93f4e183b3f5 deltas: at a `target_rev` parameter to finddeltainfo Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47452 diff changeset	1214
39085 dbb3e9e44fce revlog: do not search for delta for empty content Boris Feld <boris.feld@octobus.net> parents: 39084 diff changeset	1215 if not revinfo.textlen:
47456 93f4e183b3f5 deltas: at a `target_rev` parameter to finddeltainfo Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47452 diff changeset	1216 return self._fullsnapshotinfo(fh, revinfo, target_rev)
39085 dbb3e9e44fce revlog: do not search for delta for empty content Boris Feld <boris.feld@octobus.net> parents: 39084 diff changeset	1217
47401 1efe3cdef53a revlog: add a ways to blacklist some revision when searching for a delta Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47399 diff changeset	1218 if excluded_bases is None:
1efe3cdef53a revlog: add a ways to blacklist some revision when searching for a delta Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47399 diff changeset	1219 excluded_bases = set()
1efe3cdef53a revlog: add a ways to blacklist some revision when searching for a delta Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47399 diff changeset	1220
39332 6f4b8f607a31 revlogdeltas: move special cases around raw revisions in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39331 diff changeset	1221 # no delta for flag processor revision (see "candelta" for why)
6f4b8f607a31 revlogdeltas: move special cases around raw revisions in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39331 diff changeset	1222 # not calling candelta since only one revision needs test, also to
6f4b8f607a31 revlogdeltas: move special cases around raw revisions in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39331 diff changeset	1223 # avoid overhead fetching flags again.
6f4b8f607a31 revlogdeltas: move special cases around raw revisions in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39331 diff changeset	1224 if revinfo.flags & REVIDX_RAWTEXT_CHANGING_FLAGS:
47456 93f4e183b3f5 deltas: at a `target_rev` parameter to finddeltainfo Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47452 diff changeset	1225 return self._fullsnapshotinfo(fh, revinfo, target_rev)
39332 6f4b8f607a31 revlogdeltas: move special cases around raw revisions in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39331 diff changeset	1226
49610 35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	1227 gather_debug = (
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	1228 self._write_debug is not None or self._debug_info is not None
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	1229 )
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	1230 debug_search = self._write_debug is not None and self._debug_search
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	1231
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	1232 if gather_debug:
49226 e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	1233 start = util.timer()
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	1234
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	1235 # count the number of different delta we tried (for debug purpose)
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	1236 dbg_try_count = 0
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	1237 # count the number of "search round" we did. (for debug purpose)
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	1238 dbg_try_rounds = 0
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	1239 dbg_type = b'unknown'
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	1240
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1241 cachedelta = revinfo.cachedelta
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1242 p1 = revinfo.p1
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1243 p2 = revinfo.p2
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1244 revlog = self.revlog
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1245
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1246 deltainfo = None
39335 1441eb38849f revlogdeltas: pass revision number to _candidatesgroups Boris Feld <boris.feld@octobus.net> parents: 39334 diff changeset	1247 p1r, p2r = revlog.rev(p1), revlog.rev(p2)
49226 e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	1248
49610 35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	1249 if gather_debug:
49226 e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	1250 if p1r != nullrev:
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	1251 p1_chain_len = revlog._chaininfo(p1r)[0]
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	1252 else:
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	1253 p1_chain_len = -1
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	1254 if p2r != nullrev:
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	1255 p2_chain_len = revlog._chaininfo(p2r)[0]
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	1256 else:
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	1257 p2_chain_len = -1
49228 b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1258 if debug_search:
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1259 msg = b"DBG-DELTAS-SEARCH: SEARCH rev=%d\n"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1260 msg %= target_rev
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1261 self._write_debug(msg)
49226 e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	1262
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1263 groups = _candidategroups(
49611 2afee2176775 delta-find: expand a function definition and call before extendin it Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49610 diff changeset	1264 self.revlog,
2afee2176775 delta-find: expand a function definition and call before extendin it Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49610 diff changeset	1265 revinfo.textlen,
2afee2176775 delta-find: expand a function definition and call before extendin it Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49610 diff changeset	1266 p1r,
2afee2176775 delta-find: expand a function definition and call before extendin it Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49610 diff changeset	1267 p2r,
2afee2176775 delta-find: expand a function definition and call before extendin it Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49610 diff changeset	1268 cachedelta,
49612 e706bb41fdb3 delta-find: move pre-filtering with other pre-filtering logic Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49611 diff changeset	1269 excluded_bases,
e706bb41fdb3 delta-find: move pre-filtering with other pre-filtering logic Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49611 diff changeset	1270 target_rev,
49681 c261a628e525 delta-find: use a single snapshot cache when applying a group to an object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49680 diff changeset	1271 snapshot_cache=self._snapshot_cache,
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42992 diff changeset	1272 )
39497 5b308a4e6d03 snapshot: use None as a stop value when looking for a good delta Boris Feld <boris.feld@octobus.net> parents: 39496 diff changeset	1273 candidaterevs = next(groups)
5b308a4e6d03 snapshot: use None as a stop value when looking for a good delta Boris Feld <boris.feld@octobus.net> parents: 39496 diff changeset	1274 while candidaterevs is not None:
49226 e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	1275 dbg_try_rounds += 1
49228 b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1276 if debug_search:
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1277 prev = None
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1278 if deltainfo is not None:
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1279 prev = deltainfo.base
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1280
49608 78ba41878f2e delta-find: add debug information about reuse of cached data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49228 diff changeset	1281 if (
78ba41878f2e delta-find: add debug information about reuse of cached data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49228 diff changeset	1282 cachedelta is not None
78ba41878f2e delta-find: add debug information about reuse of cached data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49228 diff changeset	1283 and len(candidaterevs) == 1
78ba41878f2e delta-find: add debug information about reuse of cached data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49228 diff changeset	1284 and cachedelta[0] in candidaterevs
78ba41878f2e delta-find: add debug information about reuse of cached data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49228 diff changeset	1285 ):
78ba41878f2e delta-find: add debug information about reuse of cached data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49228 diff changeset	1286 round_type = b"cached-delta"
78ba41878f2e delta-find: add debug information about reuse of cached data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49228 diff changeset	1287 elif p1 in candidaterevs or p2 in candidaterevs:
49228 b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1288 round_type = b"parents"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1289 elif prev is not None and all(c < prev for c in candidaterevs):
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1290 round_type = b"refine-down"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1291 elif prev is not None and all(c > prev for c in candidaterevs):
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1292 round_type = b"refine-up"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1293 else:
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1294 round_type = b"search-down"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1295 msg = b"DBG-DELTAS-SEARCH: ROUND #%d - %d candidates - %s\n"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1296 msg %= (dbg_try_rounds, len(candidaterevs), round_type)
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1297 self._write_debug(msg)
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1298 nominateddeltas = []
39498 04b75f3a3f2a snapshot: add refining logic at the findeltainfo level Boris Feld <boris.feld@octobus.net> parents: 39497 diff changeset	1299 if deltainfo is not None:
49228 b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1300 if debug_search:
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1301 msg = (
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1302 b"DBG-DELTAS-SEARCH: CONTENDER: rev=%d - length=%d\n"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1303 )
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1304 msg %= (deltainfo.base, deltainfo.deltalen)
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1305 self._write_debug(msg)
39498 04b75f3a3f2a snapshot: add refining logic at the findeltainfo level Boris Feld <boris.feld@octobus.net> parents: 39497 diff changeset	1306 # if we already found a good delta,
04b75f3a3f2a snapshot: add refining logic at the findeltainfo level Boris Feld <boris.feld@octobus.net> parents: 39497 diff changeset	1307 # challenge it against refined candidates
04b75f3a3f2a snapshot: add refining logic at the findeltainfo level Boris Feld <boris.feld@octobus.net> parents: 39497 diff changeset	1308 nominateddeltas.append(deltainfo)
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1309 for candidaterev in candidaterevs:
49228 b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1310 if debug_search:
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1311 msg = b"DBG-DELTAS-SEARCH: CANDIDATE: rev=%d\n"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1312 msg %= candidaterev
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1313 self._write_debug(msg)
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1314 candidate_type = None
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1315 if candidaterev == p1:
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1316 candidate_type = b"p1"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1317 elif candidaterev == p2:
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1318 candidate_type = b"p2"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1319 elif self.revlog.issnapshot(candidaterev):
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1320 candidate_type = b"snapshot-%d"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1321 candidate_type %= self.revlog.snapshotdepth(
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1322 candidaterev
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1323 )
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1324
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1325 if candidate_type is not None:
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1326 msg = b"DBG-DELTAS-SEARCH: type=%s\n"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1327 msg %= candidate_type
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1328 self._write_debug(msg)
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1329 msg = b"DBG-DELTAS-SEARCH: size=%d\n"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1330 msg %= self.revlog.length(candidaterev)
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1331 self._write_debug(msg)
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1332 msg = b"DBG-DELTAS-SEARCH: base=%d\n"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1333 msg %= self.revlog.deltaparent(candidaterev)
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1334 self._write_debug(msg)
49612 e706bb41fdb3 delta-find: move pre-filtering with other pre-filtering logic Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49611 diff changeset	1335
49226 e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	1336 dbg_try_count += 1
49228 b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1337
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1338 if debug_search:
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1339 delta_start = util.timer()
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1340 candidatedelta = self._builddeltainfo(revinfo, candidaterev, fh)
49228 b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1341 if debug_search:
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1342 delta_end = util.timer()
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1343 msg = b"DBG-DELTAS-SEARCH: delta-search-time=%f\n"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1344 msg %= delta_end - delta_start
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1345 self._write_debug(msg)
42466 465f2d0df9ae deltas: accept and skip None return for delta info Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42465 diff changeset	1346 if candidatedelta is not None:
49674 5af4a0a73e4c find-delta: rename _isgooddeltainfo Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49657 diff changeset	1347 if is_good_delta_info(self.revlog, candidatedelta, revinfo):
49228 b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1348 if debug_search:
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1349 msg = b"DBG-DELTAS-SEARCH: DELTA: length=%d (GOOD)\n"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1350 msg %= candidatedelta.deltalen
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1351 self._write_debug(msg)
42466 465f2d0df9ae deltas: accept and skip None return for delta info Valentin Gatien-Baron <vgatien-baron@janestreet.com> parents: 42465 diff changeset	1352 nominateddeltas.append(candidatedelta)
49228 b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1353 elif debug_search:
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1354 msg = b"DBG-DELTAS-SEARCH: DELTA: length=%d (BAD)\n"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1355 msg %= candidatedelta.deltalen
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1356 self._write_debug(msg)
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1357 elif debug_search:
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1358 msg = b"DBG-DELTAS-SEARCH: NO-DELTA\n"
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1359 self._write_debug(msg)
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1360 if nominateddeltas:
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1361 deltainfo = min(nominateddeltas, key=lambda x: x.deltalen)
39498 04b75f3a3f2a snapshot: add refining logic at the findeltainfo level Boris Feld <boris.feld@octobus.net> parents: 39497 diff changeset	1362 if deltainfo is not None:
04b75f3a3f2a snapshot: add refining logic at the findeltainfo level Boris Feld <boris.feld@octobus.net> parents: 39497 diff changeset	1363 candidaterevs = groups.send(deltainfo.base)
04b75f3a3f2a snapshot: add refining logic at the findeltainfo level Boris Feld <boris.feld@octobus.net> parents: 39497 diff changeset	1364 else:
04b75f3a3f2a snapshot: add refining logic at the findeltainfo level Boris Feld <boris.feld@octobus.net> parents: 39497 diff changeset	1365 candidaterevs = next(groups)
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1366
39333 5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39332 diff changeset	1367 if deltainfo is None:
49226 e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	1368 dbg_type = b"full"
47456 93f4e183b3f5 deltas: at a `target_rev` parameter to finddeltainfo Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47452 diff changeset	1369 deltainfo = self._fullsnapshotinfo(fh, revinfo, target_rev)
49228 b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1370 elif deltainfo.snapshotdepth: # pytype: disable=attribute-error
49226 e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	1371 dbg_type = b"snapshot"
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	1372 else:
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	1373 dbg_type = b"delta"
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	1374
49610 35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	1375 if gather_debug:
49226 e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	1376 end = util.timer()
49608 78ba41878f2e delta-find: add debug information about reuse of cached data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49228 diff changeset	1377 used_cached = (
78ba41878f2e delta-find: add debug information about reuse of cached data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49228 diff changeset	1378 cachedelta is not None
78ba41878f2e delta-find: add debug information about reuse of cached data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49228 diff changeset	1379 and dbg_try_rounds == 1
78ba41878f2e delta-find: add debug information about reuse of cached data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49228 diff changeset	1380 and dbg_try_count == 1
78ba41878f2e delta-find: add debug information about reuse of cached data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49228 diff changeset	1381 and deltainfo.base == cachedelta[0]
78ba41878f2e delta-find: add debug information about reuse of cached data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49228 diff changeset	1382 )
49226 e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	1383 dbg = {
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	1384 'duration': end - start,
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	1385 'revision': target_rev,
49610 35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	1386 'delta-base': deltainfo.base, # pytype: disable=attribute-error
49226 e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	1387 'search_round_count': dbg_try_rounds,
49608 78ba41878f2e delta-find: add debug information about reuse of cached data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49228 diff changeset	1388 'using-cached-base': used_cached,
49226 e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	1389 'delta_try_count': dbg_try_count,
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	1390 'type': dbg_type,
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	1391 'p1-chain-len': p1_chain_len,
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	1392 'p2-chain-len': p2_chain_len,
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	1393 }
49228 b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1394 if (
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1395 deltainfo.snapshotdepth # pytype: disable=attribute-error
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1396 is not None
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1397 ):
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1398 dbg[
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1399 'snapshot-depth'
b909dd35d9ab deltas: add a debug-delta-find command to analyse delta search Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49226 diff changeset	1400 ] = deltainfo.snapshotdepth # pytype: disable=attribute-error
49226 e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	1401 else:
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	1402 dbg['snapshot-depth'] = 0
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	1403 target_revlog = b"UNKNOWN"
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	1404 target_type = self.revlog.target[0]
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	1405 target_key = self.revlog.target[1]
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	1406 if target_type == KIND_CHANGELOG:
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	1407 target_revlog = b'CHANGELOG:'
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	1408 elif target_type == KIND_MANIFESTLOG:
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	1409 target_revlog = b'MANIFESTLOG:'
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	1410 if target_key:
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	1411 target_revlog += b'%s:' % target_key
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	1412 elif target_type == KIND_FILELOG:
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	1413 target_revlog = b'FILELOG:'
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	1414 if target_key:
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	1415 target_revlog += b'%s:' % target_key
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	1416 dbg['target-revlog'] = target_revlog
e6b7c6fbeb48 deltas: add code to display information about the result of `finddeltainfo` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48946 diff changeset	1417
49610 35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	1418 if self._debug_info is not None:
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	1419 self._debug_info.append(dbg)
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	1420
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	1421 if self._write_debug is not None:
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	1422 msg = (
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	1423 b"DBG-DELTAS:"
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	1424 b" %-12s"
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	1425 b" rev=%d:"
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	1426 b" delta-base=%d"
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	1427 b" is-cached=%d"
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	1428 b" - search-rounds=%d"
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	1429 b" try-count=%d"
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	1430 b" - delta-type=%-6s"
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	1431 b" snap-depth=%d"
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	1432 b" - p1-chain-length=%d"
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	1433 b" p2-chain-length=%d"
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	1434 b" - duration=%f"
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	1435 b"\n"
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	1436 )
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	1437 msg %= (
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	1438 dbg["target-revlog"],
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	1439 dbg["revision"],
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	1440 dbg["delta-base"],
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	1441 dbg["using-cached-base"],
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	1442 dbg["search_round_count"],
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	1443 dbg["delta_try_count"],
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	1444 dbg["type"],
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	1445 dbg["snapshot-depth"],
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	1446 dbg["p1-chain-len"],
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	1447 dbg["p2-chain-len"],
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	1448 dbg["duration"],
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	1449 )
35d4c2124073 debug: add an option to display statistic about a unbundling operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 49608 diff changeset	1450 self._write_debug(msg)
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	1451 return deltainfo
47452 c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47401 diff changeset	1452
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47401 diff changeset	1453
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47401 diff changeset	1454 def delta_compression(default_compression_header, deltainfo):
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47401 diff changeset	1455 """return (COMPRESSION_MODE, deltainfo)
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47401 diff changeset	1456
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47401 diff changeset	1457 used by revlog v2+ format to dispatch between PLAIN and DEFAULT
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47401 diff changeset	1458 compression.
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47401 diff changeset	1459 """
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47401 diff changeset	1460 h, d = deltainfo.data
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47401 diff changeset	1461 compression_mode = COMP_MODE_INLINE
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47401 diff changeset	1462 if not h and not d:
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47401 diff changeset	1463 # not data to store at all... declare them uncompressed
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47401 diff changeset	1464 compression_mode = COMP_MODE_PLAIN
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47401 diff changeset	1465 elif not h:
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47401 diff changeset	1466 t = d[0:1]
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47401 diff changeset	1467 if t == b'\0':
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47401 diff changeset	1468 compression_mode = COMP_MODE_PLAIN
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47401 diff changeset	1469 elif t == default_compression_header:
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47401 diff changeset	1470 compression_mode = COMP_MODE_DEFAULT
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47401 diff changeset	1471 elif h == b'u':
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47401 diff changeset	1472 # we have a more efficient way to declare uncompressed
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47401 diff changeset	1473 h = b''
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47401 diff changeset	1474 compression_mode = COMP_MODE_PLAIN
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47401 diff changeset	1475 deltainfo = drop_u_compression(deltainfo)
c6844912c327 revlog: factor the logic to determine the delta compression out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47401 diff changeset	1476 return compression_mode, deltainfo

Mercurial > public > mercurial-scm > hg

annotate mercurial/revlogutils/deltas.py @ 49681:c261a628e525