mercurial-scm/hg-stable: mercurial/revlogutils/deltas.py annotate

annotate mercurial/revlogutils/deltas.py @ 40697:fd1d41ccbe38

sparse-revlog: use `span` variable as intended The variable was planned to be used in the while condition but was not used yet.

author	Boris Feld <boris.feld@octobus.net>
date	Thu, 15 Nov 2018 14:55:11 +0100
parents	9c3c697267db
children	2eb48aa0acce

rev	line source
39357 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	1 # revlogdeltas.py - Logic around delta computation for revlog
8226 8b2cd04a6e97 put license and copyright info into comment blocks Martin Geisler <mg@lazybytes.net> parents: 8225 diff changeset	2 #
8b2cd04a6e97 put license and copyright info into comment blocks Martin Geisler <mg@lazybytes.net> parents: 8225 diff changeset	3 # Copyright 2005-2007 Matt Mackall <mpm@selenic.com>
39357 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	4 # Copyright 2018 Octobus <contact@octobus.net>
8226 8b2cd04a6e97 put license and copyright info into comment blocks Martin Geisler <mg@lazybytes.net> parents: 8225 diff changeset	5 #
8b2cd04a6e97 put license and copyright info into comment blocks Martin Geisler <mg@lazybytes.net> parents: 8225 diff changeset	6 # This software may be used and distributed according to the terms of the
10263 25e572394f5c Update license to GPLv2+ Matt Mackall <mpm@selenic.com> parents: 10047 diff changeset	7 # GNU General Public License version 2 or any later version.
39357 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	8 """Helper class to compute deltas stored inside revlogs"""
8227 0a9542703300 turn some comments back into module docstrings Martin Geisler <mg@lazybytes.net> parents: 8226 diff changeset	9
27361 29f50344fa83 revlog: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27251 diff changeset	10 from __future__ import absolute_import
29f50344fa83 revlog: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27251 diff changeset	11
39510 3ca144f1c8dd snapshot: search for unrelated but reusable full-snapshot Boris Feld <boris.feld@octobus.net> parents: 39509 diff changeset	12 import collections
27361 29f50344fa83 revlog: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27251 diff changeset	13 import struct
29f50344fa83 revlog: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27251 diff changeset	14
29f50344fa83 revlog: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27251 diff changeset	15 # import stuff from node for others to import from revlog
39357 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	16 from ..node import (
27361 29f50344fa83 revlog: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27251 diff changeset	17 nullrev,
29f50344fa83 revlog: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27251 diff changeset	18 )
39357 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	19 from ..i18n import _
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	20
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	21 from .constants import (
39356 729082bb9938 revlog: split constants into a new `revlogutils.constants` module Boris Feld <boris.feld@octobus.net> parents: 39260 diff changeset	22 REVIDX_ISCENSORED,
729082bb9938 revlog: split constants into a new `revlogutils.constants` module Boris Feld <boris.feld@octobus.net> parents: 39260 diff changeset	23 REVIDX_RAWTEXT_CHANGING_FLAGS,
729082bb9938 revlog: split constants into a new `revlogutils.constants` module Boris Feld <boris.feld@octobus.net> parents: 39260 diff changeset	24 )
39357 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	25
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	26 from ..thirdparty import (
35638 edc9330acac1 revlog: introduce 'deltainfo' to distinguish from 'delta' Paul Morelle <paul.morelle@octobus.net> parents: 35637 diff changeset	27 attr,
edc9330acac1 revlog: introduce 'deltainfo' to distinguish from 'delta' Paul Morelle <paul.morelle@octobus.net> parents: 35637 diff changeset	28 )
39357 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	29
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	30 from .. import (
27361 29f50344fa83 revlog: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27251 diff changeset	31 error,
29f50344fa83 revlog: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27251 diff changeset	32 mdiff,
29f50344fa83 revlog: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27251 diff changeset	33 )
10913 f2ecc5733c89 revlog: factor out _maxinline global. Greg Ward <greg-hg@gerg.ca> parents: 10404 diff changeset	34
39357 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	35 # maximum <delta-chain-data>/<revision-text-length> ratio
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	36 LIMIT_DELTA2TEXT = 2
1091 d62130f99a73 Move hash function back to revlog from node mpm@selenic.com parents: 1089 diff changeset	37
38637 e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	38 class _testrevlog(object):
e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	39 """minimalist fake revlog to use in doctests"""
e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	40
40684 85b14f0dc334 doctest: add a `issnapshot` method to _testrevlog Boris Feld <boris.feld@octobus.net> parents: 40655 diff changeset	41 def __init__(self, data, density=0.5, mingap=0, snapshot=()):
38637 e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	42 """data is an list of revision payload boundaries"""
e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	43 self._data = data
e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	44 self._srdensitythreshold = density
e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	45 self._srmingapsize = mingap
40684 85b14f0dc334 doctest: add a `issnapshot` method to _testrevlog Boris Feld <boris.feld@octobus.net> parents: 40655 diff changeset	46 self._snapshot = set(snapshot)
38637 e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	47
e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	48 def start(self, rev):
e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	49 if rev == 0:
e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	50 return 0
e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	51 return self._data[rev - 1]
e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	52
e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	53 def end(self, rev):
e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	54 return self._data[rev]
e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	55
e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	56 def length(self, rev):
e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	57 return self.end(rev) - self.start(rev)
e33f784f2a44 revlog: introduce a tiny mock of a revlog class Boris Feld <boris.feld@octobus.net> parents: 38636 diff changeset	58
38718 f8762ea73e0d sparse-revlog: implement algorithm to write sparse delta chains (issue5480) Paul Morelle <paul.morelle@octobus.net> parents: 38717 diff changeset	59 def __len__(self):
f8762ea73e0d sparse-revlog: implement algorithm to write sparse delta chains (issue5480) Paul Morelle <paul.morelle@octobus.net> parents: 38717 diff changeset	60 return len(self._data)
f8762ea73e0d sparse-revlog: implement algorithm to write sparse delta chains (issue5480) Paul Morelle <paul.morelle@octobus.net> parents: 38717 diff changeset	61
40684 85b14f0dc334 doctest: add a `issnapshot` method to _testrevlog Boris Feld <boris.feld@octobus.net> parents: 40655 diff changeset	62 def issnapshot(self, rev):
85b14f0dc334 doctest: add a `issnapshot` method to _testrevlog Boris Feld <boris.feld@octobus.net> parents: 40655 diff changeset	63 return rev in self._snapshot
85b14f0dc334 doctest: add a `issnapshot` method to _testrevlog Boris Feld <boris.feld@octobus.net> parents: 40655 diff changeset	64
40651 3ac23dad6364 sparse-revlog: drop unused deltainfo parameter from _slicechunktodensity Boris Feld <boris.feld@octobus.net> parents: 40650 diff changeset	65 def slicechunk(revlog, revs, targetsize=None):
34824 e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	66 """slice revs to reduce the amount of unrelated data to be read from disk.
e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	67
e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	68 ``revs`` is sliced into groups that should be read in one time.
e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	69 Assume that revs are sorted.
38640 f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	70
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	71 The initial chunk is sliced until the overall density (payload/chunks-span
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	72 ratio) is above `revlog._srdensitythreshold`. No gap smaller than
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	73 `revlog._srmingapsize` is skipped.
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	74
38643 967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size Boris Feld <boris.feld@octobus.net> parents: 38642 diff changeset	75 If `targetsize` is set, no chunk larger than `targetsize` will be yield.
967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size Boris Feld <boris.feld@octobus.net> parents: 38642 diff changeset	76 For consistency with other slicing choice, this limit won't go lower than
967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size Boris Feld <boris.feld@octobus.net> parents: 38642 diff changeset	77 `revlog._srmingapsize`.
967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size Boris Feld <boris.feld@octobus.net> parents: 38642 diff changeset	78
967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size Boris Feld <boris.feld@octobus.net> parents: 38642 diff changeset	79 If individual revisions chunk are larger than this limit, they will still
967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size Boris Feld <boris.feld@octobus.net> parents: 38642 diff changeset	80 be raised individually.
967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size Boris Feld <boris.feld@octobus.net> parents: 38642 diff changeset	81
40685 9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	82 >>> data = [
38640 f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	83 ... 5, #00 (5)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	84 ... 10, #01 (5)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	85 ... 12, #02 (2)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	86 ... 12, #03 (empty)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	87 ... 27, #04 (15)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	88 ... 31, #05 (4)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	89 ... 31, #06 (empty)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	90 ... 42, #07 (11)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	91 ... 47, #08 (5)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	92 ... 47, #09 (empty)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	93 ... 48, #10 (1)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	94 ... 51, #11 (3)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	95 ... 74, #12 (23)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	96 ... 85, #13 (11)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	97 ... 86, #14 (1)
f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	98 ... 91, #15 (5)
40685 9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	99 ... ]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	100 >>> revlog = _testrevlog(data, snapshot=range(16))
38640 f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	101
39357 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	102 >>> list(slicechunk(revlog, list(range(16))))
38640 f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	103 [[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]]
39357 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	104 >>> list(slicechunk(revlog, [0, 15]))
38640 f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	105 [[0], [15]]
39357 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	106 >>> list(slicechunk(revlog, [0, 11, 15]))
38640 f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	107 [[0], [11], [15]]
39357 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	108 >>> list(slicechunk(revlog, [0, 11, 13, 15]))
38640 f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	109 [[0], [11, 13, 15]]
39357 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	110 >>> list(slicechunk(revlog, [1, 2, 3, 5, 8, 10, 11, 14]))
38640 f62b8fb0a484 revlog: document and test _slicechunk Boris Feld <boris.feld@octobus.net> parents: 38639 diff changeset	111 [[1, 2], [5, 8, 10, 11], [14]]
38643 967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size Boris Feld <boris.feld@octobus.net> parents: 38642 diff changeset	112
967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size Boris Feld <boris.feld@octobus.net> parents: 38642 diff changeset	113 Slicing with a maximum chunk size
39357 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	114 >>> list(slicechunk(revlog, [0, 11, 13, 15], targetsize=15))
38643 967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size Boris Feld <boris.feld@octobus.net> parents: 38642 diff changeset	115 [[0], [11], [13], [15]]
39357 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	116 >>> list(slicechunk(revlog, [0, 11, 13, 15], targetsize=20))
38643 967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size Boris Feld <boris.feld@octobus.net> parents: 38642 diff changeset	117 [[0], [11], [13, 15]]
34824 e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	118 """
38643 967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size Boris Feld <boris.feld@octobus.net> parents: 38642 diff changeset	119 if targetsize is not None:
967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size Boris Feld <boris.feld@octobus.net> parents: 38642 diff changeset	120 targetsize = max(targetsize, revlog._srmingapsize)
38718 f8762ea73e0d sparse-revlog: implement algorithm to write sparse delta chains (issue5480) Paul Morelle <paul.morelle@octobus.net> parents: 38717 diff changeset	121 # targetsize should not be specified when evaluating delta candidates:
f8762ea73e0d sparse-revlog: implement algorithm to write sparse delta chains (issue5480) Paul Morelle <paul.morelle@octobus.net> parents: 38717 diff changeset	122 # * targetsize is used to ensure we stay within specification when reading,
38641 feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	123 for chunk in _slicechunktodensity(revlog, revs,
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	124 revlog._srdensitythreshold,
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	125 revlog._srmingapsize):
38643 967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size Boris Feld <boris.feld@octobus.net> parents: 38642 diff changeset	126 for subchunk in _slicechunktosize(revlog, chunk, targetsize):
967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size Boris Feld <boris.feld@octobus.net> parents: 38642 diff changeset	127 yield subchunk
38641 feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	128
38718 f8762ea73e0d sparse-revlog: implement algorithm to write sparse delta chains (issue5480) Paul Morelle <paul.morelle@octobus.net> parents: 38717 diff changeset	129 def _slicechunktosize(revlog, revs, targetsize=None):
38642 e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	130 """slice revs to match the target size
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	131
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	132 This is intended to be used on chunk that density slicing selected by that
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	133 are still too large compared to the read garantee of revlog. This might
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	134 happens when "minimal gap size" interrupted the slicing or when chain are
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	135 built in a way that create large blocks next to each other.
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	136
40685 9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	137 >>> data = [
38642 e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	138 ... 3, #0 (3)
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	139 ... 5, #1 (2)
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	140 ... 6, #2 (1)
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	141 ... 8, #3 (2)
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	142 ... 8, #4 (empty)
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	143 ... 11, #5 (3)
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	144 ... 12, #6 (1)
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	145 ... 13, #7 (1)
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	146 ... 14, #8 (1)
40685 9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	147 ... ]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	148
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	149 == All snapshots cases ==
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	150 >>> revlog = _testrevlog(data, snapshot=range(9))
38642 e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	151
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	152 Cases where chunk is already small enough
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	153 >>> list(_slicechunktosize(revlog, [0], 3))
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	154 [[0]]
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	155 >>> list(_slicechunktosize(revlog, [6, 7], 3))
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	156 [[6, 7]]
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	157 >>> list(_slicechunktosize(revlog, [0], None))
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	158 [[0]]
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	159 >>> list(_slicechunktosize(revlog, [6, 7], None))
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	160 [[6, 7]]
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	161
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	162 cases where we need actual slicing
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	163 >>> list(_slicechunktosize(revlog, [0, 1], 3))
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	164 [[0], [1]]
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	165 >>> list(_slicechunktosize(revlog, [1, 3], 3))
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	166 [[1], [3]]
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	167 >>> list(_slicechunktosize(revlog, [1, 2, 3], 3))
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	168 [[1, 2], [3]]
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	169 >>> list(_slicechunktosize(revlog, [3, 5], 3))
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	170 [[3], [5]]
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	171 >>> list(_slicechunktosize(revlog, [3, 4, 5], 3))
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	172 [[3], [5]]
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	173 >>> list(_slicechunktosize(revlog, [5, 6, 7, 8], 3))
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	174 [[5], [6, 7, 8]]
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	175 >>> list(_slicechunktosize(revlog, [0, 1, 2, 3, 4, 5, 6, 7, 8], 3))
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	176 [[0], [1, 2], [3], [5], [6, 7, 8]]
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	177
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	178 Case with too large individual chunk (must return valid chunk)
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	179 >>> list(_slicechunktosize(revlog, [0, 1], 2))
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	180 [[0], [1]]
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	181 >>> list(_slicechunktosize(revlog, [1, 3], 1))
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	182 [[1], [3]]
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	183 >>> list(_slicechunktosize(revlog, [3, 4, 5], 2))
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	184 [[3], [5]]
40685 9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	185
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	186 == No Snapshot cases ==
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	187 >>> revlog = _testrevlog(data)
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	188
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	189 Cases where chunk is already small enough
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	190 >>> list(_slicechunktosize(revlog, [0], 3))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	191 [[0]]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	192 >>> list(_slicechunktosize(revlog, [6, 7], 3))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	193 [[6, 7]]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	194 >>> list(_slicechunktosize(revlog, [0], None))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	195 [[0]]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	196 >>> list(_slicechunktosize(revlog, [6, 7], None))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	197 [[6, 7]]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	198
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	199 cases where we need actual slicing
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	200 >>> list(_slicechunktosize(revlog, [0, 1], 3))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	201 [[0], [1]]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	202 >>> list(_slicechunktosize(revlog, [1, 3], 3))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	203 [[1], [3]]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	204 >>> list(_slicechunktosize(revlog, [1, 2, 3], 3))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	205 [[1], [2, 3]]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	206 >>> list(_slicechunktosize(revlog, [3, 5], 3))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	207 [[3], [5]]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	208 >>> list(_slicechunktosize(revlog, [3, 4, 5], 3))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	209 [[3], [4, 5]]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	210 >>> list(_slicechunktosize(revlog, [5, 6, 7, 8], 3))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	211 [[5], [6, 7, 8]]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	212 >>> list(_slicechunktosize(revlog, [0, 1, 2, 3, 4, 5, 6, 7, 8], 3))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	213 [[0], [1, 2], [3], [5], [6, 7, 8]]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	214
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	215 Case with too large individual chunk (must return valid chunk)
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	216 >>> list(_slicechunktosize(revlog, [0, 1], 2))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	217 [[0], [1]]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	218 >>> list(_slicechunktosize(revlog, [1, 3], 1))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	219 [[1], [3]]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	220 >>> list(_slicechunktosize(revlog, [3, 4, 5], 2))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	221 [[3], [5]]
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	222
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	223 == mixed case ==
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	224 >>> revlog = _testrevlog(data, snapshot=[0, 1, 2])
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	225 >>> list(_slicechunktosize(revlog, list(range(9)), 5))
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	226 [[0, 1], [2], [3, 4, 5], [6, 7, 8]]
38642 e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	227 """
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	228 assert targetsize is None or 0 <= targetsize
40685 9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	229 startdata = revlog.start(revs[0])
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	230 enddata = revlog.end(revs[-1])
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	231 fullspan = enddata - startdata
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	232 if targetsize is None or fullspan <= targetsize:
38642 e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	233 yield revs
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	234 return
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	235
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	236 startrevidx = 0
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	237 endrevidx = 0
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	238 iterrevs = enumerate(revs)
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	239 next(iterrevs) # skip first rev.
40685 9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	240 # first step: get snapshots out of the way
38642 e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	241 for idx, r in iterrevs:
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	242 span = revlog.end(r) - startdata
40685 9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	243 snapshot = revlog.issnapshot(r)
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	244 if span <= targetsize and snapshot:
38642 e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	245 endrevidx = idx
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	246 else:
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	247 chunk = _trimchunk(revlog, revs, startrevidx, endrevidx + 1)
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	248 if chunk:
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	249 yield chunk
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	250 startrevidx = idx
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	251 startdata = revlog.start(r)
e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	252 endrevidx = idx
40685 9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	253 if not snapshot:
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	254 break
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	255
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	256 # for the others, we use binary slicing to quickly converge toward valid
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	257 # chunks (otherwise, we might end up looking for start/end of many
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	258 # revisions). This logic is not looking for the perfect slicing point, it
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	259 # focuses on quickly converging toward valid chunks.
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	260 nbitem = len(revs)
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	261 while (enddata - startdata) > targetsize:
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	262 endrevidx = nbitem
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	263 if nbitem - startrevidx <= 1:
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	264 break # protect against individual chunk larger than limit
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	265 localenddata = revlog.end(revs[endrevidx - 1])
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	266 span = localenddata - startdata
40697 fd1d41ccbe38 sparse-revlog: use `span` variable as intended Boris Feld <boris.feld@octobus.net> parents: 40685 diff changeset	267 while span > targetsize:
40685 9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	268 if endrevidx - startrevidx <= 1:
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	269 break # protect against individual chunk larger than limit
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	270 endrevidx -= (endrevidx - startrevidx) // 2
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	271 localenddata = revlog.end(revs[endrevidx - 1])
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	272 span = localenddata - startdata
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	273 chunk = _trimchunk(revlog, revs, startrevidx, endrevidx)
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	274 if chunk:
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	275 yield chunk
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	276 startrevidx = endrevidx
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	277 startdata = revlog.start(revs[startrevidx])
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	278
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	279 chunk = _trimchunk(revlog, revs, startrevidx)
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	280 if chunk:
9c3c697267db sparse-revlog: rework the way we enforce chunk size limit Boris Feld <boris.feld@octobus.net> parents: 40684 diff changeset	281 yield chunk
38642 e59e27e52297 revlog: add function to slice chunk down to a given size Boris Feld <boris.feld@octobus.net> parents: 38641 diff changeset	282
40651 3ac23dad6364 sparse-revlog: drop unused deltainfo parameter from _slicechunktodensity Boris Feld <boris.feld@octobus.net> parents: 40650 diff changeset	283 def _slicechunktodensity(revlog, revs, targetdensity=0.5,
38718 f8762ea73e0d sparse-revlog: implement algorithm to write sparse delta chains (issue5480) Paul Morelle <paul.morelle@octobus.net> parents: 38717 diff changeset	284 mingapsize=0):
38641 feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	285 """slice revs to reduce the amount of unrelated data to be read from disk.
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	286
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	287 ``revs`` is sliced into groups that should be read in one time.
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	288 Assume that revs are sorted.
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	289
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	290 The initial chunk is sliced until the overall density (payload/chunks-span
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	291 ratio) is above `targetdensity`. No gap smaller than `mingapsize` is
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	292 skipped.
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	293
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	294 >>> revlog = _testrevlog([
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	295 ... 5, #00 (5)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	296 ... 10, #01 (5)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	297 ... 12, #02 (2)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	298 ... 12, #03 (empty)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	299 ... 27, #04 (15)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	300 ... 31, #05 (4)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	301 ... 31, #06 (empty)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	302 ... 42, #07 (11)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	303 ... 47, #08 (5)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	304 ... 47, #09 (empty)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	305 ... 48, #10 (1)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	306 ... 51, #11 (3)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	307 ... 74, #12 (23)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	308 ... 85, #13 (11)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	309 ... 86, #14 (1)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	310 ... 91, #15 (5)
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	311 ... ])
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	312
38655 cd1c484e31e8 revlog: adjust doctest examples to be portable to Python 3 Augie Fackler <augie@google.com> parents: 38644 diff changeset	313 >>> list(_slicechunktodensity(revlog, list(range(16))))
38641 feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	314 [[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]]
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	315 >>> list(_slicechunktodensity(revlog, [0, 15]))
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	316 [[0], [15]]
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	317 >>> list(_slicechunktodensity(revlog, [0, 11, 15]))
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	318 [[0], [11], [15]]
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	319 >>> list(_slicechunktodensity(revlog, [0, 11, 13, 15]))
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	320 [[0], [11, 13, 15]]
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	321 >>> list(_slicechunktodensity(revlog, [1, 2, 3, 5, 8, 10, 11, 14]))
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	322 [[1, 2], [5, 8, 10, 11], [14]]
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	323 >>> list(_slicechunktodensity(revlog, [1, 2, 3, 5, 8, 10, 11, 14],
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	324 ... mingapsize=20))
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	325 [[1, 2, 3, 5, 8, 10, 11], [14]]
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	326 >>> list(_slicechunktodensity(revlog, [1, 2, 3, 5, 8, 10, 11, 14],
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	327 ... targetdensity=0.95))
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	328 [[1, 2], [5], [8, 10, 11], [14]]
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	329 >>> list(_slicechunktodensity(revlog, [1, 2, 3, 5, 8, 10, 11, 14],
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	330 ... targetdensity=0.95, mingapsize=12))
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	331 [[1, 2], [5, 8, 10, 11], [14]]
feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	332 """
34824 e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	333 start = revlog.start
e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	334 length = revlog.length
e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	335
34880 9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	336 if len(revs) <= 1:
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	337 yield revs
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	338 return
34824 e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	339
40651 3ac23dad6364 sparse-revlog: drop unused deltainfo parameter from _slicechunktodensity Boris Feld <boris.feld@octobus.net> parents: 40650 diff changeset	340 deltachainspan = segmentspan(revlog, revs)
34880 9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	341
38641 feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	342 if deltachainspan < mingapsize:
38635 d083ae26c325 revlog: early return in _slicechunk when span is already small enough Boris Feld <boris.feld@octobus.net> parents: 38634 diff changeset	343 yield revs
d083ae26c325 revlog: early return in _slicechunk when span is already small enough Boris Feld <boris.feld@octobus.net> parents: 38634 diff changeset	344 return
d083ae26c325 revlog: early return in _slicechunk when span is already small enough Boris Feld <boris.feld@octobus.net> parents: 38634 diff changeset	345
38718 f8762ea73e0d sparse-revlog: implement algorithm to write sparse delta chains (issue5480) Paul Morelle <paul.morelle@octobus.net> parents: 38717 diff changeset	346 readdata = deltachainspan
40653 bfbfd15d65bd sparse-revlog: fast-path before computing payload size Boris Feld <boris.feld@octobus.net> parents: 40652 diff changeset	347 chainpayload = sum(length(r) for r in revs)
34824 e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	348
34880 9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	349 if deltachainspan:
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	350 density = chainpayload / float(deltachainspan)
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	351 else:
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	352 density = 1.0
34824 e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	353
38641 feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	354 if density >= targetdensity:
38634 f0ea8b847831 revlog: early return in _slicechunk when density is already good Paul Morelle <paul.morelle@octobus.net> parents: 38632 diff changeset	355 yield revs
f0ea8b847831 revlog: early return in _slicechunk when density is already good Paul Morelle <paul.morelle@octobus.net> parents: 38632 diff changeset	356 return
f0ea8b847831 revlog: early return in _slicechunk when density is already good Paul Morelle <paul.morelle@octobus.net> parents: 38632 diff changeset	357
34880 9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	358 # Store the gaps in a heap to have them sorted by decreasing size
40654 54de23400b2a sparse-revlog: stop using a heap to track gaps Boris Feld <boris.feld@octobus.net> parents: 40653 diff changeset	359 gaps = []
34880 9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	360 prevend = None
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	361 for i, rev in enumerate(revs):
40651 3ac23dad6364 sparse-revlog: drop unused deltainfo parameter from _slicechunktodensity Boris Feld <boris.feld@octobus.net> parents: 40650 diff changeset	362 revstart = start(rev)
3ac23dad6364 sparse-revlog: drop unused deltainfo parameter from _slicechunktodensity Boris Feld <boris.feld@octobus.net> parents: 40650 diff changeset	363 revlen = length(rev)
34824 e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	364
34898 1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk Paul Morelle <paul.morelle@octobus.net> parents: 34881 diff changeset	365 # Skip empty revisions to form larger holes
1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk Paul Morelle <paul.morelle@octobus.net> parents: 34881 diff changeset	366 if revlen == 0:
1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk Paul Morelle <paul.morelle@octobus.net> parents: 34881 diff changeset	367 continue
1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk Paul Morelle <paul.morelle@octobus.net> parents: 34881 diff changeset	368
34880 9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	369 if prevend is not None:
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	370 gapsize = revstart - prevend
34881 8c9b08a0c48c sparse-read: skip gaps too small to be worth splitting Paul Morelle <paul.morelle@octobus.net> parents: 34880 diff changeset	371 # only consider holes that are large enough
38641 feba6be0941b revlog: extract density based slicing into its own function Boris Feld <boris.feld@octobus.net> parents: 38640 diff changeset	372 if gapsize > mingapsize:
40654 54de23400b2a sparse-revlog: stop using a heap to track gaps Boris Feld <boris.feld@octobus.net> parents: 40653 diff changeset	373 gaps.append((gapsize, i))
34880 9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	374
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	375 prevend = revstart + revlen
40654 54de23400b2a sparse-revlog: stop using a heap to track gaps Boris Feld <boris.feld@octobus.net> parents: 40653 diff changeset	376 # sort the gaps to pop them from largest to small
54de23400b2a sparse-revlog: stop using a heap to track gaps Boris Feld <boris.feld@octobus.net> parents: 40653 diff changeset	377 gaps.sort()
34880 9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	378
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	379 # Collect the indices of the largest holes until the density is acceptable
40655 526ee887c4d5 sparse-revlog: stop using a heap to track selected gap Boris Feld <boris.feld@octobus.net> parents: 40654 diff changeset	380 selected = []
40654 54de23400b2a sparse-revlog: stop using a heap to track gaps Boris Feld <boris.feld@octobus.net> parents: 40653 diff changeset	381 while gaps and density < targetdensity:
54de23400b2a sparse-revlog: stop using a heap to track gaps Boris Feld <boris.feld@octobus.net> parents: 40653 diff changeset	382 gapsize, gapidx = gaps.pop()
34880 9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	383
40655 526ee887c4d5 sparse-revlog: stop using a heap to track selected gap Boris Feld <boris.feld@octobus.net> parents: 40654 diff changeset	384 selected.append(gapidx)
34824 e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	385
34880 9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	386 # the gap sizes are stored as negatives to be sorted decreasingly
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	387 # by the heap
40654 54de23400b2a sparse-revlog: stop using a heap to track gaps Boris Feld <boris.feld@octobus.net> parents: 40653 diff changeset	388 readdata -= gapsize
34880 9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	389 if readdata > 0:
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	390 density = chainpayload / float(readdata)
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	391 else:
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	392 density = 1.0
40655 526ee887c4d5 sparse-revlog: stop using a heap to track selected gap Boris Feld <boris.feld@octobus.net> parents: 40654 diff changeset	393 selected.sort()
34824 e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	394
34880 9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	395 # Cut the revs at collected indices
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	396 previdx = 0
40655 526ee887c4d5 sparse-revlog: stop using a heap to track selected gap Boris Feld <boris.feld@octobus.net> parents: 40654 diff changeset	397 for idx in selected:
34898 1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk Paul Morelle <paul.morelle@octobus.net> parents: 34881 diff changeset	398
1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk Paul Morelle <paul.morelle@octobus.net> parents: 34881 diff changeset	399 chunk = _trimchunk(revlog, revs, previdx, idx)
1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk Paul Morelle <paul.morelle@octobus.net> parents: 34881 diff changeset	400 if chunk:
1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk Paul Morelle <paul.morelle@octobus.net> parents: 34881 diff changeset	401 yield chunk
1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk Paul Morelle <paul.morelle@octobus.net> parents: 34881 diff changeset	402
34880 9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one Boris Feld <boris.feld@octobus.net> parents: 34825 diff changeset	403 previdx = idx
34898 1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk Paul Morelle <paul.morelle@octobus.net> parents: 34881 diff changeset	404
1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk Paul Morelle <paul.morelle@octobus.net> parents: 34881 diff changeset	405 chunk = _trimchunk(revlog, revs, previdx)
1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk Paul Morelle <paul.morelle@octobus.net> parents: 34881 diff changeset	406 if chunk:
1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk Paul Morelle <paul.morelle@octobus.net> parents: 34881 diff changeset	407 yield chunk
34824 e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse Paul Morelle <paul.morelle@octobus.net> parents: 34823 diff changeset	408
39357 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	409 def _trimchunk(revlog, revs, startidx, endidx=None):
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	410 """returns revs[startidx:endidx] without empty trailing revs
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	411
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	412 Doctest Setup
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	413 >>> revlog = _testrevlog([
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	414 ... 5, #0
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	415 ... 10, #1
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	416 ... 12, #2
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	417 ... 12, #3 (empty)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	418 ... 17, #4
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	419 ... 21, #5
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	420 ... 21, #6 (empty)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	421 ... ])
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	422
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	423 Contiguous cases:
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	424 >>> _trimchunk(revlog, [0, 1, 2, 3, 4, 5, 6], 0)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	425 [0, 1, 2, 3, 4, 5]
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	426 >>> _trimchunk(revlog, [0, 1, 2, 3, 4, 5, 6], 0, 5)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	427 [0, 1, 2, 3, 4]
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	428 >>> _trimchunk(revlog, [0, 1, 2, 3, 4, 5, 6], 0, 4)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	429 [0, 1, 2]
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	430 >>> _trimchunk(revlog, [0, 1, 2, 3, 4, 5, 6], 2, 4)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	431 [2]
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	432 >>> _trimchunk(revlog, [0, 1, 2, 3, 4, 5, 6], 3)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	433 [3, 4, 5]
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	434 >>> _trimchunk(revlog, [0, 1, 2, 3, 4, 5, 6], 3, 5)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	435 [3, 4]
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	436
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	437 Discontiguous cases:
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	438 >>> _trimchunk(revlog, [1, 3, 5, 6], 0)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	439 [1, 3, 5]
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	440 >>> _trimchunk(revlog, [1, 3, 5, 6], 0, 2)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	441 [1]
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	442 >>> _trimchunk(revlog, [1, 3, 5, 6], 1, 3)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	443 [3, 5]
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	444 >>> _trimchunk(revlog, [1, 3, 5, 6], 1)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	445 [3, 5]
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	446 """
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	447 length = revlog.length
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	448
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	449 if endidx is None:
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	450 endidx = len(revs)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	451
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	452 # If we have a non-emtpy delta candidate, there are nothing to trim
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	453 if revs[endidx - 1] < len(revlog):
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	454 # Trim empty revs at the end, except the very first revision of a chain
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	455 while (endidx > 1
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	456 and endidx > startidx
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	457 and length(revs[endidx - 1]) == 0):
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	458 endidx -= 1
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	459
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	460 return revs[startidx:endidx]
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	461
40652 a32ccd32982b sparse-revlog: drop unused deltainfo parameter from segmentspan Boris Feld <boris.feld@octobus.net> parents: 40651 diff changeset	462 def segmentspan(revlog, revs):
39357 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	463 """Get the byte span of a segment of revisions
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	464
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	465 revs is a sorted array of revision numbers
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	466
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	467 >>> revlog = _testrevlog([
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	468 ... 5, #0
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	469 ... 10, #1
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	470 ... 12, #2
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	471 ... 12, #3 (empty)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	472 ... 17, #4
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	473 ... ])
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	474
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	475 >>> segmentspan(revlog, [0, 1, 2, 3, 4])
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	476 17
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	477 >>> segmentspan(revlog, [0, 4])
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	478 17
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	479 >>> segmentspan(revlog, [3, 4])
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	480 5
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	481 >>> segmentspan(revlog, [1, 2, 3,])
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	482 7
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	483 >>> segmentspan(revlog, [1, 3])
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	484 7
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	485 """
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	486 if not revs:
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	487 return 0
40652 a32ccd32982b sparse-revlog: drop unused deltainfo parameter from segmentspan Boris Feld <boris.feld@octobus.net> parents: 40651 diff changeset	488 end = revlog.end(revs[-1])
39357 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	489 return end - revlog.start(revs[0])
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	490
39358 fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39357 diff changeset	491 def _textfromdelta(fh, revlog, baserev, delta, p1, p2, flags, expectednode):
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39357 diff changeset	492 """build full text from a (base, delta) pair and other metadata"""
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39357 diff changeset	493 # special case deltas which replace entire base; no need to decode
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39357 diff changeset	494 # base revision. this neatly avoids censored bases, which throw when
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39357 diff changeset	495 # they're decoded.
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39357 diff changeset	496 hlen = struct.calcsize(">lll")
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39357 diff changeset	497 if delta[:hlen] == mdiff.replacediffheader(revlog.rawsize(baserev),
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39357 diff changeset	498 len(delta) - hlen):
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39357 diff changeset	499 fulltext = delta[hlen:]
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39357 diff changeset	500 else:
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39357 diff changeset	501 # deltabase is rawtext before changed by flag processors, which is
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39357 diff changeset	502 # equivalent to non-raw text
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39357 diff changeset	503 basetext = revlog.revision(baserev, _df=fh, raw=False)
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39357 diff changeset	504 fulltext = mdiff.patch(basetext, delta)
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39357 diff changeset	505
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39357 diff changeset	506 try:
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39357 diff changeset	507 res = revlog._processflags(fulltext, flags, 'read', raw=True)
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39357 diff changeset	508 fulltext, validatehash = res
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39357 diff changeset	509 if validatehash:
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39357 diff changeset	510 revlog.checkhash(fulltext, expectednode, p1=p1, p2=p2)
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39357 diff changeset	511 if flags & REVIDX_ISCENSORED:
39793 b63dee7bd0d9 global: replace most uses of RevlogError with StorageError (API) Gregory Szorc <gregory.szorc@gmail.com> parents: 39790 diff changeset	512 raise error.StorageError(_('node %s is not censored') %
b63dee7bd0d9 global: replace most uses of RevlogError with StorageError (API) Gregory Szorc <gregory.szorc@gmail.com> parents: 39790 diff changeset	513 expectednode)
39790 4a2466b2a434 revlog: drop some more error aliases (API) Gregory Szorc <gregory.szorc@gmail.com> parents: 39789 diff changeset	514 except error.CensoredNodeError:
39358 fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39357 diff changeset	515 # must pass the censored index flag to add censored revisions
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39357 diff changeset	516 if not flags & REVIDX_ISCENSORED:
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39357 diff changeset	517 raise
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39357 diff changeset	518 return fulltext
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39357 diff changeset	519
35638 edc9330acac1 revlog: introduce 'deltainfo' to distinguish from 'delta' Paul Morelle <paul.morelle@octobus.net> parents: 35637 diff changeset	520 @attr.s(slots=True, frozen=True)
edc9330acac1 revlog: introduce 'deltainfo' to distinguish from 'delta' Paul Morelle <paul.morelle@octobus.net> parents: 35637 diff changeset	521 class _deltainfo(object):
edc9330acac1 revlog: introduce 'deltainfo' to distinguish from 'delta' Paul Morelle <paul.morelle@octobus.net> parents: 35637 diff changeset	522 distance = attr.ib()
edc9330acac1 revlog: introduce 'deltainfo' to distinguish from 'delta' Paul Morelle <paul.morelle@octobus.net> parents: 35637 diff changeset	523 deltalen = attr.ib()
edc9330acac1 revlog: introduce 'deltainfo' to distinguish from 'delta' Paul Morelle <paul.morelle@octobus.net> parents: 35637 diff changeset	524 data = attr.ib()
edc9330acac1 revlog: introduce 'deltainfo' to distinguish from 'delta' Paul Morelle <paul.morelle@octobus.net> parents: 35637 diff changeset	525 base = attr.ib()
edc9330acac1 revlog: introduce 'deltainfo' to distinguish from 'delta' Paul Morelle <paul.morelle@octobus.net> parents: 35637 diff changeset	526 chainbase = attr.ib()
edc9330acac1 revlog: introduce 'deltainfo' to distinguish from 'delta' Paul Morelle <paul.morelle@octobus.net> parents: 35637 diff changeset	527 chainlen = attr.ib()
edc9330acac1 revlog: introduce 'deltainfo' to distinguish from 'delta' Paul Morelle <paul.morelle@octobus.net> parents: 35637 diff changeset	528 compresseddeltalen = attr.ib()
39187 e0da43e2f71f revlog: compute snapshot depth on delta info Boris Feld <boris.feld@octobus.net> parents: 39185 diff changeset	529 snapshotdepth = attr.ib()
35638 edc9330acac1 revlog: introduce 'deltainfo' to distinguish from 'delta' Paul Morelle <paul.morelle@octobus.net> parents: 35637 diff changeset	530
39357 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	531 def isgooddeltainfo(revlog, deltainfo, revinfo):
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	532 """Returns True if the given delta is good. Good means that it is within
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	533 the disk span, disk size, and chain length bounds that we know to be
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	534 performant."""
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	535 if deltainfo is None:
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	536 return False
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	537
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	538 # - 'deltainfo.distance' is the distance from the base revision --
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	539 # bounding it limits the amount of I/O we need to do.
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	540 # - 'deltainfo.compresseddeltalen' is the sum of the total size of
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	541 # deltas we need to apply -- bounding it limits the amount of CPU
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	542 # we consume.
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	543
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	544 textlen = revinfo.textlen
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	545 defaultmax = textlen * 4
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	546 maxdist = revlog._maxdeltachainspan
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	547 if not maxdist:
40650 2f7e531ef3e7 sparse-revlog: skip the span check in the sparse-revlog case Boris Feld <boris.feld@octobus.net> parents: 40425 diff changeset	548 maxdist = deltainfo.distance # ensure the conditional pass
39357 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	549 maxdist = max(maxdist, defaultmax)
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	550
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	551 # Bad delta from read span:
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	552 #
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	553 # If the span of data read is larger than the maximum allowed.
40650 2f7e531ef3e7 sparse-revlog: skip the span check in the sparse-revlog case Boris Feld <boris.feld@octobus.net> parents: 40425 diff changeset	554 #
2f7e531ef3e7 sparse-revlog: skip the span check in the sparse-revlog case Boris Feld <boris.feld@octobus.net> parents: 40425 diff changeset	555 # In the sparse-revlog case, we rely on the associated "sparse reading"
2f7e531ef3e7 sparse-revlog: skip the span check in the sparse-revlog case Boris Feld <boris.feld@octobus.net> parents: 40425 diff changeset	556 # to avoid issue related to the span of data. In theory, it would be
2f7e531ef3e7 sparse-revlog: skip the span check in the sparse-revlog case Boris Feld <boris.feld@octobus.net> parents: 40425 diff changeset	557 # possible to build pathological revlog where delta pattern would lead
2f7e531ef3e7 sparse-revlog: skip the span check in the sparse-revlog case Boris Feld <boris.feld@octobus.net> parents: 40425 diff changeset	558 # to too many reads. However, they do not happen in practice at all. So
2f7e531ef3e7 sparse-revlog: skip the span check in the sparse-revlog case Boris Feld <boris.feld@octobus.net> parents: 40425 diff changeset	559 # we skip the span check entirely.
2f7e531ef3e7 sparse-revlog: skip the span check in the sparse-revlog case Boris Feld <boris.feld@octobus.net> parents: 40425 diff changeset	560 if not revlog._sparserevlog and maxdist < deltainfo.distance:
39357 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	561 return False
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	562
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	563 # Bad delta from new delta size:
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	564 #
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	565 # If the delta size is larger than the target text, storing the
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	566 # delta will be inefficient.
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	567 if textlen < deltainfo.deltalen:
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	568 return False
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	569
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	570 # Bad delta from cumulated payload size:
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	571 #
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	572 # If the sum of delta get larger than K * target text length.
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	573 if textlen * LIMIT_DELTA2TEXT < deltainfo.compresseddeltalen:
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	574 return False
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	575
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	576 # Bad delta from chain length:
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	577 #
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	578 # If the number of delta in the chain gets too high.
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	579 if (revlog._maxchainlen
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	580 and revlog._maxchainlen < deltainfo.chainlen):
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	581 return False
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	582
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	583 # bad delta from intermediate snapshot size limit
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	584 #
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	585 # If an intermediate snapshot size is higher than the limit. The
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	586 # limit exist to prevent endless chain of intermediate delta to be
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	587 # created.
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	588 if (deltainfo.snapshotdepth is not None and
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	589 (textlen >> deltainfo.snapshotdepth) < deltainfo.deltalen):
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	590 return False
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	591
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	592 # bad delta if new intermediate snapshot is larger than the previous
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	593 # snapshot
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	594 if (deltainfo.snapshotdepth
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	595 and revlog.length(deltainfo.base) < deltainfo.deltalen):
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	596 return False
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	597
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	598 return True
655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	599
39364 37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups Boris Feld <boris.feld@octobus.net> parents: 39363 diff changeset	600 def _candidategroups(revlog, textlen, p1, p2, cachedelta):
39363 1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39362 diff changeset	601 """Provides group of revision to be tested as delta base
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39362 diff changeset	602
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39362 diff changeset	603 This top level function focus on emitting groups with unique and worthwhile
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39362 diff changeset	604 content. See _raw_candidate_groups for details about the group order.
39361 507f5b1dd7c8 revlogdeltas: extract _getcandidaterevs in a function Boris Feld <boris.feld@octobus.net> parents: 39360 diff changeset	605 """
39363 1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39362 diff changeset	606 # should we try to build a delta?
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39362 diff changeset	607 if not (len(revlog) and revlog._storedeltachains):
39514 5b308a4e6d03 snapshot: use None as a stop value when looking for a good delta Boris Feld <boris.feld@octobus.net> parents: 39513 diff changeset	608 yield None
39363 1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39362 diff changeset	609 return
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39362 diff changeset	610
39364 37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups Boris Feld <boris.feld@octobus.net> parents: 39363 diff changeset	611 deltalength = revlog.length
37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups Boris Feld <boris.feld@octobus.net> parents: 39363 diff changeset	612 deltaparent = revlog.deltaparent
39515 04b75f3a3f2a snapshot: add refining logic at the findeltainfo level Boris Feld <boris.feld@octobus.net> parents: 39514 diff changeset	613 good = None
39364 37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups Boris Feld <boris.feld@octobus.net> parents: 39363 diff changeset	614
37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups Boris Feld <boris.feld@octobus.net> parents: 39363 diff changeset	615 deltas_limit = textlen * LIMIT_DELTA2TEXT
37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups Boris Feld <boris.feld@octobus.net> parents: 39363 diff changeset	616
39363 1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39362 diff changeset	617 tested = set([nullrev])
39516 51cec7fb672e snapshot: also use None as a stop value for `_refinegroup` Boris Feld <boris.feld@octobus.net> parents: 39515 diff changeset	618 candidates = _refinedgroups(revlog, p1, p2, cachedelta)
51cec7fb672e snapshot: also use None as a stop value for `_refinegroup` Boris Feld <boris.feld@octobus.net> parents: 39515 diff changeset	619 while True:
39517 cc85ebb68ff9 snapshot: turn _refinedgroups into a coroutine Boris Feld <boris.feld@octobus.net> parents: 39516 diff changeset	620 temptative = candidates.send(good)
39516 51cec7fb672e snapshot: also use None as a stop value for `_refinegroup` Boris Feld <boris.feld@octobus.net> parents: 39515 diff changeset	621 if temptative is None:
51cec7fb672e snapshot: also use None as a stop value for `_refinegroup` Boris Feld <boris.feld@octobus.net> parents: 39515 diff changeset	622 break
39364 37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups Boris Feld <boris.feld@octobus.net> parents: 39363 diff changeset	623 group = []
37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups Boris Feld <boris.feld@octobus.net> parents: 39363 diff changeset	624 for rev in temptative:
37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups Boris Feld <boris.feld@octobus.net> parents: 39363 diff changeset	625 # skip over empty delta (no need to include them in a chain)
40425 324ba8b14d78 delta: skip "empty delta" optimisation for non-general case (issue6006) Boris Feld <boris.feld@octobus.net> parents: 40415 diff changeset	626 while (revlog._generaldelta
324ba8b14d78 delta: skip "empty delta" optimisation for non-general case (issue6006) Boris Feld <boris.feld@octobus.net> parents: 40415 diff changeset	627 and not (rev == nullrev
324ba8b14d78 delta: skip "empty delta" optimisation for non-general case (issue6006) Boris Feld <boris.feld@octobus.net> parents: 40415 diff changeset	628 or rev in tested
324ba8b14d78 delta: skip "empty delta" optimisation for non-general case (issue6006) Boris Feld <boris.feld@octobus.net> parents: 40415 diff changeset	629 or deltalength(rev))):
39610 bdb41eaa8b59 snapshot: fix line order when skipping over empty deltas Boris Feld <boris.feld@octobus.net> parents: 39522 diff changeset	630 tested.add(rev)
39364 37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups Boris Feld <boris.feld@octobus.net> parents: 39363 diff changeset	631 rev = deltaparent(rev)
37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups Boris Feld <boris.feld@octobus.net> parents: 39363 diff changeset	632 # filter out revision we tested already
37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups Boris Feld <boris.feld@octobus.net> parents: 39363 diff changeset	633 if rev in tested:
37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups Boris Feld <boris.feld@octobus.net> parents: 39363 diff changeset	634 continue
37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups Boris Feld <boris.feld@octobus.net> parents: 39363 diff changeset	635 tested.add(rev)
37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups Boris Feld <boris.feld@octobus.net> parents: 39363 diff changeset	636 # filter out delta base that will never produce good delta
37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups Boris Feld <boris.feld@octobus.net> parents: 39363 diff changeset	637 if deltas_limit < revlog.length(rev):
37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups Boris Feld <boris.feld@octobus.net> parents: 39363 diff changeset	638 continue
37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups Boris Feld <boris.feld@octobus.net> parents: 39363 diff changeset	639 # no need to try a delta against nullrev, this will be done as a
37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups Boris Feld <boris.feld@octobus.net> parents: 39363 diff changeset	640 # last resort.
37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups Boris Feld <boris.feld@octobus.net> parents: 39363 diff changeset	641 if rev == nullrev:
37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups Boris Feld <boris.feld@octobus.net> parents: 39363 diff changeset	642 continue
37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups Boris Feld <boris.feld@octobus.net> parents: 39363 diff changeset	643 # no delta for rawtext-changing revs (see "candelta" for why)
37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups Boris Feld <boris.feld@octobus.net> parents: 39363 diff changeset	644 if revlog.flags(rev) & REVIDX_RAWTEXT_CHANGING_FLAGS:
37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups Boris Feld <boris.feld@octobus.net> parents: 39363 diff changeset	645 continue
37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups Boris Feld <boris.feld@octobus.net> parents: 39363 diff changeset	646 group.append(rev)
39363 1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39362 diff changeset	647 if group:
39510 3ca144f1c8dd snapshot: search for unrelated but reusable full-snapshot Boris Feld <boris.feld@octobus.net> parents: 39509 diff changeset	648 # XXX: in the sparse revlog case, group can become large,
3ca144f1c8dd snapshot: search for unrelated but reusable full-snapshot Boris Feld <boris.feld@octobus.net> parents: 39509 diff changeset	649 # impacting performances. Some bounding or slicing mecanism
3ca144f1c8dd snapshot: search for unrelated but reusable full-snapshot Boris Feld <boris.feld@octobus.net> parents: 39509 diff changeset	650 # would help to reduce this impact.
39515 04b75f3a3f2a snapshot: add refining logic at the findeltainfo level Boris Feld <boris.feld@octobus.net> parents: 39514 diff changeset	651 good = yield tuple(group)
39514 5b308a4e6d03 snapshot: use None as a stop value when looking for a good delta Boris Feld <boris.feld@octobus.net> parents: 39513 diff changeset	652 yield None
39363 1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39362 diff changeset	653
39510 3ca144f1c8dd snapshot: search for unrelated but reusable full-snapshot Boris Feld <boris.feld@octobus.net> parents: 39509 diff changeset	654 def _findsnapshots(revlog, cache, start_rev):
3ca144f1c8dd snapshot: search for unrelated but reusable full-snapshot Boris Feld <boris.feld@octobus.net> parents: 39509 diff changeset	655 """find snapshot from start_rev to tip"""
3ca144f1c8dd snapshot: search for unrelated but reusable full-snapshot Boris Feld <boris.feld@octobus.net> parents: 39509 diff changeset	656 deltaparent = revlog.deltaparent
39512 6a53842727c1 snapshot: consider unrelated snapshots at a similar level first Boris Feld <boris.feld@octobus.net> parents: 39511 diff changeset	657 issnapshot = revlog.issnapshot
39510 3ca144f1c8dd snapshot: search for unrelated but reusable full-snapshot Boris Feld <boris.feld@octobus.net> parents: 39509 diff changeset	658 for rev in revlog.revs(start_rev):
39512 6a53842727c1 snapshot: consider unrelated snapshots at a similar level first Boris Feld <boris.feld@octobus.net> parents: 39511 diff changeset	659 if issnapshot(rev):
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first Boris Feld <boris.feld@octobus.net> parents: 39511 diff changeset	660 cache[deltaparent(rev)].append(rev)
39510 3ca144f1c8dd snapshot: search for unrelated but reusable full-snapshot Boris Feld <boris.feld@octobus.net> parents: 39509 diff changeset	661
39513 2f9f7889549b snapshot: introduce an intermediate `_refinedgroups` generator Boris Feld <boris.feld@octobus.net> parents: 39512 diff changeset	662 def _refinedgroups(revlog, p1, p2, cachedelta):
2f9f7889549b snapshot: introduce an intermediate `_refinedgroups` generator Boris Feld <boris.feld@octobus.net> parents: 39512 diff changeset	663 good = None
39518 993d7e2c8b79 snapshot: make sure we'll never refine delta base from a reused source Boris Feld <boris.feld@octobus.net> parents: 39517 diff changeset	664 # First we try to reuse a the delta contained in the bundle.
993d7e2c8b79 snapshot: make sure we'll never refine delta base from a reused source Boris Feld <boris.feld@octobus.net> parents: 39517 diff changeset	665 # (or from the source revlog)
993d7e2c8b79 snapshot: make sure we'll never refine delta base from a reused source Boris Feld <boris.feld@octobus.net> parents: 39517 diff changeset	666 #
993d7e2c8b79 snapshot: make sure we'll never refine delta base from a reused source Boris Feld <boris.feld@octobus.net> parents: 39517 diff changeset	667 # This logic only applies to general delta repositories and can be disabled
993d7e2c8b79 snapshot: make sure we'll never refine delta base from a reused source Boris Feld <boris.feld@octobus.net> parents: 39517 diff changeset	668 # through configuration. Disabling reuse source delta is useful when
993d7e2c8b79 snapshot: make sure we'll never refine delta base from a reused source Boris Feld <boris.feld@octobus.net> parents: 39517 diff changeset	669 # we want to make sure we recomputed "optimal" deltas.
993d7e2c8b79 snapshot: make sure we'll never refine delta base from a reused source Boris Feld <boris.feld@octobus.net> parents: 39517 diff changeset	670 if cachedelta and revlog._generaldelta and revlog._lazydeltabase:
993d7e2c8b79 snapshot: make sure we'll never refine delta base from a reused source Boris Feld <boris.feld@octobus.net> parents: 39517 diff changeset	671 # Assume what we received from the server is a good choice
993d7e2c8b79 snapshot: make sure we'll never refine delta base from a reused source Boris Feld <boris.feld@octobus.net> parents: 39517 diff changeset	672 # build delta will reuse the cache
993d7e2c8b79 snapshot: make sure we'll never refine delta base from a reused source Boris Feld <boris.feld@octobus.net> parents: 39517 diff changeset	673 good = yield (cachedelta[0],)
993d7e2c8b79 snapshot: make sure we'll never refine delta base from a reused source Boris Feld <boris.feld@octobus.net> parents: 39517 diff changeset	674 if good is not None:
993d7e2c8b79 snapshot: make sure we'll never refine delta base from a reused source Boris Feld <boris.feld@octobus.net> parents: 39517 diff changeset	675 yield None
993d7e2c8b79 snapshot: make sure we'll never refine delta base from a reused source Boris Feld <boris.feld@octobus.net> parents: 39517 diff changeset	676 return
39513 2f9f7889549b snapshot: introduce an intermediate `_refinedgroups` generator Boris Feld <boris.feld@octobus.net> parents: 39512 diff changeset	677 for candidates in _rawgroups(revlog, p1, p2, cachedelta):
2f9f7889549b snapshot: introduce an intermediate `_refinedgroups` generator Boris Feld <boris.feld@octobus.net> parents: 39512 diff changeset	678 good = yield candidates
2f9f7889549b snapshot: introduce an intermediate `_refinedgroups` generator Boris Feld <boris.feld@octobus.net> parents: 39512 diff changeset	679 if good is not None:
2f9f7889549b snapshot: introduce an intermediate `_refinedgroups` generator Boris Feld <boris.feld@octobus.net> parents: 39512 diff changeset	680 break
39519 e4d4361d0bcd snapshot: try to refine new snapshot base down the chain Boris Feld <boris.feld@octobus.net> parents: 39518 diff changeset	681
40415 bafa1c4bb7a8 sparse-revlog: only refine delta candidates in the sparse case (issue6006) Boris Feld <boris.feld@octobus.net> parents: 39793 diff changeset	682 # If sparse revlog is enabled, we can try to refine the available deltas
bafa1c4bb7a8 sparse-revlog: only refine delta candidates in the sparse case (issue6006) Boris Feld <boris.feld@octobus.net> parents: 39793 diff changeset	683 if not revlog._sparserevlog:
bafa1c4bb7a8 sparse-revlog: only refine delta candidates in the sparse case (issue6006) Boris Feld <boris.feld@octobus.net> parents: 39793 diff changeset	684 yield None
bafa1c4bb7a8 sparse-revlog: only refine delta candidates in the sparse case (issue6006) Boris Feld <boris.feld@octobus.net> parents: 39793 diff changeset	685 return
bafa1c4bb7a8 sparse-revlog: only refine delta candidates in the sparse case (issue6006) Boris Feld <boris.feld@octobus.net> parents: 39793 diff changeset	686
39519 e4d4361d0bcd snapshot: try to refine new snapshot base down the chain Boris Feld <boris.feld@octobus.net> parents: 39518 diff changeset	687 # if we have a refinable value, try to refine it
e4d4361d0bcd snapshot: try to refine new snapshot base down the chain Boris Feld <boris.feld@octobus.net> parents: 39518 diff changeset	688 if good is not None and good not in (p1, p2) and revlog.issnapshot(good):
e4d4361d0bcd snapshot: try to refine new snapshot base down the chain Boris Feld <boris.feld@octobus.net> parents: 39518 diff changeset	689 # refine snapshot down
e4d4361d0bcd snapshot: try to refine new snapshot base down the chain Boris Feld <boris.feld@octobus.net> parents: 39518 diff changeset	690 previous = None
e4d4361d0bcd snapshot: try to refine new snapshot base down the chain Boris Feld <boris.feld@octobus.net> parents: 39518 diff changeset	691 while previous != good:
e4d4361d0bcd snapshot: try to refine new snapshot base down the chain Boris Feld <boris.feld@octobus.net> parents: 39518 diff changeset	692 previous = good
e4d4361d0bcd snapshot: try to refine new snapshot base down the chain Boris Feld <boris.feld@octobus.net> parents: 39518 diff changeset	693 base = revlog.deltaparent(good)
e4d4361d0bcd snapshot: try to refine new snapshot base down the chain Boris Feld <boris.feld@octobus.net> parents: 39518 diff changeset	694 if base == nullrev:
e4d4361d0bcd snapshot: try to refine new snapshot base down the chain Boris Feld <boris.feld@octobus.net> parents: 39518 diff changeset	695 break
e4d4361d0bcd snapshot: try to refine new snapshot base down the chain Boris Feld <boris.feld@octobus.net> parents: 39518 diff changeset	696 good = yield (base,)
39520 5aef5afa8654 snapshot: refine candidate snapshot base upward Boris Feld <boris.feld@octobus.net> parents: 39519 diff changeset	697 # refine snapshot up
5aef5afa8654 snapshot: refine candidate snapshot base upward Boris Feld <boris.feld@octobus.net> parents: 39519 diff changeset	698 #
5aef5afa8654 snapshot: refine candidate snapshot base upward Boris Feld <boris.feld@octobus.net> parents: 39519 diff changeset	699 # XXX the _findsnapshots call can be expensive and is "duplicated" with
5aef5afa8654 snapshot: refine candidate snapshot base upward Boris Feld <boris.feld@octobus.net> parents: 39519 diff changeset	700 # the one done in `_rawgroups`. Once we start working on performance,
5aef5afa8654 snapshot: refine candidate snapshot base upward Boris Feld <boris.feld@octobus.net> parents: 39519 diff changeset	701 # we should make the two logics share this computation.
5aef5afa8654 snapshot: refine candidate snapshot base upward Boris Feld <boris.feld@octobus.net> parents: 39519 diff changeset	702 snapshots = collections.defaultdict(list)
5aef5afa8654 snapshot: refine candidate snapshot base upward Boris Feld <boris.feld@octobus.net> parents: 39519 diff changeset	703 _findsnapshots(revlog, snapshots, good + 1)
5aef5afa8654 snapshot: refine candidate snapshot base upward Boris Feld <boris.feld@octobus.net> parents: 39519 diff changeset	704 previous = None
5aef5afa8654 snapshot: refine candidate snapshot base upward Boris Feld <boris.feld@octobus.net> parents: 39519 diff changeset	705 while good != previous:
5aef5afa8654 snapshot: refine candidate snapshot base upward Boris Feld <boris.feld@octobus.net> parents: 39519 diff changeset	706 previous = good
5aef5afa8654 snapshot: refine candidate snapshot base upward Boris Feld <boris.feld@octobus.net> parents: 39519 diff changeset	707 children = tuple(sorted(c for c in snapshots[good]))
5aef5afa8654 snapshot: refine candidate snapshot base upward Boris Feld <boris.feld@octobus.net> parents: 39519 diff changeset	708 good = yield children
5aef5afa8654 snapshot: refine candidate snapshot base upward Boris Feld <boris.feld@octobus.net> parents: 39519 diff changeset	709
39516 51cec7fb672e snapshot: also use None as a stop value for `_refinegroup` Boris Feld <boris.feld@octobus.net> parents: 39515 diff changeset	710 # we have found nothing
51cec7fb672e snapshot: also use None as a stop value for `_refinegroup` Boris Feld <boris.feld@octobus.net> parents: 39515 diff changeset	711 yield None
39513 2f9f7889549b snapshot: introduce an intermediate `_refinedgroups` generator Boris Feld <boris.feld@octobus.net> parents: 39512 diff changeset	712
39363 1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39362 diff changeset	713 def _rawgroups(revlog, p1, p2, cachedelta):
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39362 diff changeset	714 """Provides group of revision to be tested as delta base
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39362 diff changeset	715
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39362 diff changeset	716 This lower level function focus on emitting delta theorically interresting
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39362 diff changeset	717 without looking it any practical details.
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39362 diff changeset	718
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39362 diff changeset	719 The group order aims at providing fast or small candidates first.
39361 507f5b1dd7c8 revlogdeltas: extract _getcandidaterevs in a function Boris Feld <boris.feld@octobus.net> parents: 39360 diff changeset	720 """
507f5b1dd7c8 revlogdeltas: extract _getcandidaterevs in a function Boris Feld <boris.feld@octobus.net> parents: 39360 diff changeset	721 gdelta = revlog._generaldelta
39509 a33f394b2bfd snapshot: try intermediate snapshot against parents' base Boris Feld <boris.feld@octobus.net> parents: 39505 diff changeset	722 sparse = revlog._sparserevlog
39361 507f5b1dd7c8 revlogdeltas: extract _getcandidaterevs in a function Boris Feld <boris.feld@octobus.net> parents: 39360 diff changeset	723 curr = len(revlog)
507f5b1dd7c8 revlogdeltas: extract _getcandidaterevs in a function Boris Feld <boris.feld@octobus.net> parents: 39360 diff changeset	724 prev = curr - 1
39509 a33f394b2bfd snapshot: try intermediate snapshot against parents' base Boris Feld <boris.feld@octobus.net> parents: 39505 diff changeset	725 deltachain = lambda rev: revlog._deltachain(rev)[0]
39361 507f5b1dd7c8 revlogdeltas: extract _getcandidaterevs in a function Boris Feld <boris.feld@octobus.net> parents: 39360 diff changeset	726
39363 1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39362 diff changeset	727 if gdelta:
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39362 diff changeset	728 # exclude already lazy tested base if any
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39362 diff changeset	729 parents = [p for p in (p1, p2) if p != nullrev]
39361 507f5b1dd7c8 revlogdeltas: extract _getcandidaterevs in a function Boris Feld <boris.feld@octobus.net> parents: 39360 diff changeset	730
39363 1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39362 diff changeset	731 if not revlog._deltabothparents and len(parents) == 2:
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39362 diff changeset	732 parents.sort()
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39362 diff changeset	733 # To minimize the chance of having to build a fulltext,
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39362 diff changeset	734 # pick first whichever parent is closest to us (max rev)
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39362 diff changeset	735 yield (parents[1],)
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39362 diff changeset	736 # then the other one (min rev) if the first did not fit
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39362 diff changeset	737 yield (parents[0],)
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39362 diff changeset	738 elif len(parents) > 0:
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39362 diff changeset	739 # Test all parents (1 or 2), and keep the best candidate
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic Boris Feld <boris.feld@octobus.net> parents: 39362 diff changeset	740 yield parents
39361 507f5b1dd7c8 revlogdeltas: extract _getcandidaterevs in a function Boris Feld <boris.feld@octobus.net> parents: 39360 diff changeset	741
39509 a33f394b2bfd snapshot: try intermediate snapshot against parents' base Boris Feld <boris.feld@octobus.net> parents: 39505 diff changeset	742 if sparse and parents:
39511 e72130f58f5d snapshot: consider all snapshots in the parents' chains Boris Feld <boris.feld@octobus.net> parents: 39510 diff changeset	743 snapshots = collections.defaultdict(list) # map: base-rev: snapshot-rev
39509 a33f394b2bfd snapshot: try intermediate snapshot against parents' base Boris Feld <boris.feld@octobus.net> parents: 39505 diff changeset	744 # See if we can use an existing snapshot in the parent chains to use as
a33f394b2bfd snapshot: try intermediate snapshot against parents' base Boris Feld <boris.feld@octobus.net> parents: 39505 diff changeset	745 # a base for a new intermediate-snapshot
39511 e72130f58f5d snapshot: consider all snapshots in the parents' chains Boris Feld <boris.feld@octobus.net> parents: 39510 diff changeset	746 #
e72130f58f5d snapshot: consider all snapshots in the parents' chains Boris Feld <boris.feld@octobus.net> parents: 39510 diff changeset	747 # search for snapshot in parents delta chain
e72130f58f5d snapshot: consider all snapshots in the parents' chains Boris Feld <boris.feld@octobus.net> parents: 39510 diff changeset	748 # map: snapshot-level: snapshot-rev
e72130f58f5d snapshot: consider all snapshots in the parents' chains Boris Feld <boris.feld@octobus.net> parents: 39510 diff changeset	749 parents_snaps = collections.defaultdict(set)
39521 05a165dc4f55 snapshot: extract parent chain computation Boris Feld <boris.feld@octobus.net> parents: 39520 diff changeset	750 candidate_chains = [deltachain(p) for p in parents]
05a165dc4f55 snapshot: extract parent chain computation Boris Feld <boris.feld@octobus.net> parents: 39520 diff changeset	751 for chain in candidate_chains:
05a165dc4f55 snapshot: extract parent chain computation Boris Feld <boris.feld@octobus.net> parents: 39520 diff changeset	752 for idx, s in enumerate(chain):
39511 e72130f58f5d snapshot: consider all snapshots in the parents' chains Boris Feld <boris.feld@octobus.net> parents: 39510 diff changeset	753 if not revlog.issnapshot(s):
e72130f58f5d snapshot: consider all snapshots in the parents' chains Boris Feld <boris.feld@octobus.net> parents: 39510 diff changeset	754 break
e72130f58f5d snapshot: consider all snapshots in the parents' chains Boris Feld <boris.feld@octobus.net> parents: 39510 diff changeset	755 parents_snaps[idx].add(s)
39512 6a53842727c1 snapshot: consider unrelated snapshots at a similar level first Boris Feld <boris.feld@octobus.net> parents: 39511 diff changeset	756 snapfloor = min(parents_snaps[0]) + 1
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first Boris Feld <boris.feld@octobus.net> parents: 39511 diff changeset	757 _findsnapshots(revlog, snapshots, snapfloor)
39522 c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39521 diff changeset	758 # search for the highest "unrelated" revision
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39521 diff changeset	759 #
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39521 diff changeset	760 # Adding snapshots used by "unrelated" revision increase the odd we
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39521 diff changeset	761 # reuse an independant, yet better snapshot chain.
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39521 diff changeset	762 #
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39521 diff changeset	763 # XXX instead of building a set of revisions, we could lazily enumerate
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39521 diff changeset	764 # over the chains. That would be more efficient, however we stick to
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39521 diff changeset	765 # simple code for now.
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39521 diff changeset	766 all_revs = set()
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39521 diff changeset	767 for chain in candidate_chains:
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39521 diff changeset	768 all_revs.update(chain)
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39521 diff changeset	769 other = None
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39521 diff changeset	770 for r in revlog.revs(prev, snapfloor):
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39521 diff changeset	771 if r not in all_revs:
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39521 diff changeset	772 other = r
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39521 diff changeset	773 break
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39521 diff changeset	774 if other is not None:
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39521 diff changeset	775 # To avoid unfair competition, we won't use unrelated intermediate
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39521 diff changeset	776 # snapshot that are deeper than the ones from the parent delta
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39521 diff changeset	777 # chain.
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39521 diff changeset	778 max_depth = max(parents_snaps.keys())
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39521 diff changeset	779 chain = deltachain(other)
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39521 diff changeset	780 for idx, s in enumerate(chain):
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39521 diff changeset	781 if s < snapfloor:
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39521 diff changeset	782 continue
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39521 diff changeset	783 if max_depth < idx:
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39521 diff changeset	784 break
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39521 diff changeset	785 if not revlog.issnapshot(s):
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39521 diff changeset	786 break
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39521 diff changeset	787 parents_snaps[idx].add(s)
39511 e72130f58f5d snapshot: consider all snapshots in the parents' chains Boris Feld <boris.feld@octobus.net> parents: 39510 diff changeset	788 # Test them as possible intermediate snapshot base
e72130f58f5d snapshot: consider all snapshots in the parents' chains Boris Feld <boris.feld@octobus.net> parents: 39510 diff changeset	789 # We test them from highest to lowest level. High level one are more
e72130f58f5d snapshot: consider all snapshots in the parents' chains Boris Feld <boris.feld@octobus.net> parents: 39510 diff changeset	790 # likely to result in small delta
39512 6a53842727c1 snapshot: consider unrelated snapshots at a similar level first Boris Feld <boris.feld@octobus.net> parents: 39511 diff changeset	791 floor = None
39511 e72130f58f5d snapshot: consider all snapshots in the parents' chains Boris Feld <boris.feld@octobus.net> parents: 39510 diff changeset	792 for idx, snaps in sorted(parents_snaps.items(), reverse=True):
39512 6a53842727c1 snapshot: consider unrelated snapshots at a similar level first Boris Feld <boris.feld@octobus.net> parents: 39511 diff changeset	793 siblings = set()
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first Boris Feld <boris.feld@octobus.net> parents: 39511 diff changeset	794 for s in snaps:
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first Boris Feld <boris.feld@octobus.net> parents: 39511 diff changeset	795 siblings.update(snapshots[s])
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first Boris Feld <boris.feld@octobus.net> parents: 39511 diff changeset	796 # Before considering making a new intermediate snapshot, we check
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first Boris Feld <boris.feld@octobus.net> parents: 39511 diff changeset	797 # if an existing snapshot, children of base we consider, would be
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first Boris Feld <boris.feld@octobus.net> parents: 39511 diff changeset	798 # suitable.
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first Boris Feld <boris.feld@octobus.net> parents: 39511 diff changeset	799 #
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first Boris Feld <boris.feld@octobus.net> parents: 39511 diff changeset	800 # It give a change to reuse a delta chain "unrelated" to the
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first Boris Feld <boris.feld@octobus.net> parents: 39511 diff changeset	801 # current revision instead of starting our own. Without such
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first Boris Feld <boris.feld@octobus.net> parents: 39511 diff changeset	802 # re-use, topological branches would keep reopening new chains.
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first Boris Feld <boris.feld@octobus.net> parents: 39511 diff changeset	803 # Creating more and more snapshot as the repository grow.
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first Boris Feld <boris.feld@octobus.net> parents: 39511 diff changeset	804
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first Boris Feld <boris.feld@octobus.net> parents: 39511 diff changeset	805 if floor is not None:
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first Boris Feld <boris.feld@octobus.net> parents: 39511 diff changeset	806 # We only do this for siblings created after the one in our
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first Boris Feld <boris.feld@octobus.net> parents: 39511 diff changeset	807 # parent's delta chain. Those created before has less chances
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first Boris Feld <boris.feld@octobus.net> parents: 39511 diff changeset	808 # to be valid base since our ancestors had to create a new
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first Boris Feld <boris.feld@octobus.net> parents: 39511 diff changeset	809 # snapshot.
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first Boris Feld <boris.feld@octobus.net> parents: 39511 diff changeset	810 siblings = [r for r in siblings if floor < r]
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first Boris Feld <boris.feld@octobus.net> parents: 39511 diff changeset	811 yield tuple(sorted(siblings))
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first Boris Feld <boris.feld@octobus.net> parents: 39511 diff changeset	812 # then test the base from our parent's delta chain.
39511 e72130f58f5d snapshot: consider all snapshots in the parents' chains Boris Feld <boris.feld@octobus.net> parents: 39510 diff changeset	813 yield tuple(sorted(snaps))
39512 6a53842727c1 snapshot: consider unrelated snapshots at a similar level first Boris Feld <boris.feld@octobus.net> parents: 39511 diff changeset	814 floor = min(snaps)
39510 3ca144f1c8dd snapshot: search for unrelated but reusable full-snapshot Boris Feld <boris.feld@octobus.net> parents: 39509 diff changeset	815 # No suitable base found in the parent chain, search if any full
3ca144f1c8dd snapshot: search for unrelated but reusable full-snapshot Boris Feld <boris.feld@octobus.net> parents: 39509 diff changeset	816 # snapshots emitted since parent's base would be a suitable base for an
3ca144f1c8dd snapshot: search for unrelated but reusable full-snapshot Boris Feld <boris.feld@octobus.net> parents: 39509 diff changeset	817 # intermediate snapshot.
3ca144f1c8dd snapshot: search for unrelated but reusable full-snapshot Boris Feld <boris.feld@octobus.net> parents: 39509 diff changeset	818 #
3ca144f1c8dd snapshot: search for unrelated but reusable full-snapshot Boris Feld <boris.feld@octobus.net> parents: 39509 diff changeset	819 # It give a chance to reuse a delta chain unrelated to the current
3ca144f1c8dd snapshot: search for unrelated but reusable full-snapshot Boris Feld <boris.feld@octobus.net> parents: 39509 diff changeset	820 # revisions instead of starting our own. Without such re-use,
3ca144f1c8dd snapshot: search for unrelated but reusable full-snapshot Boris Feld <boris.feld@octobus.net> parents: 39509 diff changeset	821 # topological branches would keep reopening new full chains. Creating
3ca144f1c8dd snapshot: search for unrelated but reusable full-snapshot Boris Feld <boris.feld@octobus.net> parents: 39509 diff changeset	822 # more and more snapshot as the repository grow.
3ca144f1c8dd snapshot: search for unrelated but reusable full-snapshot Boris Feld <boris.feld@octobus.net> parents: 39509 diff changeset	823 yield tuple(snapshots[nullrev])
39509 a33f394b2bfd snapshot: try intermediate snapshot against parents' base Boris Feld <boris.feld@octobus.net> parents: 39505 diff changeset	824
39522 c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39521 diff changeset	825 if not sparse:
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39521 diff changeset	826 # other approach failed try against prev to hopefully save us a
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39521 diff changeset	827 # fulltext.
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision Boris Feld <boris.feld@octobus.net> parents: 39521 diff changeset	828 yield (prev,)
39361 507f5b1dd7c8 revlogdeltas: extract _getcandidaterevs in a function Boris Feld <boris.feld@octobus.net> parents: 39360 diff changeset	829
39357 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	830 class deltacomputer(object):
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	831 def __init__(self, revlog):
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	832 self.revlog = revlog
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	833
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	834 def buildtext(self, revinfo, fh):
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	835 """Builds a fulltext version of a revision
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	836
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	837 revinfo: _revisioninfo instance that contains all needed info
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	838 fh: file handle to either the .i or the .d revlog file,
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	839 depending on whether it is inlined or not
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	840 """
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	841 btext = revinfo.btext
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	842 if btext[0] is not None:
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	843 return btext[0]
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	844
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	845 revlog = self.revlog
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	846 cachedelta = revinfo.cachedelta
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	847 baserev = cachedelta[0]
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	848 delta = cachedelta[1]
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	849
39358 fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39357 diff changeset	850 fulltext = btext[0] = _textfromdelta(fh, revlog, baserev, delta,
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39357 diff changeset	851 revinfo.p1, revinfo.p2,
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39357 diff changeset	852 revinfo.flags, revinfo.node)
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function Boris Feld <boris.feld@octobus.net> parents: 39357 diff changeset	853 return fulltext
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	854
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	855 def _builddeltadiff(self, base, revinfo, fh):
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	856 revlog = self.revlog
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	857 t = self.buildtext(revinfo, fh)
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	858 if revlog.iscensored(base):
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	859 # deltas based on a censored revision must replace the
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	860 # full content in one patch, so delta works everywhere
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	861 header = mdiff.replacediffheader(revlog.rawsize(base), len(t))
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	862 delta = header + t
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	863 else:
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	864 ptext = revlog.revision(base, _df=fh, raw=True)
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	865 delta = mdiff.textdiff(ptext, t)
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	866
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	867 return delta
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	868
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	869 def _builddeltainfo(self, revinfo, base, fh):
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	870 # can we use the cached delta?
39611 a911932d5003 revlog: reuse cached delta for identical base revision (issue5975) Boris Feld <boris.feld@octobus.net> parents: 39610 diff changeset	871 delta = None
a911932d5003 revlog: reuse cached delta for identical base revision (issue5975) Boris Feld <boris.feld@octobus.net> parents: 39610 diff changeset	872 if revinfo.cachedelta:
a911932d5003 revlog: reuse cached delta for identical base revision (issue5975) Boris Feld <boris.feld@octobus.net> parents: 39610 diff changeset	873 cachebase, cachediff = revinfo.cachedelta
a911932d5003 revlog: reuse cached delta for identical base revision (issue5975) Boris Feld <boris.feld@octobus.net> parents: 39610 diff changeset	874 #check if the diff still apply
a911932d5003 revlog: reuse cached delta for identical base revision (issue5975) Boris Feld <boris.feld@octobus.net> parents: 39610 diff changeset	875 currentbase = cachebase
a911932d5003 revlog: reuse cached delta for identical base revision (issue5975) Boris Feld <boris.feld@octobus.net> parents: 39610 diff changeset	876 while (currentbase != nullrev
a911932d5003 revlog: reuse cached delta for identical base revision (issue5975) Boris Feld <boris.feld@octobus.net> parents: 39610 diff changeset	877 and currentbase != base
a911932d5003 revlog: reuse cached delta for identical base revision (issue5975) Boris Feld <boris.feld@octobus.net> parents: 39610 diff changeset	878 and self.revlog.length(currentbase) == 0):
a911932d5003 revlog: reuse cached delta for identical base revision (issue5975) Boris Feld <boris.feld@octobus.net> parents: 39610 diff changeset	879 currentbase = self.revlog.deltaparent(currentbase)
a911932d5003 revlog: reuse cached delta for identical base revision (issue5975) Boris Feld <boris.feld@octobus.net> parents: 39610 diff changeset	880 if currentbase == base:
a911932d5003 revlog: reuse cached delta for identical base revision (issue5975) Boris Feld <boris.feld@octobus.net> parents: 39610 diff changeset	881 delta = revinfo.cachedelta[1]
a911932d5003 revlog: reuse cached delta for identical base revision (issue5975) Boris Feld <boris.feld@octobus.net> parents: 39610 diff changeset	882 if delta is None:
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	883 delta = self._builddeltadiff(base, revinfo, fh)
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	884 revlog = self.revlog
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	885 header, data = revlog.compress(delta)
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	886 deltalen = len(header) + len(data)
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	887 chainbase = revlog.chainbase(base)
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	888 offset = revlog.end(len(revlog) - 1)
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	889 dist = deltalen + offset - revlog.start(chainbase)
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	890 if revlog._generaldelta:
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	891 deltabase = base
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	892 else:
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	893 deltabase = chainbase
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	894 chainlen, compresseddeltalen = revlog._chaininfo(base)
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	895 chainlen += 1
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	896 compresseddeltalen += deltalen
39187 e0da43e2f71f revlog: compute snapshot depth on delta info Boris Feld <boris.feld@octobus.net> parents: 39185 diff changeset	897
e0da43e2f71f revlog: compute snapshot depth on delta info Boris Feld <boris.feld@octobus.net> parents: 39185 diff changeset	898 revlog = self.revlog
e0da43e2f71f revlog: compute snapshot depth on delta info Boris Feld <boris.feld@octobus.net> parents: 39185 diff changeset	899 snapshotdepth = None
e0da43e2f71f revlog: compute snapshot depth on delta info Boris Feld <boris.feld@octobus.net> parents: 39185 diff changeset	900 if deltabase == nullrev:
e0da43e2f71f revlog: compute snapshot depth on delta info Boris Feld <boris.feld@octobus.net> parents: 39185 diff changeset	901 snapshotdepth = 0
e0da43e2f71f revlog: compute snapshot depth on delta info Boris Feld <boris.feld@octobus.net> parents: 39185 diff changeset	902 elif revlog._sparserevlog and revlog.issnapshot(deltabase):
e0da43e2f71f revlog: compute snapshot depth on delta info Boris Feld <boris.feld@octobus.net> parents: 39185 diff changeset	903 # A delta chain should always be one full snapshot,
e0da43e2f71f revlog: compute snapshot depth on delta info Boris Feld <boris.feld@octobus.net> parents: 39185 diff changeset	904 # zero or more semi-snapshots, and zero or more deltas
e0da43e2f71f revlog: compute snapshot depth on delta info Boris Feld <boris.feld@octobus.net> parents: 39185 diff changeset	905 p1, p2 = revlog.rev(revinfo.p1), revlog.rev(revinfo.p2)
e0da43e2f71f revlog: compute snapshot depth on delta info Boris Feld <boris.feld@octobus.net> parents: 39185 diff changeset	906 if deltabase not in (p1, p2) and revlog.issnapshot(deltabase):
e0da43e2f71f revlog: compute snapshot depth on delta info Boris Feld <boris.feld@octobus.net> parents: 39185 diff changeset	907 snapshotdepth = len(revlog._deltachain(deltabase)[0])
e0da43e2f71f revlog: compute snapshot depth on delta info Boris Feld <boris.feld@octobus.net> parents: 39185 diff changeset	908
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	909 return _deltainfo(dist, deltalen, (header, data), deltabase,
39187 e0da43e2f71f revlog: compute snapshot depth on delta info Boris Feld <boris.feld@octobus.net> parents: 39185 diff changeset	910 chainbase, chainlen, compresseddeltalen,
e0da43e2f71f revlog: compute snapshot depth on delta info Boris Feld <boris.feld@octobus.net> parents: 39185 diff changeset	911 snapshotdepth)
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	912
39360 5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39359 diff changeset	913 def _fullsnapshotinfo(self, fh, revinfo):
5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39359 diff changeset	914 curr = len(self.revlog)
5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39359 diff changeset	915 rawtext = self.buildtext(revinfo, fh)
5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39359 diff changeset	916 data = self.revlog.compress(rawtext)
5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39359 diff changeset	917 compresseddeltalen = deltalen = dist = len(data[1]) + len(data[0])
5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39359 diff changeset	918 deltabase = chainbase = curr
5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39359 diff changeset	919 snapshotdepth = 0
5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39359 diff changeset	920 chainlen = 1
5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39359 diff changeset	921
5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39359 diff changeset	922 return _deltainfo(dist, deltalen, data, deltabase,
5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39359 diff changeset	923 chainbase, chainlen, compresseddeltalen,
5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39359 diff changeset	924 snapshotdepth)
5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39359 diff changeset	925
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	926 def finddeltainfo(self, revinfo, fh):
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	927 """Find an acceptable delta against a candidate revision
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	928
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	929 revinfo: information about the revision (instance of _revisioninfo)
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	930 fh: file handle to either the .i or the .d revlog file,
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	931 depending on whether it is inlined or not
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	932
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	933 Returns the first acceptable candidate revision, as ordered by
39361 507f5b1dd7c8 revlogdeltas: extract _getcandidaterevs in a function Boris Feld <boris.feld@octobus.net> parents: 39360 diff changeset	934 _candidategroups
39360 5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39359 diff changeset	935
5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39359 diff changeset	936 If no suitable deltabase is found, we return delta info for a full
5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39359 diff changeset	937 snapshot.
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	938 """
39122 dbb3e9e44fce revlog: do not search for delta for empty content Boris Feld <boris.feld@octobus.net> parents: 39121 diff changeset	939 if not revinfo.textlen:
39360 5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39359 diff changeset	940 return self._fullsnapshotinfo(fh, revinfo)
39122 dbb3e9e44fce revlog: do not search for delta for empty content Boris Feld <boris.feld@octobus.net> parents: 39121 diff changeset	941
39359 6f4b8f607a31 revlogdeltas: move special cases around raw revisions in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39358 diff changeset	942 # no delta for flag processor revision (see "candelta" for why)
6f4b8f607a31 revlogdeltas: move special cases around raw revisions in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39358 diff changeset	943 # not calling candelta since only one revision needs test, also to
6f4b8f607a31 revlogdeltas: move special cases around raw revisions in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39358 diff changeset	944 # avoid overhead fetching flags again.
6f4b8f607a31 revlogdeltas: move special cases around raw revisions in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39358 diff changeset	945 if revinfo.flags & REVIDX_RAWTEXT_CHANGING_FLAGS:
39360 5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39359 diff changeset	946 return self._fullsnapshotinfo(fh, revinfo)
39359 6f4b8f607a31 revlogdeltas: move special cases around raw revisions in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39358 diff changeset	947
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	948 cachedelta = revinfo.cachedelta
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	949 p1 = revinfo.p1
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	950 p2 = revinfo.p2
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	951 revlog = self.revlog
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	952
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	953 deltainfo = None
39362 1441eb38849f revlogdeltas: pass revision number to _candidatesgroups Boris Feld <boris.feld@octobus.net> parents: 39361 diff changeset	954 p1r, p2r = revlog.rev(p1), revlog.rev(p2)
39364 37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups Boris Feld <boris.feld@octobus.net> parents: 39363 diff changeset	955 groups = _candidategroups(self.revlog, revinfo.textlen,
37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups Boris Feld <boris.feld@octobus.net> parents: 39363 diff changeset	956 p1r, p2r, cachedelta)
39514 5b308a4e6d03 snapshot: use None as a stop value when looking for a good delta Boris Feld <boris.feld@octobus.net> parents: 39513 diff changeset	957 candidaterevs = next(groups)
5b308a4e6d03 snapshot: use None as a stop value when looking for a good delta Boris Feld <boris.feld@octobus.net> parents: 39513 diff changeset	958 while candidaterevs is not None:
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	959 nominateddeltas = []
39515 04b75f3a3f2a snapshot: add refining logic at the findeltainfo level Boris Feld <boris.feld@octobus.net> parents: 39514 diff changeset	960 if deltainfo is not None:
04b75f3a3f2a snapshot: add refining logic at the findeltainfo level Boris Feld <boris.feld@octobus.net> parents: 39514 diff changeset	961 # if we already found a good delta,
04b75f3a3f2a snapshot: add refining logic at the findeltainfo level Boris Feld <boris.feld@octobus.net> parents: 39514 diff changeset	962 # challenge it against refined candidates
04b75f3a3f2a snapshot: add refining logic at the findeltainfo level Boris Feld <boris.feld@octobus.net> parents: 39514 diff changeset	963 nominateddeltas.append(deltainfo)
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	964 for candidaterev in candidaterevs:
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	965 candidatedelta = self._builddeltainfo(revinfo, candidaterev, fh)
39357 655b5b465953 revlog: split functionality related to deltas computation in a new module Boris Feld <boris.feld@octobus.net> parents: 39356 diff changeset	966 if isgooddeltainfo(self.revlog, candidatedelta, revinfo):
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	967 nominateddeltas.append(candidatedelta)
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	968 if nominateddeltas:
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	969 deltainfo = min(nominateddeltas, key=lambda x: x.deltalen)
39515 04b75f3a3f2a snapshot: add refining logic at the findeltainfo level Boris Feld <boris.feld@octobus.net> parents: 39514 diff changeset	970 if deltainfo is not None:
04b75f3a3f2a snapshot: add refining logic at the findeltainfo level Boris Feld <boris.feld@octobus.net> parents: 39514 diff changeset	971 candidaterevs = groups.send(deltainfo.base)
04b75f3a3f2a snapshot: add refining logic at the findeltainfo level Boris Feld <boris.feld@octobus.net> parents: 39514 diff changeset	972 else:
04b75f3a3f2a snapshot: add refining logic at the findeltainfo level Boris Feld <boris.feld@octobus.net> parents: 39514 diff changeset	973 candidaterevs = next(groups)
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	974
39360 5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39359 diff changeset	975 if deltainfo is None:
5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo Boris Feld <boris.feld@octobus.net> parents: 39359 diff changeset	976 deltainfo = self._fullsnapshotinfo(fh, revinfo)
35738 f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object Paul Morelle <paul.morelle@octobus.net> parents: 35737 diff changeset	977 return deltainfo

Mercurial > public > mercurial-scm > hg-stable

annotate mercurial/revlogutils/deltas.py @ 40697:fd1d41ccbe38