Mercurial > public > mercurial-scm > hg

annotate mercurial/revlogutils/deltas.py @ 40608:526ee887c4d5

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
sparse-revlog: stop using a heap to track selected gap Same logic as for 'gapsheap', we don't actually need a heap.
author Boris Feld <boris.feld@octobus.net>
date Thu, 08 Nov 2018 16:07:16 +0100
parents 54de23400b2a
children 85b14f0dc334
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
39330
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
1 # revlogdeltas.py - Logic around delta computation for revlog
8226
8b2cd04a6e97 put license and copyright info into comment blocks
Martin Geisler <mg@lazybytes.net>
parents: 8225
diff changeset
2 #
8b2cd04a6e97 put license and copyright info into comment blocks
Martin Geisler <mg@lazybytes.net>
parents: 8225
diff changeset
3 # Copyright 2005-2007 Matt Mackall <mpm@selenic.com>
39330
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
4 # Copyright 2018 Octobus <contact@octobus.net>
8226
8b2cd04a6e97 put license and copyright info into comment blocks
Martin Geisler <mg@lazybytes.net>
parents: 8225
diff changeset
5 #
8b2cd04a6e97 put license and copyright info into comment blocks
Martin Geisler <mg@lazybytes.net>
parents: 8225
diff changeset
6 # This software may be used and distributed according to the terms of the
10263
25e572394f5c Update license to GPLv2+
Matt Mackall <mpm@selenic.com>
parents: 10047
diff changeset
7 # GNU General Public License version 2 or any later version.
39330
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
8 """Helper class to compute deltas stored inside revlogs"""
8227
0a9542703300 turn some comments back into module docstrings
Martin Geisler <mg@lazybytes.net>
parents: 8226
diff changeset
9
27361
29f50344fa83 revlog: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27251
diff changeset
10 from __future__ import absolute_import
29f50344fa83 revlog: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27251
diff changeset
11
39493
3ca144f1c8dd snapshot: search for unrelated but reusable full-snapshot
Boris Feld <boris.feld@octobus.net>
parents: 39492
diff changeset
12 import collections
27361
29f50344fa83 revlog: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27251
diff changeset
13 import struct
29f50344fa83 revlog: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27251
diff changeset
14
29f50344fa83 revlog: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27251
diff changeset
15 # import stuff from node for others to import from revlog
39330
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
16 from ..node import (
27361
29f50344fa83 revlog: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27251
diff changeset
17 nullrev,
29f50344fa83 revlog: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27251
diff changeset
18 )
39330
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
19 from ..i18n import _
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
20
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
21 from .constants import (
39329
729082bb9938 revlog: split constants into a new `revlogutils.constants` module
Boris Feld <boris.feld@octobus.net>
parents: 39232
diff changeset
22 REVIDX_ISCENSORED,
729082bb9938 revlog: split constants into a new `revlogutils.constants` module
Boris Feld <boris.feld@octobus.net>
parents: 39232
diff changeset
23 REVIDX_RAWTEXT_CHANGING_FLAGS,
729082bb9938 revlog: split constants into a new `revlogutils.constants` module
Boris Feld <boris.feld@octobus.net>
parents: 39232
diff changeset
24 )
39330
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
25
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
26 from ..thirdparty import (
35638
edc9330acac1 revlog: introduce 'deltainfo' to distinguish from 'delta'
Paul Morelle <paul.morelle@octobus.net>
parents: 35637
diff changeset
27 attr,
edc9330acac1 revlog: introduce 'deltainfo' to distinguish from 'delta'
Paul Morelle <paul.morelle@octobus.net>
parents: 35637
diff changeset
28 )
39330
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
29
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
30 from .. import (
27361
29f50344fa83 revlog: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27251
diff changeset
31 error,
29f50344fa83 revlog: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27251
diff changeset
32 mdiff,
29f50344fa83 revlog: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27251
diff changeset
33 )
10913
f2ecc5733c89 revlog: factor out _maxinline global.
Greg Ward <greg-hg@gerg.ca>
parents: 10404
diff changeset
34
39330
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
35 # maximum <delta-chain-data>/<revision-text-length> ratio
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
36 LIMIT_DELTA2TEXT = 2
1091
d62130f99a73 Move hash function back to revlog from node
mpm@selenic.com
parents: 1089
diff changeset
37
38637
e33f784f2a44 revlog: introduce a tiny mock of a revlog class
Boris Feld <boris.feld@octobus.net>
parents: 38636
diff changeset
38 class _testrevlog(object):
e33f784f2a44 revlog: introduce a tiny mock of a revlog class
Boris Feld <boris.feld@octobus.net>
parents: 38636
diff changeset
39 """minimalist fake revlog to use in doctests"""
e33f784f2a44 revlog: introduce a tiny mock of a revlog class
Boris Feld <boris.feld@octobus.net>
parents: 38636
diff changeset
40
e33f784f2a44 revlog: introduce a tiny mock of a revlog class
Boris Feld <boris.feld@octobus.net>
parents: 38636
diff changeset
41 def __init__(self, data, density=0.5, mingap=0):
e33f784f2a44 revlog: introduce a tiny mock of a revlog class
Boris Feld <boris.feld@octobus.net>
parents: 38636
diff changeset
42 """data is an list of revision payload boundaries"""
e33f784f2a44 revlog: introduce a tiny mock of a revlog class
Boris Feld <boris.feld@octobus.net>
parents: 38636
diff changeset
43 self._data = data
e33f784f2a44 revlog: introduce a tiny mock of a revlog class
Boris Feld <boris.feld@octobus.net>
parents: 38636
diff changeset
44 self._srdensitythreshold = density
e33f784f2a44 revlog: introduce a tiny mock of a revlog class
Boris Feld <boris.feld@octobus.net>
parents: 38636
diff changeset
45 self._srmingapsize = mingap
e33f784f2a44 revlog: introduce a tiny mock of a revlog class
Boris Feld <boris.feld@octobus.net>
parents: 38636
diff changeset
46
e33f784f2a44 revlog: introduce a tiny mock of a revlog class
Boris Feld <boris.feld@octobus.net>
parents: 38636
diff changeset
47 def start(self, rev):
e33f784f2a44 revlog: introduce a tiny mock of a revlog class
Boris Feld <boris.feld@octobus.net>
parents: 38636
diff changeset
48 if rev == 0:
e33f784f2a44 revlog: introduce a tiny mock of a revlog class
Boris Feld <boris.feld@octobus.net>
parents: 38636
diff changeset
49 return 0
e33f784f2a44 revlog: introduce a tiny mock of a revlog class
Boris Feld <boris.feld@octobus.net>
parents: 38636
diff changeset
50 return self._data[rev - 1]
e33f784f2a44 revlog: introduce a tiny mock of a revlog class
Boris Feld <boris.feld@octobus.net>
parents: 38636
diff changeset
51
e33f784f2a44 revlog: introduce a tiny mock of a revlog class
Boris Feld <boris.feld@octobus.net>
parents: 38636
diff changeset
52 def end(self, rev):
e33f784f2a44 revlog: introduce a tiny mock of a revlog class
Boris Feld <boris.feld@octobus.net>
parents: 38636
diff changeset
53 return self._data[rev]
e33f784f2a44 revlog: introduce a tiny mock of a revlog class
Boris Feld <boris.feld@octobus.net>
parents: 38636
diff changeset
54
e33f784f2a44 revlog: introduce a tiny mock of a revlog class
Boris Feld <boris.feld@octobus.net>
parents: 38636
diff changeset
55 def length(self, rev):
e33f784f2a44 revlog: introduce a tiny mock of a revlog class
Boris Feld <boris.feld@octobus.net>
parents: 38636
diff changeset
56 return self.end(rev) - self.start(rev)
e33f784f2a44 revlog: introduce a tiny mock of a revlog class
Boris Feld <boris.feld@octobus.net>
parents: 38636
diff changeset
57
38718
f8762ea73e0d sparse-revlog: implement algorithm to write sparse delta chains (issue5480)
Paul Morelle <paul.morelle@octobus.net>
parents: 38717
diff changeset
58 def __len__(self):
f8762ea73e0d sparse-revlog: implement algorithm to write sparse delta chains (issue5480)
Paul Morelle <paul.morelle@octobus.net>
parents: 38717
diff changeset
59 return len(self._data)
f8762ea73e0d sparse-revlog: implement algorithm to write sparse delta chains (issue5480)
Paul Morelle <paul.morelle@octobus.net>
parents: 38717
diff changeset
60
40604
3ac23dad6364 sparse-revlog: drop unused deltainfo parameter from _slicechunktodensity
Boris Feld <boris.feld@octobus.net>
parents: 40603
diff changeset
61 def slicechunk(revlog, revs, targetsize=None):
34824
e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse
Paul Morelle <paul.morelle@octobus.net>
parents: 34823
diff changeset
62 """slice revs to reduce the amount of unrelated data to be read from disk.
e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse
Paul Morelle <paul.morelle@octobus.net>
parents: 34823
diff changeset
63
e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse
Paul Morelle <paul.morelle@octobus.net>
parents: 34823
diff changeset
64 ``revs`` is sliced into groups that should be read in one time.
e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse
Paul Morelle <paul.morelle@octobus.net>
parents: 34823
diff changeset
65 Assume that revs are sorted.
38640
f62b8fb0a484 revlog: document and test _slicechunk
Boris Feld <boris.feld@octobus.net>
parents: 38639
diff changeset
66
f62b8fb0a484 revlog: document and test _slicechunk
Boris Feld <boris.feld@octobus.net>
parents: 38639
diff changeset
67 The initial chunk is sliced until the overall density (payload/chunks-span
f62b8fb0a484 revlog: document and test _slicechunk
Boris Feld <boris.feld@octobus.net>
parents: 38639
diff changeset
68 ratio) is above `revlog._srdensitythreshold`. No gap smaller than
f62b8fb0a484 revlog: document and test _slicechunk
Boris Feld <boris.feld@octobus.net>
parents: 38639
diff changeset
69 `revlog._srmingapsize` is skipped.
f62b8fb0a484 revlog: document and test _slicechunk
Boris Feld <boris.feld@octobus.net>
parents: 38639
diff changeset
70
38643
967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size
Boris Feld <boris.feld@octobus.net>
parents: 38642
diff changeset
71 If `targetsize` is set, no chunk larger than `targetsize` will be yield.
967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size
Boris Feld <boris.feld@octobus.net>
parents: 38642
diff changeset
72 For consistency with other slicing choice, this limit won't go lower than
967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size
Boris Feld <boris.feld@octobus.net>
parents: 38642
diff changeset
73 `revlog._srmingapsize`.
967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size
Boris Feld <boris.feld@octobus.net>
parents: 38642
diff changeset
74
967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size
Boris Feld <boris.feld@octobus.net>
parents: 38642
diff changeset
75 If individual revisions chunk are larger than this limit, they will still
967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size
Boris Feld <boris.feld@octobus.net>
parents: 38642
diff changeset
76 be raised individually.
967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size
Boris Feld <boris.feld@octobus.net>
parents: 38642
diff changeset
77
38640
f62b8fb0a484 revlog: document and test _slicechunk
Boris Feld <boris.feld@octobus.net>
parents: 38639
diff changeset
78 >>> revlog = _testrevlog([
f62b8fb0a484 revlog: document and test _slicechunk
Boris Feld <boris.feld@octobus.net>
parents: 38639
diff changeset
79 ... 5, #00 (5)
f62b8fb0a484 revlog: document and test _slicechunk
Boris Feld <boris.feld@octobus.net>
parents: 38639
diff changeset
80 ... 10, #01 (5)
f62b8fb0a484 revlog: document and test _slicechunk
Boris Feld <boris.feld@octobus.net>
parents: 38639
diff changeset
81 ... 12, #02 (2)
f62b8fb0a484 revlog: document and test _slicechunk
Boris Feld <boris.feld@octobus.net>
parents: 38639
diff changeset
82 ... 12, #03 (empty)
f62b8fb0a484 revlog: document and test _slicechunk
Boris Feld <boris.feld@octobus.net>
parents: 38639
diff changeset
83 ... 27, #04 (15)
f62b8fb0a484 revlog: document and test _slicechunk
Boris Feld <boris.feld@octobus.net>
parents: 38639
diff changeset
84 ... 31, #05 (4)
f62b8fb0a484 revlog: document and test _slicechunk
Boris Feld <boris.feld@octobus.net>
parents: 38639
diff changeset
85 ... 31, #06 (empty)
f62b8fb0a484 revlog: document and test _slicechunk
Boris Feld <boris.feld@octobus.net>
parents: 38639
diff changeset
86 ... 42, #07 (11)
f62b8fb0a484 revlog: document and test _slicechunk
Boris Feld <boris.feld@octobus.net>
parents: 38639
diff changeset
87 ... 47, #08 (5)
f62b8fb0a484 revlog: document and test _slicechunk
Boris Feld <boris.feld@octobus.net>
parents: 38639
diff changeset
88 ... 47, #09 (empty)
f62b8fb0a484 revlog: document and test _slicechunk
Boris Feld <boris.feld@octobus.net>
parents: 38639
diff changeset
89 ... 48, #10 (1)
f62b8fb0a484 revlog: document and test _slicechunk
Boris Feld <boris.feld@octobus.net>
parents: 38639
diff changeset
90 ... 51, #11 (3)
f62b8fb0a484 revlog: document and test _slicechunk
Boris Feld <boris.feld@octobus.net>
parents: 38639
diff changeset
91 ... 74, #12 (23)
f62b8fb0a484 revlog: document and test _slicechunk
Boris Feld <boris.feld@octobus.net>
parents: 38639
diff changeset
92 ... 85, #13 (11)
f62b8fb0a484 revlog: document and test _slicechunk
Boris Feld <boris.feld@octobus.net>
parents: 38639
diff changeset
93 ... 86, #14 (1)
f62b8fb0a484 revlog: document and test _slicechunk
Boris Feld <boris.feld@octobus.net>
parents: 38639
diff changeset
94 ... 91, #15 (5)
f62b8fb0a484 revlog: document and test _slicechunk
Boris Feld <boris.feld@octobus.net>
parents: 38639
diff changeset
95 ... ])
f62b8fb0a484 revlog: document and test _slicechunk
Boris Feld <boris.feld@octobus.net>
parents: 38639
diff changeset
96
39330
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
97 >>> list(slicechunk(revlog, list(range(16))))
38640
f62b8fb0a484 revlog: document and test _slicechunk
Boris Feld <boris.feld@octobus.net>
parents: 38639
diff changeset
98 [[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]]
39330
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
99 >>> list(slicechunk(revlog, [0, 15]))
38640
f62b8fb0a484 revlog: document and test _slicechunk
Boris Feld <boris.feld@octobus.net>
parents: 38639
diff changeset
100 [[0], [15]]
39330
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
101 >>> list(slicechunk(revlog, [0, 11, 15]))
38640
f62b8fb0a484 revlog: document and test _slicechunk
Boris Feld <boris.feld@octobus.net>
parents: 38639
diff changeset
102 [[0], [11], [15]]
39330
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
103 >>> list(slicechunk(revlog, [0, 11, 13, 15]))
38640
f62b8fb0a484 revlog: document and test _slicechunk
Boris Feld <boris.feld@octobus.net>
parents: 38639
diff changeset
104 [[0], [11, 13, 15]]
39330
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
105 >>> list(slicechunk(revlog, [1, 2, 3, 5, 8, 10, 11, 14]))
38640
f62b8fb0a484 revlog: document and test _slicechunk
Boris Feld <boris.feld@octobus.net>
parents: 38639
diff changeset
106 [[1, 2], [5, 8, 10, 11], [14]]
38643
967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size
Boris Feld <boris.feld@octobus.net>
parents: 38642
diff changeset
107
967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size
Boris Feld <boris.feld@octobus.net>
parents: 38642
diff changeset
108 Slicing with a maximum chunk size
39330
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
109 >>> list(slicechunk(revlog, [0, 11, 13, 15], targetsize=15))
38643
967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size
Boris Feld <boris.feld@octobus.net>
parents: 38642
diff changeset
110 [[0], [11], [13], [15]]
39330
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
111 >>> list(slicechunk(revlog, [0, 11, 13, 15], targetsize=20))
38643
967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size
Boris Feld <boris.feld@octobus.net>
parents: 38642
diff changeset
112 [[0], [11], [13, 15]]
34824
e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse
Paul Morelle <paul.morelle@octobus.net>
parents: 34823
diff changeset
113 """
38643
967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size
Boris Feld <boris.feld@octobus.net>
parents: 38642
diff changeset
114 if targetsize is not None:
967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size
Boris Feld <boris.feld@octobus.net>
parents: 38642
diff changeset
115 targetsize = max(targetsize, revlog._srmingapsize)
38718
f8762ea73e0d sparse-revlog: implement algorithm to write sparse delta chains (issue5480)
Paul Morelle <paul.morelle@octobus.net>
parents: 38717
diff changeset
116 # targetsize should not be specified when evaluating delta candidates:
f8762ea73e0d sparse-revlog: implement algorithm to write sparse delta chains (issue5480)
Paul Morelle <paul.morelle@octobus.net>
parents: 38717
diff changeset
117 # * targetsize is used to ensure we stay within specification when reading,
38641
feba6be0941b revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents: 38640
diff changeset
118 for chunk in _slicechunktodensity(revlog, revs,
feba6be0941b revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents: 38640
diff changeset
119 revlog._srdensitythreshold,
feba6be0941b revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents: 38640
diff changeset
120 revlog._srmingapsize):
38643
967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size
Boris Feld <boris.feld@octobus.net>
parents: 38642
diff changeset
121 for subchunk in _slicechunktosize(revlog, chunk, targetsize):
967fee55e8d9 revlog: postprocess chunk to slice them down to a certain size
Boris Feld <boris.feld@octobus.net>
parents: 38642
diff changeset
122 yield subchunk
38641
feba6be0941b revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents: 38640
diff changeset
123
38718
f8762ea73e0d sparse-revlog: implement algorithm to write sparse delta chains (issue5480)
Paul Morelle <paul.morelle@octobus.net>
parents: 38717
diff changeset
124 def _slicechunktosize(revlog, revs, targetsize=None):
38642
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
125 """slice revs to match the target size
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
126
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
127 This is intended to be used on chunk that density slicing selected by that
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
128 are still too large compared to the read garantee of revlog. This might
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
129 happens when "minimal gap size" interrupted the slicing or when chain are
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
130 built in a way that create large blocks next to each other.
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
131
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
132 >>> revlog = _testrevlog([
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
133 ... 3, #0 (3)
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
134 ... 5, #1 (2)
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
135 ... 6, #2 (1)
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
136 ... 8, #3 (2)
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
137 ... 8, #4 (empty)
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
138 ... 11, #5 (3)
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
139 ... 12, #6 (1)
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
140 ... 13, #7 (1)
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
141 ... 14, #8 (1)
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
142 ... ])
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
143
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
144 Cases where chunk is already small enough
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
145 >>> list(_slicechunktosize(revlog, [0], 3))
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
146 [[0]]
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
147 >>> list(_slicechunktosize(revlog, [6, 7], 3))
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
148 [[6, 7]]
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
149 >>> list(_slicechunktosize(revlog, [0], None))
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
150 [[0]]
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
151 >>> list(_slicechunktosize(revlog, [6, 7], None))
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
152 [[6, 7]]
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
153
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
154 cases where we need actual slicing
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
155 >>> list(_slicechunktosize(revlog, [0, 1], 3))
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
156 [[0], [1]]
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
157 >>> list(_slicechunktosize(revlog, [1, 3], 3))
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
158 [[1], [3]]
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
159 >>> list(_slicechunktosize(revlog, [1, 2, 3], 3))
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
160 [[1, 2], [3]]
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
161 >>> list(_slicechunktosize(revlog, [3, 5], 3))
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
162 [[3], [5]]
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
163 >>> list(_slicechunktosize(revlog, [3, 4, 5], 3))
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
164 [[3], [5]]
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
165 >>> list(_slicechunktosize(revlog, [5, 6, 7, 8], 3))
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
166 [[5], [6, 7, 8]]
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
167 >>> list(_slicechunktosize(revlog, [0, 1, 2, 3, 4, 5, 6, 7, 8], 3))
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
168 [[0], [1, 2], [3], [5], [6, 7, 8]]
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
169
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
170 Case with too large individual chunk (must return valid chunk)
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
171 >>> list(_slicechunktosize(revlog, [0, 1], 2))
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
172 [[0], [1]]
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
173 >>> list(_slicechunktosize(revlog, [1, 3], 1))
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
174 [[1], [3]]
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
175 >>> list(_slicechunktosize(revlog, [3, 4, 5], 2))
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
176 [[3], [5]]
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
177 """
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
178 assert targetsize is None or 0 <= targetsize
39330
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
179 if targetsize is None or segmentspan(revlog, revs) <= targetsize:
38642
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
180 yield revs
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
181 return
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
182
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
183 startrevidx = 0
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
184 startdata = revlog.start(revs[0])
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
185 endrevidx = 0
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
186 iterrevs = enumerate(revs)
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
187 next(iterrevs) # skip first rev.
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
188 for idx, r in iterrevs:
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
189 span = revlog.end(r) - startdata
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
190 if span <= targetsize:
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
191 endrevidx = idx
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
192 else:
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
193 chunk = _trimchunk(revlog, revs, startrevidx, endrevidx + 1)
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
194 if chunk:
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
195 yield chunk
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
196 startrevidx = idx
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
197 startdata = revlog.start(r)
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
198 endrevidx = idx
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
199 yield _trimchunk(revlog, revs, startrevidx)
e59e27e52297 revlog: add function to slice chunk down to a given size
Boris Feld <boris.feld@octobus.net>
parents: 38641
diff changeset
200
40604
3ac23dad6364 sparse-revlog: drop unused deltainfo parameter from _slicechunktodensity
Boris Feld <boris.feld@octobus.net>
parents: 40603
diff changeset
201 def _slicechunktodensity(revlog, revs, targetdensity=0.5,
38718
f8762ea73e0d sparse-revlog: implement algorithm to write sparse delta chains (issue5480)
Paul Morelle <paul.morelle@octobus.net>
parents: 38717
diff changeset
202 mingapsize=0):
38641
feba6be0941b revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents: 38640
diff changeset
203 """slice revs to reduce the amount of unrelated data to be read from disk.
feba6be0941b revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents: 38640
diff changeset
204
feba6be0941b revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents: 38640
diff changeset
205 ``revs`` is sliced into groups that should be read in one time.
feba6be0941b revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents: 38640
diff changeset
206 Assume that revs are sorted.
feba6be0941b revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents: 38640
diff changeset
207
feba6be0941b revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents: 38640
diff changeset
208 The initial chunk is sliced until the overall density (payload/chunks-span
feba6be0941b revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents: 38640
diff changeset
209 ratio) is above `targetdensity`. No gap smaller than `mingapsize` is
feba6be0941b revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents: 38640
diff changeset
210 skipped.
feba6be0941b revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents: 38640
diff changeset
211
feba6be0941b revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents: 38640
diff changeset
212 >>> revlog = _testrevlog([
feba6be0941b revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents: 38640
diff changeset
213 ... 5, #00 (5)
feba6be0941b revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents: 38640
diff changeset
214 ... 10, #01 (5)
feba6be0941b revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents: 38640
diff changeset
215 ... 12, #02 (2)
feba6be0941b revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents: 38640
diff changeset
216 ... 12, #03 (empty)
feba6be0941b revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents: 38640
diff changeset
217 ... 27, #04 (15)
feba6be0941b revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents: 38640
diff changeset
218 ... 31, #05 (4)
feba6be0941b revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents: 38640
diff changeset
219 ... 31, #06 (empty)
feba6be0941b revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents: 38640
diff changeset
220 ... 42, #07 (11)
feba6be0941b revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents: 38640
diff changeset
221 ... 47, #08 (5)
feba6be0941b revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents: 38640
diff changeset
222 ... 47, #09 (empty)
feba6be0941b revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents: 38640
diff changeset
223 ... 48, #10 (1)
feba6be0941b revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents: 38640
diff changeset
224 ... 51, #11 (3)
feba6be0941b revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents: 38640
diff changeset
225 ... 74, #12 (23)
feba6be0941b revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents: 38640
diff changeset
226 ... 85, #13 (11)
feba6be0941b revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents: 38640
diff changeset
227 ... 86, #14 (1)
feba6be0941b revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents: 38640
diff changeset
228 ... 91, #15 (5)
feba6be0941b revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents: 38640
diff changeset
229 ... ])
feba6be0941b revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents: 38640
diff changeset
230
38655
cd1c484e31e8 revlog: adjust doctest examples to be portable to Python 3
Augie Fackler <augie@google.com>
parents: 38644
diff changeset
231 >>> list(_slicechunktodensity(revlog, list(range(16))))
38641
feba6be0941b revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents: 38640
diff changeset
232 [[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]]
feba6be0941b revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents: 38640
diff changeset
233 >>> list(_slicechunktodensity(revlog, [0, 15]))
feba6be0941b revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents: 38640
diff changeset
234 [[0], [15]]
feba6be0941b revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents: 38640
diff changeset
235 >>> list(_slicechunktodensity(revlog, [0, 11, 15]))
feba6be0941b revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents: 38640
diff changeset
236 [[0], [11], [15]]
feba6be0941b revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents: 38640
diff changeset
237 >>> list(_slicechunktodensity(revlog, [0, 11, 13, 15]))
feba6be0941b revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents: 38640
diff changeset
238 [[0], [11, 13, 15]]
feba6be0941b revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents: 38640
diff changeset
239 >>> list(_slicechunktodensity(revlog, [1, 2, 3, 5, 8, 10, 11, 14]))
feba6be0941b revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents: 38640
diff changeset
240 [[1, 2], [5, 8, 10, 11], [14]]
feba6be0941b revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents: 38640
diff changeset
241 >>> list(_slicechunktodensity(revlog, [1, 2, 3, 5, 8, 10, 11, 14],
feba6be0941b revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents: 38640
diff changeset
242 ... mingapsize=20))
feba6be0941b revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents: 38640
diff changeset
243 [[1, 2, 3, 5, 8, 10, 11], [14]]
feba6be0941b revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents: 38640
diff changeset
244 >>> list(_slicechunktodensity(revlog, [1, 2, 3, 5, 8, 10, 11, 14],
feba6be0941b revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents: 38640
diff changeset
245 ... targetdensity=0.95))
feba6be0941b revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents: 38640
diff changeset
246 [[1, 2], [5], [8, 10, 11], [14]]
feba6be0941b revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents: 38640
diff changeset
247 >>> list(_slicechunktodensity(revlog, [1, 2, 3, 5, 8, 10, 11, 14],
feba6be0941b revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents: 38640
diff changeset
248 ... targetdensity=0.95, mingapsize=12))
feba6be0941b revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents: 38640
diff changeset
249 [[1, 2], [5, 8, 10, 11], [14]]
feba6be0941b revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents: 38640
diff changeset
250 """
34824
e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse
Paul Morelle <paul.morelle@octobus.net>
parents: 34823
diff changeset
251 start = revlog.start
e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse
Paul Morelle <paul.morelle@octobus.net>
parents: 34823
diff changeset
252 length = revlog.length
e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse
Paul Morelle <paul.morelle@octobus.net>
parents: 34823
diff changeset
253
34880
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one
Boris Feld <boris.feld@octobus.net>
parents: 34825
diff changeset
254 if len(revs) <= 1:
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one
Boris Feld <boris.feld@octobus.net>
parents: 34825
diff changeset
255 yield revs
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one
Boris Feld <boris.feld@octobus.net>
parents: 34825
diff changeset
256 return
34824
e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse
Paul Morelle <paul.morelle@octobus.net>
parents: 34823
diff changeset
257
40604
3ac23dad6364 sparse-revlog: drop unused deltainfo parameter from _slicechunktodensity
Boris Feld <boris.feld@octobus.net>
parents: 40603
diff changeset
258 deltachainspan = segmentspan(revlog, revs)
34880
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one
Boris Feld <boris.feld@octobus.net>
parents: 34825
diff changeset
259
38641
feba6be0941b revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents: 38640
diff changeset
260 if deltachainspan < mingapsize:
38635
d083ae26c325 revlog: early return in _slicechunk when span is already small enough
Boris Feld <boris.feld@octobus.net>
parents: 38634
diff changeset
261 yield revs
d083ae26c325 revlog: early return in _slicechunk when span is already small enough
Boris Feld <boris.feld@octobus.net>
parents: 38634
diff changeset
262 return
d083ae26c325 revlog: early return in _slicechunk when span is already small enough
Boris Feld <boris.feld@octobus.net>
parents: 38634
diff changeset
263
38718
f8762ea73e0d sparse-revlog: implement algorithm to write sparse delta chains (issue5480)
Paul Morelle <paul.morelle@octobus.net>
parents: 38717
diff changeset
264 readdata = deltachainspan
40606
bfbfd15d65bd sparse-revlog: fast-path before computing payload size
Boris Feld <boris.feld@octobus.net>
parents: 40605
diff changeset
265 chainpayload = sum(length(r) for r in revs)
34824
e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse
Paul Morelle <paul.morelle@octobus.net>
parents: 34823
diff changeset
266
34880
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one
Boris Feld <boris.feld@octobus.net>
parents: 34825
diff changeset
267 if deltachainspan:
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one
Boris Feld <boris.feld@octobus.net>
parents: 34825
diff changeset
268 density = chainpayload / float(deltachainspan)
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one
Boris Feld <boris.feld@octobus.net>
parents: 34825
diff changeset
269 else:
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one
Boris Feld <boris.feld@octobus.net>
parents: 34825
diff changeset
270 density = 1.0
34824
e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse
Paul Morelle <paul.morelle@octobus.net>
parents: 34823
diff changeset
271
38641
feba6be0941b revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents: 38640
diff changeset
272 if density >= targetdensity:
38634
f0ea8b847831 revlog: early return in _slicechunk when density is already good
Paul Morelle <paul.morelle@octobus.net>
parents: 38632
diff changeset
273 yield revs
f0ea8b847831 revlog: early return in _slicechunk when density is already good
Paul Morelle <paul.morelle@octobus.net>
parents: 38632
diff changeset
274 return
f0ea8b847831 revlog: early return in _slicechunk when density is already good
Paul Morelle <paul.morelle@octobus.net>
parents: 38632
diff changeset
275
34880
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one
Boris Feld <boris.feld@octobus.net>
parents: 34825
diff changeset
276 # Store the gaps in a heap to have them sorted by decreasing size
40607
54de23400b2a sparse-revlog: stop using a heap to track gaps
Boris Feld <boris.feld@octobus.net>
parents: 40606
diff changeset
277 gaps = []
34880
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one
Boris Feld <boris.feld@octobus.net>
parents: 34825
diff changeset
278 prevend = None
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one
Boris Feld <boris.feld@octobus.net>
parents: 34825
diff changeset
279 for i, rev in enumerate(revs):
40604
3ac23dad6364 sparse-revlog: drop unused deltainfo parameter from _slicechunktodensity
Boris Feld <boris.feld@octobus.net>
parents: 40603
diff changeset
280 revstart = start(rev)
3ac23dad6364 sparse-revlog: drop unused deltainfo parameter from _slicechunktodensity
Boris Feld <boris.feld@octobus.net>
parents: 40603
diff changeset
281 revlen = length(rev)
34824
e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse
Paul Morelle <paul.morelle@octobus.net>
parents: 34823
diff changeset
282
34898
1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk
Paul Morelle <paul.morelle@octobus.net>
parents: 34881
diff changeset
283 # Skip empty revisions to form larger holes
1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk
Paul Morelle <paul.morelle@octobus.net>
parents: 34881
diff changeset
284 if revlen == 0:
1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk
Paul Morelle <paul.morelle@octobus.net>
parents: 34881
diff changeset
285 continue
1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk
Paul Morelle <paul.morelle@octobus.net>
parents: 34881
diff changeset
286
34880
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one
Boris Feld <boris.feld@octobus.net>
parents: 34825
diff changeset
287 if prevend is not None:
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one
Boris Feld <boris.feld@octobus.net>
parents: 34825
diff changeset
288 gapsize = revstart - prevend
34881
8c9b08a0c48c sparse-read: skip gaps too small to be worth splitting
Paul Morelle <paul.morelle@octobus.net>
parents: 34880
diff changeset
289 # only consider holes that are large enough
38641
feba6be0941b revlog: extract density based slicing into its own function
Boris Feld <boris.feld@octobus.net>
parents: 38640
diff changeset
290 if gapsize > mingapsize:
40607
54de23400b2a sparse-revlog: stop using a heap to track gaps
Boris Feld <boris.feld@octobus.net>
parents: 40606
diff changeset
291 gaps.append((gapsize, i))
34880
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one
Boris Feld <boris.feld@octobus.net>
parents: 34825
diff changeset
292
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one
Boris Feld <boris.feld@octobus.net>
parents: 34825
diff changeset
293 prevend = revstart + revlen
40607
54de23400b2a sparse-revlog: stop using a heap to track gaps
Boris Feld <boris.feld@octobus.net>
parents: 40606
diff changeset
294 # sort the gaps to pop them from largest to small
54de23400b2a sparse-revlog: stop using a heap to track gaps
Boris Feld <boris.feld@octobus.net>
parents: 40606
diff changeset
295 gaps.sort()
34880
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one
Boris Feld <boris.feld@octobus.net>
parents: 34825
diff changeset
296
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one
Boris Feld <boris.feld@octobus.net>
parents: 34825
diff changeset
297 # Collect the indices of the largest holes until the density is acceptable
40608
526ee887c4d5 sparse-revlog: stop using a heap to track selected gap
Boris Feld <boris.feld@octobus.net>
parents: 40607
diff changeset
298 selected = []
40607
54de23400b2a sparse-revlog: stop using a heap to track gaps
Boris Feld <boris.feld@octobus.net>
parents: 40606
diff changeset
299 while gaps and density < targetdensity:
54de23400b2a sparse-revlog: stop using a heap to track gaps
Boris Feld <boris.feld@octobus.net>
parents: 40606
diff changeset
300 gapsize, gapidx = gaps.pop()
34880
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one
Boris Feld <boris.feld@octobus.net>
parents: 34825
diff changeset
301
40608
526ee887c4d5 sparse-revlog: stop using a heap to track selected gap
Boris Feld <boris.feld@octobus.net>
parents: 40607
diff changeset
302 selected.append(gapidx)
34824
e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse
Paul Morelle <paul.morelle@octobus.net>
parents: 34823
diff changeset
303
34880
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one
Boris Feld <boris.feld@octobus.net>
parents: 34825
diff changeset
304 # the gap sizes are stored as negatives to be sorted decreasingly
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one
Boris Feld <boris.feld@octobus.net>
parents: 34825
diff changeset
305 # by the heap
40607
54de23400b2a sparse-revlog: stop using a heap to track gaps
Boris Feld <boris.feld@octobus.net>
parents: 40606
diff changeset
306 readdata -= gapsize
34880
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one
Boris Feld <boris.feld@octobus.net>
parents: 34825
diff changeset
307 if readdata > 0:
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one
Boris Feld <boris.feld@octobus.net>
parents: 34825
diff changeset
308 density = chainpayload / float(readdata)
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one
Boris Feld <boris.feld@octobus.net>
parents: 34825
diff changeset
309 else:
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one
Boris Feld <boris.feld@octobus.net>
parents: 34825
diff changeset
310 density = 1.0
40608
526ee887c4d5 sparse-revlog: stop using a heap to track selected gap
Boris Feld <boris.feld@octobus.net>
parents: 40607
diff changeset
311 selected.sort()
34824
e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse
Paul Morelle <paul.morelle@octobus.net>
parents: 34823
diff changeset
312
34880
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one
Boris Feld <boris.feld@octobus.net>
parents: 34825
diff changeset
313 # Cut the revs at collected indices
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one
Boris Feld <boris.feld@octobus.net>
parents: 34825
diff changeset
314 previdx = 0
40608
526ee887c4d5 sparse-revlog: stop using a heap to track selected gap
Boris Feld <boris.feld@octobus.net>
parents: 40607
diff changeset
315 for idx in selected:
34898
1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk
Paul Morelle <paul.morelle@octobus.net>
parents: 34881
diff changeset
316
1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk
Paul Morelle <paul.morelle@octobus.net>
parents: 34881
diff changeset
317 chunk = _trimchunk(revlog, revs, previdx, idx)
1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk
Paul Morelle <paul.morelle@octobus.net>
parents: 34881
diff changeset
318 if chunk:
1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk
Paul Morelle <paul.morelle@octobus.net>
parents: 34881
diff changeset
319 yield chunk
1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk
Paul Morelle <paul.morelle@octobus.net>
parents: 34881
diff changeset
320
34880
9e18ab7f7240 sparse-read: move from a recursive-based approach to a heap-based one
Boris Feld <boris.feld@octobus.net>
parents: 34825
diff changeset
321 previdx = idx
34898
1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk
Paul Morelle <paul.morelle@octobus.net>
parents: 34881
diff changeset
322
1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk
Paul Morelle <paul.morelle@octobus.net>
parents: 34881
diff changeset
323 chunk = _trimchunk(revlog, revs, previdx)
1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk
Paul Morelle <paul.morelle@octobus.net>
parents: 34881
diff changeset
324 if chunk:
1bde8e8e5de0 sparse-read: ignore trailing empty revs in each read chunk
Paul Morelle <paul.morelle@octobus.net>
parents: 34881
diff changeset
325 yield chunk
34824
e2ad93bcc084 revlog: introduce an experimental flag to slice chunks reads when too sparse
Paul Morelle <paul.morelle@octobus.net>
parents: 34823
diff changeset
326
39330
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
327 def _trimchunk(revlog, revs, startidx, endidx=None):
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
328 """returns revs[startidx:endidx] without empty trailing revs
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
329
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
330 Doctest Setup
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
331 >>> revlog = _testrevlog([
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
332 ... 5, #0
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
333 ... 10, #1
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
334 ... 12, #2
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
335 ... 12, #3 (empty)
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
336 ... 17, #4
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
337 ... 21, #5
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
338 ... 21, #6 (empty)
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
339 ... ])
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
340
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
341 Contiguous cases:
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
342 >>> _trimchunk(revlog, [0, 1, 2, 3, 4, 5, 6], 0)
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
343 [0, 1, 2, 3, 4, 5]
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
344 >>> _trimchunk(revlog, [0, 1, 2, 3, 4, 5, 6], 0, 5)
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
345 [0, 1, 2, 3, 4]
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
346 >>> _trimchunk(revlog, [0, 1, 2, 3, 4, 5, 6], 0, 4)
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
347 [0, 1, 2]
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
348 >>> _trimchunk(revlog, [0, 1, 2, 3, 4, 5, 6], 2, 4)
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
349 [2]
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
350 >>> _trimchunk(revlog, [0, 1, 2, 3, 4, 5, 6], 3)
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
351 [3, 4, 5]
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
352 >>> _trimchunk(revlog, [0, 1, 2, 3, 4, 5, 6], 3, 5)
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
353 [3, 4]
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
354
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
355 Discontiguous cases:
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
356 >>> _trimchunk(revlog, [1, 3, 5, 6], 0)
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
357 [1, 3, 5]
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
358 >>> _trimchunk(revlog, [1, 3, 5, 6], 0, 2)
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
359 [1]
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
360 >>> _trimchunk(revlog, [1, 3, 5, 6], 1, 3)
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
361 [3, 5]
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
362 >>> _trimchunk(revlog, [1, 3, 5, 6], 1)
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
363 [3, 5]
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
364 """
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
365 length = revlog.length
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
366
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
367 if endidx is None:
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
368 endidx = len(revs)
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
369
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
370 # If we have a non-emtpy delta candidate, there are nothing to trim
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
371 if revs[endidx - 1] < len(revlog):
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
372 # Trim empty revs at the end, except the very first revision of a chain
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
373 while (endidx > 1
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
374 and endidx > startidx
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
375 and length(revs[endidx - 1]) == 0):
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
376 endidx -= 1
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
377
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
378 return revs[startidx:endidx]
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
379
40605
a32ccd32982b sparse-revlog: drop unused deltainfo parameter from segmentspan
Boris Feld <boris.feld@octobus.net>
parents: 40604
diff changeset
380 def segmentspan(revlog, revs):
39330
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
381 """Get the byte span of a segment of revisions
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
382
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
383 revs is a sorted array of revision numbers
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
384
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
385 >>> revlog = _testrevlog([
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
386 ... 5, #0
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
387 ... 10, #1
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
388 ... 12, #2
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
389 ... 12, #3 (empty)
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
390 ... 17, #4
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
391 ... ])
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
392
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
393 >>> segmentspan(revlog, [0, 1, 2, 3, 4])
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
394 17
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
395 >>> segmentspan(revlog, [0, 4])
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
396 17
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
397 >>> segmentspan(revlog, [3, 4])
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
398 5
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
399 >>> segmentspan(revlog, [1, 2, 3,])
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
400 7
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
401 >>> segmentspan(revlog, [1, 3])
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
402 7
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
403 """
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
404 if not revs:
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
405 return 0
40605
a32ccd32982b sparse-revlog: drop unused deltainfo parameter from segmentspan
Boris Feld <boris.feld@octobus.net>
parents: 40604
diff changeset
406 end = revlog.end(revs[-1])
39330
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
407 return end - revlog.start(revs[0])
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
408
39331
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function
Boris Feld <boris.feld@octobus.net>
parents: 39330
diff changeset
409 def _textfromdelta(fh, revlog, baserev, delta, p1, p2, flags, expectednode):
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function
Boris Feld <boris.feld@octobus.net>
parents: 39330
diff changeset
410 """build full text from a (base, delta) pair and other metadata"""
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function
Boris Feld <boris.feld@octobus.net>
parents: 39330
diff changeset
411 # special case deltas which replace entire base; no need to decode
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function
Boris Feld <boris.feld@octobus.net>
parents: 39330
diff changeset
412 # base revision. this neatly avoids censored bases, which throw when
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function
Boris Feld <boris.feld@octobus.net>
parents: 39330
diff changeset
413 # they're decoded.
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function
Boris Feld <boris.feld@octobus.net>
parents: 39330
diff changeset
414 hlen = struct.calcsize(">lll")
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function
Boris Feld <boris.feld@octobus.net>
parents: 39330
diff changeset
415 if delta[:hlen] == mdiff.replacediffheader(revlog.rawsize(baserev),
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function
Boris Feld <boris.feld@octobus.net>
parents: 39330
diff changeset
416 len(delta) - hlen):
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function
Boris Feld <boris.feld@octobus.net>
parents: 39330
diff changeset
417 fulltext = delta[hlen:]
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function
Boris Feld <boris.feld@octobus.net>
parents: 39330
diff changeset
418 else:
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function
Boris Feld <boris.feld@octobus.net>
parents: 39330
diff changeset
419 # deltabase is rawtext before changed by flag processors, which is
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function
Boris Feld <boris.feld@octobus.net>
parents: 39330
diff changeset
420 # equivalent to non-raw text
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function
Boris Feld <boris.feld@octobus.net>
parents: 39330
diff changeset
421 basetext = revlog.revision(baserev, _df=fh, raw=False)
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function
Boris Feld <boris.feld@octobus.net>
parents: 39330
diff changeset
422 fulltext = mdiff.patch(basetext, delta)
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function
Boris Feld <boris.feld@octobus.net>
parents: 39330
diff changeset
423
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function
Boris Feld <boris.feld@octobus.net>
parents: 39330
diff changeset
424 try:
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function
Boris Feld <boris.feld@octobus.net>
parents: 39330
diff changeset
425 res = revlog._processflags(fulltext, flags, 'read', raw=True)
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function
Boris Feld <boris.feld@octobus.net>
parents: 39330
diff changeset
426 fulltext, validatehash = res
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function
Boris Feld <boris.feld@octobus.net>
parents: 39330
diff changeset
427 if validatehash:
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function
Boris Feld <boris.feld@octobus.net>
parents: 39330
diff changeset
428 revlog.checkhash(fulltext, expectednode, p1=p1, p2=p2)
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function
Boris Feld <boris.feld@octobus.net>
parents: 39330
diff changeset
429 if flags & REVIDX_ISCENSORED:
39777
b63dee7bd0d9 global: replace most uses of RevlogError with StorageError (API)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39774
diff changeset
430 raise error.StorageError(_('node %s is not censored') %
b63dee7bd0d9 global: replace most uses of RevlogError with StorageError (API)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39774
diff changeset
431 expectednode)
39774
4a2466b2a434 revlog: drop some more error aliases (API)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39773
diff changeset
432 except error.CensoredNodeError:
39331
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function
Boris Feld <boris.feld@octobus.net>
parents: 39330
diff changeset
433 # must pass the censored index flag to add censored revisions
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function
Boris Feld <boris.feld@octobus.net>
parents: 39330
diff changeset
434 if not flags & REVIDX_ISCENSORED:
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function
Boris Feld <boris.feld@octobus.net>
parents: 39330
diff changeset
435 raise
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function
Boris Feld <boris.feld@octobus.net>
parents: 39330
diff changeset
436 return fulltext
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function
Boris Feld <boris.feld@octobus.net>
parents: 39330
diff changeset
437
35638
edc9330acac1 revlog: introduce 'deltainfo' to distinguish from 'delta'
Paul Morelle <paul.morelle@octobus.net>
parents: 35637
diff changeset
438 @attr.s(slots=True, frozen=True)
edc9330acac1 revlog: introduce 'deltainfo' to distinguish from 'delta'
Paul Morelle <paul.morelle@octobus.net>
parents: 35637
diff changeset
439 class _deltainfo(object):
edc9330acac1 revlog: introduce 'deltainfo' to distinguish from 'delta'
Paul Morelle <paul.morelle@octobus.net>
parents: 35637
diff changeset
440 distance = attr.ib()
edc9330acac1 revlog: introduce 'deltainfo' to distinguish from 'delta'
Paul Morelle <paul.morelle@octobus.net>
parents: 35637
diff changeset
441 deltalen = attr.ib()
edc9330acac1 revlog: introduce 'deltainfo' to distinguish from 'delta'
Paul Morelle <paul.morelle@octobus.net>
parents: 35637
diff changeset
442 data = attr.ib()
edc9330acac1 revlog: introduce 'deltainfo' to distinguish from 'delta'
Paul Morelle <paul.morelle@octobus.net>
parents: 35637
diff changeset
443 base = attr.ib()
edc9330acac1 revlog: introduce 'deltainfo' to distinguish from 'delta'
Paul Morelle <paul.morelle@octobus.net>
parents: 35637
diff changeset
444 chainbase = attr.ib()
edc9330acac1 revlog: introduce 'deltainfo' to distinguish from 'delta'
Paul Morelle <paul.morelle@octobus.net>
parents: 35637
diff changeset
445 chainlen = attr.ib()
edc9330acac1 revlog: introduce 'deltainfo' to distinguish from 'delta'
Paul Morelle <paul.morelle@octobus.net>
parents: 35637
diff changeset
446 compresseddeltalen = attr.ib()
39154
e0da43e2f71f revlog: compute snapshot depth on delta info
Boris Feld <boris.feld@octobus.net>
parents: 39152
diff changeset
447 snapshotdepth = attr.ib()
35638
edc9330acac1 revlog: introduce 'deltainfo' to distinguish from 'delta'
Paul Morelle <paul.morelle@octobus.net>
parents: 35637
diff changeset
448
39330
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
449 def isgooddeltainfo(revlog, deltainfo, revinfo):
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
450 """Returns True if the given delta is good. Good means that it is within
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
451 the disk span, disk size, and chain length bounds that we know to be
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
452 performant."""
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
453 if deltainfo is None:
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
454 return False
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
455
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
456 # - 'deltainfo.distance' is the distance from the base revision --
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
457 # bounding it limits the amount of I/O we need to do.
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
458 # - 'deltainfo.compresseddeltalen' is the sum of the total size of
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
459 # deltas we need to apply -- bounding it limits the amount of CPU
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
460 # we consume.
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
461
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
462 textlen = revinfo.textlen
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
463 defaultmax = textlen * 4
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
464 maxdist = revlog._maxdeltachainspan
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
465 if not maxdist:
40603
2f7e531ef3e7 sparse-revlog: skip the span check in the sparse-revlog case
Boris Feld <boris.feld@octobus.net>
parents: 40451
diff changeset
466 maxdist = deltainfo.distance # ensure the conditional pass
39330
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
467 maxdist = max(maxdist, defaultmax)
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
468
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
469 # Bad delta from read span:
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
470 #
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
471 # If the span of data read is larger than the maximum allowed.
40603
2f7e531ef3e7 sparse-revlog: skip the span check in the sparse-revlog case
Boris Feld <boris.feld@octobus.net>
parents: 40451
diff changeset
472 #
2f7e531ef3e7 sparse-revlog: skip the span check in the sparse-revlog case
Boris Feld <boris.feld@octobus.net>
parents: 40451
diff changeset
473 # In the sparse-revlog case, we rely on the associated "sparse reading"
2f7e531ef3e7 sparse-revlog: skip the span check in the sparse-revlog case
Boris Feld <boris.feld@octobus.net>
parents: 40451
diff changeset
474 # to avoid issue related to the span of data. In theory, it would be
2f7e531ef3e7 sparse-revlog: skip the span check in the sparse-revlog case
Boris Feld <boris.feld@octobus.net>
parents: 40451
diff changeset
475 # possible to build pathological revlog where delta pattern would lead
2f7e531ef3e7 sparse-revlog: skip the span check in the sparse-revlog case
Boris Feld <boris.feld@octobus.net>
parents: 40451
diff changeset
476 # to too many reads. However, they do not happen in practice at all. So
2f7e531ef3e7 sparse-revlog: skip the span check in the sparse-revlog case
Boris Feld <boris.feld@octobus.net>
parents: 40451
diff changeset
477 # we skip the span check entirely.
2f7e531ef3e7 sparse-revlog: skip the span check in the sparse-revlog case
Boris Feld <boris.feld@octobus.net>
parents: 40451
diff changeset
478 if not revlog._sparserevlog and maxdist < deltainfo.distance:
39330
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
479 return False
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
480
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
481 # Bad delta from new delta size:
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
482 #
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
483 # If the delta size is larger than the target text, storing the
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
484 # delta will be inefficient.
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
485 if textlen < deltainfo.deltalen:
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
486 return False
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
487
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
488 # Bad delta from cumulated payload size:
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
489 #
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
490 # If the sum of delta get larger than K * target text length.
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
491 if textlen * LIMIT_DELTA2TEXT < deltainfo.compresseddeltalen:
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
492 return False
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
493
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
494 # Bad delta from chain length:
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
495 #
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
496 # If the number of delta in the chain gets too high.
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
497 if (revlog._maxchainlen
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
498 and revlog._maxchainlen < deltainfo.chainlen):
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
499 return False
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
500
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
501 # bad delta from intermediate snapshot size limit
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
502 #
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
503 # If an intermediate snapshot size is higher than the limit. The
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
504 # limit exist to prevent endless chain of intermediate delta to be
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
505 # created.
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
506 if (deltainfo.snapshotdepth is not None and
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
507 (textlen >> deltainfo.snapshotdepth) < deltainfo.deltalen):
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
508 return False
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
509
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
510 # bad delta if new intermediate snapshot is larger than the previous
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
511 # snapshot
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
512 if (deltainfo.snapshotdepth
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
513 and revlog.length(deltainfo.base) < deltainfo.deltalen):
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
514 return False
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
515
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
516 return True
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
517
39337
37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups
Boris Feld <boris.feld@octobus.net>
parents: 39336
diff changeset
518 def _candidategroups(revlog, textlen, p1, p2, cachedelta):
39336
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic
Boris Feld <boris.feld@octobus.net>
parents: 39335
diff changeset
519 """Provides group of revision to be tested as delta base
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic
Boris Feld <boris.feld@octobus.net>
parents: 39335
diff changeset
520
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic
Boris Feld <boris.feld@octobus.net>
parents: 39335
diff changeset
521 This top level function focus on emitting groups with unique and worthwhile
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic
Boris Feld <boris.feld@octobus.net>
parents: 39335
diff changeset
522 content. See _raw_candidate_groups for details about the group order.
39334
507f5b1dd7c8 revlogdeltas: extract _getcandidaterevs in a function
Boris Feld <boris.feld@octobus.net>
parents: 39333
diff changeset
523 """
39336
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic
Boris Feld <boris.feld@octobus.net>
parents: 39335
diff changeset
524 # should we try to build a delta?
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic
Boris Feld <boris.feld@octobus.net>
parents: 39335
diff changeset
525 if not (len(revlog) and revlog._storedeltachains):
39497
5b308a4e6d03 snapshot: use None as a stop value when looking for a good delta
Boris Feld <boris.feld@octobus.net>
parents: 39496
diff changeset
526 yield None
39336
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic
Boris Feld <boris.feld@octobus.net>
parents: 39335
diff changeset
527 return
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic
Boris Feld <boris.feld@octobus.net>
parents: 39335
diff changeset
528
39337
37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups
Boris Feld <boris.feld@octobus.net>
parents: 39336
diff changeset
529 deltalength = revlog.length
37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups
Boris Feld <boris.feld@octobus.net>
parents: 39336
diff changeset
530 deltaparent = revlog.deltaparent
39498
04b75f3a3f2a snapshot: add refining logic at the findeltainfo level
Boris Feld <boris.feld@octobus.net>
parents: 39497
diff changeset
531 good = None
39337
37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups
Boris Feld <boris.feld@octobus.net>
parents: 39336
diff changeset
532
37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups
Boris Feld <boris.feld@octobus.net>
parents: 39336
diff changeset
533 deltas_limit = textlen * LIMIT_DELTA2TEXT
37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups
Boris Feld <boris.feld@octobus.net>
parents: 39336
diff changeset
534
39336
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic
Boris Feld <boris.feld@octobus.net>
parents: 39335
diff changeset
535 tested = set([nullrev])
39499
51cec7fb672e snapshot: also use None as a stop value for `_refinegroup`
Boris Feld <boris.feld@octobus.net>
parents: 39498
diff changeset
536 candidates = _refinedgroups(revlog, p1, p2, cachedelta)
51cec7fb672e snapshot: also use None as a stop value for `_refinegroup`
Boris Feld <boris.feld@octobus.net>
parents: 39498
diff changeset
537 while True:
39500
cc85ebb68ff9 snapshot: turn _refinedgroups into a coroutine
Boris Feld <boris.feld@octobus.net>
parents: 39499
diff changeset
538 temptative = candidates.send(good)
39499
51cec7fb672e snapshot: also use None as a stop value for `_refinegroup`
Boris Feld <boris.feld@octobus.net>
parents: 39498
diff changeset
539 if temptative is None:
51cec7fb672e snapshot: also use None as a stop value for `_refinegroup`
Boris Feld <boris.feld@octobus.net>
parents: 39498
diff changeset
540 break
39337
37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups
Boris Feld <boris.feld@octobus.net>
parents: 39336
diff changeset
541 group = []
37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups
Boris Feld <boris.feld@octobus.net>
parents: 39336
diff changeset
542 for rev in temptative:
37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups
Boris Feld <boris.feld@octobus.net>
parents: 39336
diff changeset
543 # skip over empty delta (no need to include them in a chain)
40451
324ba8b14d78 delta: skip "empty delta" optimisation for non-general case (issue6006)
Boris Feld <boris.feld@octobus.net>
parents: 40428
diff changeset
544 while (revlog._generaldelta
324ba8b14d78 delta: skip "empty delta" optimisation for non-general case (issue6006)
Boris Feld <boris.feld@octobus.net>
parents: 40428
diff changeset
545 and not (rev == nullrev
324ba8b14d78 delta: skip "empty delta" optimisation for non-general case (issue6006)
Boris Feld <boris.feld@octobus.net>
parents: 40428
diff changeset
546 or rev in tested
324ba8b14d78 delta: skip "empty delta" optimisation for non-general case (issue6006)
Boris Feld <boris.feld@octobus.net>
parents: 40428
diff changeset
547 or deltalength(rev))):
39594
bdb41eaa8b59 snapshot: fix line order when skipping over empty deltas
Boris Feld <boris.feld@octobus.net>
parents: 39505
diff changeset
548 tested.add(rev)
39337
37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups
Boris Feld <boris.feld@octobus.net>
parents: 39336
diff changeset
549 rev = deltaparent(rev)
37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups
Boris Feld <boris.feld@octobus.net>
parents: 39336
diff changeset
550 # filter out revision we tested already
37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups
Boris Feld <boris.feld@octobus.net>
parents: 39336
diff changeset
551 if rev in tested:
37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups
Boris Feld <boris.feld@octobus.net>
parents: 39336
diff changeset
552 continue
37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups
Boris Feld <boris.feld@octobus.net>
parents: 39336
diff changeset
553 tested.add(rev)
37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups
Boris Feld <boris.feld@octobus.net>
parents: 39336
diff changeset
554 # filter out delta base that will never produce good delta
37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups
Boris Feld <boris.feld@octobus.net>
parents: 39336
diff changeset
555 if deltas_limit < revlog.length(rev):
37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups
Boris Feld <boris.feld@octobus.net>
parents: 39336
diff changeset
556 continue
37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups
Boris Feld <boris.feld@octobus.net>
parents: 39336
diff changeset
557 # no need to try a delta against nullrev, this will be done as a
37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups
Boris Feld <boris.feld@octobus.net>
parents: 39336
diff changeset
558 # last resort.
37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups
Boris Feld <boris.feld@octobus.net>
parents: 39336
diff changeset
559 if rev == nullrev:
37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups
Boris Feld <boris.feld@octobus.net>
parents: 39336
diff changeset
560 continue
37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups
Boris Feld <boris.feld@octobus.net>
parents: 39336
diff changeset
561 # no delta for rawtext-changing revs (see "candelta" for why)
37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups
Boris Feld <boris.feld@octobus.net>
parents: 39336
diff changeset
562 if revlog.flags(rev) & REVIDX_RAWTEXT_CHANGING_FLAGS:
37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups
Boris Feld <boris.feld@octobus.net>
parents: 39336
diff changeset
563 continue
37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups
Boris Feld <boris.feld@octobus.net>
parents: 39336
diff changeset
564 group.append(rev)
39336
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic
Boris Feld <boris.feld@octobus.net>
parents: 39335
diff changeset
565 if group:
39493
3ca144f1c8dd snapshot: search for unrelated but reusable full-snapshot
Boris Feld <boris.feld@octobus.net>
parents: 39492
diff changeset
566 # XXX: in the sparse revlog case, group can become large,
3ca144f1c8dd snapshot: search for unrelated but reusable full-snapshot
Boris Feld <boris.feld@octobus.net>
parents: 39492
diff changeset
567 # impacting performances. Some bounding or slicing mecanism
3ca144f1c8dd snapshot: search for unrelated but reusable full-snapshot
Boris Feld <boris.feld@octobus.net>
parents: 39492
diff changeset
568 # would help to reduce this impact.
39498
04b75f3a3f2a snapshot: add refining logic at the findeltainfo level
Boris Feld <boris.feld@octobus.net>
parents: 39497
diff changeset
569 good = yield tuple(group)
39497
5b308a4e6d03 snapshot: use None as a stop value when looking for a good delta
Boris Feld <boris.feld@octobus.net>
parents: 39496
diff changeset
570 yield None
39336
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic
Boris Feld <boris.feld@octobus.net>
parents: 39335
diff changeset
571
39493
3ca144f1c8dd snapshot: search for unrelated but reusable full-snapshot
Boris Feld <boris.feld@octobus.net>
parents: 39492
diff changeset
572 def _findsnapshots(revlog, cache, start_rev):
3ca144f1c8dd snapshot: search for unrelated but reusable full-snapshot
Boris Feld <boris.feld@octobus.net>
parents: 39492
diff changeset
573 """find snapshot from start_rev to tip"""
3ca144f1c8dd snapshot: search for unrelated but reusable full-snapshot
Boris Feld <boris.feld@octobus.net>
parents: 39492
diff changeset
574 deltaparent = revlog.deltaparent
39495
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first
Boris Feld <boris.feld@octobus.net>
parents: 39494
diff changeset
575 issnapshot = revlog.issnapshot
39493
3ca144f1c8dd snapshot: search for unrelated but reusable full-snapshot
Boris Feld <boris.feld@octobus.net>
parents: 39492
diff changeset
576 for rev in revlog.revs(start_rev):
39495
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first
Boris Feld <boris.feld@octobus.net>
parents: 39494
diff changeset
577 if issnapshot(rev):
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first
Boris Feld <boris.feld@octobus.net>
parents: 39494
diff changeset
578 cache[deltaparent(rev)].append(rev)
39493
3ca144f1c8dd snapshot: search for unrelated but reusable full-snapshot
Boris Feld <boris.feld@octobus.net>
parents: 39492
diff changeset
579
39496
2f9f7889549b snapshot: introduce an intermediate `_refinedgroups` generator
Boris Feld <boris.feld@octobus.net>
parents: 39495
diff changeset
580 def _refinedgroups(revlog, p1, p2, cachedelta):
2f9f7889549b snapshot: introduce an intermediate `_refinedgroups` generator
Boris Feld <boris.feld@octobus.net>
parents: 39495
diff changeset
581 good = None
39501
993d7e2c8b79 snapshot: make sure we'll never refine delta base from a reused source
Boris Feld <boris.feld@octobus.net>
parents: 39500
diff changeset
582 # First we try to reuse a the delta contained in the bundle.
993d7e2c8b79 snapshot: make sure we'll never refine delta base from a reused source
Boris Feld <boris.feld@octobus.net>
parents: 39500
diff changeset
583 # (or from the source revlog)
993d7e2c8b79 snapshot: make sure we'll never refine delta base from a reused source
Boris Feld <boris.feld@octobus.net>
parents: 39500
diff changeset
584 #
993d7e2c8b79 snapshot: make sure we'll never refine delta base from a reused source
Boris Feld <boris.feld@octobus.net>
parents: 39500
diff changeset
585 # This logic only applies to general delta repositories and can be disabled
993d7e2c8b79 snapshot: make sure we'll never refine delta base from a reused source
Boris Feld <boris.feld@octobus.net>
parents: 39500
diff changeset
586 # through configuration. Disabling reuse source delta is useful when
993d7e2c8b79 snapshot: make sure we'll never refine delta base from a reused source
Boris Feld <boris.feld@octobus.net>
parents: 39500
diff changeset
587 # we want to make sure we recomputed "optimal" deltas.
993d7e2c8b79 snapshot: make sure we'll never refine delta base from a reused source
Boris Feld <boris.feld@octobus.net>
parents: 39500
diff changeset
588 if cachedelta and revlog._generaldelta and revlog._lazydeltabase:
993d7e2c8b79 snapshot: make sure we'll never refine delta base from a reused source
Boris Feld <boris.feld@octobus.net>
parents: 39500
diff changeset
589 # Assume what we received from the server is a good choice
993d7e2c8b79 snapshot: make sure we'll never refine delta base from a reused source
Boris Feld <boris.feld@octobus.net>
parents: 39500
diff changeset
590 # build delta will reuse the cache
993d7e2c8b79 snapshot: make sure we'll never refine delta base from a reused source
Boris Feld <boris.feld@octobus.net>
parents: 39500
diff changeset
591 good = yield (cachedelta[0],)
993d7e2c8b79 snapshot: make sure we'll never refine delta base from a reused source
Boris Feld <boris.feld@octobus.net>
parents: 39500
diff changeset
592 if good is not None:
993d7e2c8b79 snapshot: make sure we'll never refine delta base from a reused source
Boris Feld <boris.feld@octobus.net>
parents: 39500
diff changeset
593 yield None
993d7e2c8b79 snapshot: make sure we'll never refine delta base from a reused source
Boris Feld <boris.feld@octobus.net>
parents: 39500
diff changeset
594 return
39496
2f9f7889549b snapshot: introduce an intermediate `_refinedgroups` generator
Boris Feld <boris.feld@octobus.net>
parents: 39495
diff changeset
595 for candidates in _rawgroups(revlog, p1, p2, cachedelta):
2f9f7889549b snapshot: introduce an intermediate `_refinedgroups` generator
Boris Feld <boris.feld@octobus.net>
parents: 39495
diff changeset
596 good = yield candidates
2f9f7889549b snapshot: introduce an intermediate `_refinedgroups` generator
Boris Feld <boris.feld@octobus.net>
parents: 39495
diff changeset
597 if good is not None:
2f9f7889549b snapshot: introduce an intermediate `_refinedgroups` generator
Boris Feld <boris.feld@octobus.net>
parents: 39495
diff changeset
598 break
39502
e4d4361d0bcd snapshot: try to refine new snapshot base down the chain
Boris Feld <boris.feld@octobus.net>
parents: 39501
diff changeset
599
40428
bafa1c4bb7a8 sparse-revlog: only refine delta candidates in the sparse case (issue6006)
Boris Feld <boris.feld@octobus.net>
parents: 39777
diff changeset
600 # If sparse revlog is enabled, we can try to refine the available deltas
bafa1c4bb7a8 sparse-revlog: only refine delta candidates in the sparse case (issue6006)
Boris Feld <boris.feld@octobus.net>
parents: 39777
diff changeset
601 if not revlog._sparserevlog:
bafa1c4bb7a8 sparse-revlog: only refine delta candidates in the sparse case (issue6006)
Boris Feld <boris.feld@octobus.net>
parents: 39777
diff changeset
602 yield None
bafa1c4bb7a8 sparse-revlog: only refine delta candidates in the sparse case (issue6006)
Boris Feld <boris.feld@octobus.net>
parents: 39777
diff changeset
603 return
bafa1c4bb7a8 sparse-revlog: only refine delta candidates in the sparse case (issue6006)
Boris Feld <boris.feld@octobus.net>
parents: 39777
diff changeset
604
39502
e4d4361d0bcd snapshot: try to refine new snapshot base down the chain
Boris Feld <boris.feld@octobus.net>
parents: 39501
diff changeset
605 # if we have a refinable value, try to refine it
e4d4361d0bcd snapshot: try to refine new snapshot base down the chain
Boris Feld <boris.feld@octobus.net>
parents: 39501
diff changeset
606 if good is not None and good not in (p1, p2) and revlog.issnapshot(good):
e4d4361d0bcd snapshot: try to refine new snapshot base down the chain
Boris Feld <boris.feld@octobus.net>
parents: 39501
diff changeset
607 # refine snapshot down
e4d4361d0bcd snapshot: try to refine new snapshot base down the chain
Boris Feld <boris.feld@octobus.net>
parents: 39501
diff changeset
608 previous = None
e4d4361d0bcd snapshot: try to refine new snapshot base down the chain
Boris Feld <boris.feld@octobus.net>
parents: 39501
diff changeset
609 while previous != good:
e4d4361d0bcd snapshot: try to refine new snapshot base down the chain
Boris Feld <boris.feld@octobus.net>
parents: 39501
diff changeset
610 previous = good
e4d4361d0bcd snapshot: try to refine new snapshot base down the chain
Boris Feld <boris.feld@octobus.net>
parents: 39501
diff changeset
611 base = revlog.deltaparent(good)
e4d4361d0bcd snapshot: try to refine new snapshot base down the chain
Boris Feld <boris.feld@octobus.net>
parents: 39501
diff changeset
612 if base == nullrev:
e4d4361d0bcd snapshot: try to refine new snapshot base down the chain
Boris Feld <boris.feld@octobus.net>
parents: 39501
diff changeset
613 break
e4d4361d0bcd snapshot: try to refine new snapshot base down the chain
Boris Feld <boris.feld@octobus.net>
parents: 39501
diff changeset
614 good = yield (base,)
39503
5aef5afa8654 snapshot: refine candidate snapshot base upward
Boris Feld <boris.feld@octobus.net>
parents: 39502
diff changeset
615 # refine snapshot up
5aef5afa8654 snapshot: refine candidate snapshot base upward
Boris Feld <boris.feld@octobus.net>
parents: 39502
diff changeset
616 #
5aef5afa8654 snapshot: refine candidate snapshot base upward
Boris Feld <boris.feld@octobus.net>
parents: 39502
diff changeset
617 # XXX the _findsnapshots call can be expensive and is "duplicated" with
5aef5afa8654 snapshot: refine candidate snapshot base upward
Boris Feld <boris.feld@octobus.net>
parents: 39502
diff changeset
618 # the one done in `_rawgroups`. Once we start working on performance,
5aef5afa8654 snapshot: refine candidate snapshot base upward
Boris Feld <boris.feld@octobus.net>
parents: 39502
diff changeset
619 # we should make the two logics share this computation.
5aef5afa8654 snapshot: refine candidate snapshot base upward
Boris Feld <boris.feld@octobus.net>
parents: 39502
diff changeset
620 snapshots = collections.defaultdict(list)
5aef5afa8654 snapshot: refine candidate snapshot base upward
Boris Feld <boris.feld@octobus.net>
parents: 39502
diff changeset
621 _findsnapshots(revlog, snapshots, good + 1)
5aef5afa8654 snapshot: refine candidate snapshot base upward
Boris Feld <boris.feld@octobus.net>
parents: 39502
diff changeset
622 previous = None
5aef5afa8654 snapshot: refine candidate snapshot base upward
Boris Feld <boris.feld@octobus.net>
parents: 39502
diff changeset
623 while good != previous:
5aef5afa8654 snapshot: refine candidate snapshot base upward
Boris Feld <boris.feld@octobus.net>
parents: 39502
diff changeset
624 previous = good
5aef5afa8654 snapshot: refine candidate snapshot base upward
Boris Feld <boris.feld@octobus.net>
parents: 39502
diff changeset
625 children = tuple(sorted(c for c in snapshots[good]))
5aef5afa8654 snapshot: refine candidate snapshot base upward
Boris Feld <boris.feld@octobus.net>
parents: 39502
diff changeset
626 good = yield children
5aef5afa8654 snapshot: refine candidate snapshot base upward
Boris Feld <boris.feld@octobus.net>
parents: 39502
diff changeset
627
39499
51cec7fb672e snapshot: also use None as a stop value for `_refinegroup`
Boris Feld <boris.feld@octobus.net>
parents: 39498
diff changeset
628 # we have found nothing
51cec7fb672e snapshot: also use None as a stop value for `_refinegroup`
Boris Feld <boris.feld@octobus.net>
parents: 39498
diff changeset
629 yield None
39496
2f9f7889549b snapshot: introduce an intermediate `_refinedgroups` generator
Boris Feld <boris.feld@octobus.net>
parents: 39495
diff changeset
630
39336
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic
Boris Feld <boris.feld@octobus.net>
parents: 39335
diff changeset
631 def _rawgroups(revlog, p1, p2, cachedelta):
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic
Boris Feld <boris.feld@octobus.net>
parents: 39335
diff changeset
632 """Provides group of revision to be tested as delta base
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic
Boris Feld <boris.feld@octobus.net>
parents: 39335
diff changeset
633
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic
Boris Feld <boris.feld@octobus.net>
parents: 39335
diff changeset
634 This lower level function focus on emitting delta theorically interresting
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic
Boris Feld <boris.feld@octobus.net>
parents: 39335
diff changeset
635 without looking it any practical details.
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic
Boris Feld <boris.feld@octobus.net>
parents: 39335
diff changeset
636
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic
Boris Feld <boris.feld@octobus.net>
parents: 39335
diff changeset
637 The group order aims at providing fast or small candidates first.
39334
507f5b1dd7c8 revlogdeltas: extract _getcandidaterevs in a function
Boris Feld <boris.feld@octobus.net>
parents: 39333
diff changeset
638 """
507f5b1dd7c8 revlogdeltas: extract _getcandidaterevs in a function
Boris Feld <boris.feld@octobus.net>
parents: 39333
diff changeset
639 gdelta = revlog._generaldelta
39492
a33f394b2bfd snapshot: try intermediate snapshot against parents' base
Boris Feld <boris.feld@octobus.net>
parents: 39488
diff changeset
640 sparse = revlog._sparserevlog
39334
507f5b1dd7c8 revlogdeltas: extract _getcandidaterevs in a function
Boris Feld <boris.feld@octobus.net>
parents: 39333
diff changeset
641 curr = len(revlog)
507f5b1dd7c8 revlogdeltas: extract _getcandidaterevs in a function
Boris Feld <boris.feld@octobus.net>
parents: 39333
diff changeset
642 prev = curr - 1
39492
a33f394b2bfd snapshot: try intermediate snapshot against parents' base
Boris Feld <boris.feld@octobus.net>
parents: 39488
diff changeset
643 deltachain = lambda rev: revlog._deltachain(rev)[0]
39334
507f5b1dd7c8 revlogdeltas: extract _getcandidaterevs in a function
Boris Feld <boris.feld@octobus.net>
parents: 39333
diff changeset
644
39336
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic
Boris Feld <boris.feld@octobus.net>
parents: 39335
diff changeset
645 if gdelta:
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic
Boris Feld <boris.feld@octobus.net>
parents: 39335
diff changeset
646 # exclude already lazy tested base if any
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic
Boris Feld <boris.feld@octobus.net>
parents: 39335
diff changeset
647 parents = [p for p in (p1, p2) if p != nullrev]
39334
507f5b1dd7c8 revlogdeltas: extract _getcandidaterevs in a function
Boris Feld <boris.feld@octobus.net>
parents: 39333
diff changeset
648
39336
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic
Boris Feld <boris.feld@octobus.net>
parents: 39335
diff changeset
649 if not revlog._deltabothparents and len(parents) == 2:
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic
Boris Feld <boris.feld@octobus.net>
parents: 39335
diff changeset
650 parents.sort()
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic
Boris Feld <boris.feld@octobus.net>
parents: 39335
diff changeset
651 # To minimize the chance of having to build a fulltext,
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic
Boris Feld <boris.feld@octobus.net>
parents: 39335
diff changeset
652 # pick first whichever parent is closest to us (max rev)
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic
Boris Feld <boris.feld@octobus.net>
parents: 39335
diff changeset
653 yield (parents[1],)
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic
Boris Feld <boris.feld@octobus.net>
parents: 39335
diff changeset
654 # then the other one (min rev) if the first did not fit
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic
Boris Feld <boris.feld@octobus.net>
parents: 39335
diff changeset
655 yield (parents[0],)
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic
Boris Feld <boris.feld@octobus.net>
parents: 39335
diff changeset
656 elif len(parents) > 0:
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic
Boris Feld <boris.feld@octobus.net>
parents: 39335
diff changeset
657 # Test all parents (1 or 2), and keep the best candidate
1c6ff52fe9cf revlogdeltas: split candidate groups selection from the filtering logic
Boris Feld <boris.feld@octobus.net>
parents: 39335
diff changeset
658 yield parents
39334
507f5b1dd7c8 revlogdeltas: extract _getcandidaterevs in a function
Boris Feld <boris.feld@octobus.net>
parents: 39333
diff changeset
659
39492
a33f394b2bfd snapshot: try intermediate snapshot against parents' base
Boris Feld <boris.feld@octobus.net>
parents: 39488
diff changeset
660 if sparse and parents:
39494
e72130f58f5d snapshot: consider all snapshots in the parents' chains
Boris Feld <boris.feld@octobus.net>
parents: 39493
diff changeset
661 snapshots = collections.defaultdict(list) # map: base-rev: snapshot-rev
39492
a33f394b2bfd snapshot: try intermediate snapshot against parents' base
Boris Feld <boris.feld@octobus.net>
parents: 39488
diff changeset
662 # See if we can use an existing snapshot in the parent chains to use as
a33f394b2bfd snapshot: try intermediate snapshot against parents' base
Boris Feld <boris.feld@octobus.net>
parents: 39488
diff changeset
663 # a base for a new intermediate-snapshot
39494
e72130f58f5d snapshot: consider all snapshots in the parents' chains
Boris Feld <boris.feld@octobus.net>
parents: 39493
diff changeset
664 #
e72130f58f5d snapshot: consider all snapshots in the parents' chains
Boris Feld <boris.feld@octobus.net>
parents: 39493
diff changeset
665 # search for snapshot in parents delta chain
e72130f58f5d snapshot: consider all snapshots in the parents' chains
Boris Feld <boris.feld@octobus.net>
parents: 39493
diff changeset
666 # map: snapshot-level: snapshot-rev
e72130f58f5d snapshot: consider all snapshots in the parents' chains
Boris Feld <boris.feld@octobus.net>
parents: 39493
diff changeset
667 parents_snaps = collections.defaultdict(set)
39504
05a165dc4f55 snapshot: extract parent chain computation
Boris Feld <boris.feld@octobus.net>
parents: 39503
diff changeset
668 candidate_chains = [deltachain(p) for p in parents]
05a165dc4f55 snapshot: extract parent chain computation
Boris Feld <boris.feld@octobus.net>
parents: 39503
diff changeset
669 for chain in candidate_chains:
05a165dc4f55 snapshot: extract parent chain computation
Boris Feld <boris.feld@octobus.net>
parents: 39503
diff changeset
670 for idx, s in enumerate(chain):
39494
e72130f58f5d snapshot: consider all snapshots in the parents' chains
Boris Feld <boris.feld@octobus.net>
parents: 39493
diff changeset
671 if not revlog.issnapshot(s):
e72130f58f5d snapshot: consider all snapshots in the parents' chains
Boris Feld <boris.feld@octobus.net>
parents: 39493
diff changeset
672 break
e72130f58f5d snapshot: consider all snapshots in the parents' chains
Boris Feld <boris.feld@octobus.net>
parents: 39493
diff changeset
673 parents_snaps[idx].add(s)
39495
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first
Boris Feld <boris.feld@octobus.net>
parents: 39494
diff changeset
674 snapfloor = min(parents_snaps[0]) + 1
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first
Boris Feld <boris.feld@octobus.net>
parents: 39494
diff changeset
675 _findsnapshots(revlog, snapshots, snapfloor)
39505
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision
Boris Feld <boris.feld@octobus.net>
parents: 39504
diff changeset
676 # search for the highest "unrelated" revision
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision
Boris Feld <boris.feld@octobus.net>
parents: 39504
diff changeset
677 #
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision
Boris Feld <boris.feld@octobus.net>
parents: 39504
diff changeset
678 # Adding snapshots used by "unrelated" revision increase the odd we
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision
Boris Feld <boris.feld@octobus.net>
parents: 39504
diff changeset
679 # reuse an independant, yet better snapshot chain.
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision
Boris Feld <boris.feld@octobus.net>
parents: 39504
diff changeset
680 #
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision
Boris Feld <boris.feld@octobus.net>
parents: 39504
diff changeset
681 # XXX instead of building a set of revisions, we could lazily enumerate
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision
Boris Feld <boris.feld@octobus.net>
parents: 39504
diff changeset
682 # over the chains. That would be more efficient, however we stick to
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision
Boris Feld <boris.feld@octobus.net>
parents: 39504
diff changeset
683 # simple code for now.
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision
Boris Feld <boris.feld@octobus.net>
parents: 39504
diff changeset
684 all_revs = set()
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision
Boris Feld <boris.feld@octobus.net>
parents: 39504
diff changeset
685 for chain in candidate_chains:
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision
Boris Feld <boris.feld@octobus.net>
parents: 39504
diff changeset
686 all_revs.update(chain)
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision
Boris Feld <boris.feld@octobus.net>
parents: 39504
diff changeset
687 other = None
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision
Boris Feld <boris.feld@octobus.net>
parents: 39504
diff changeset
688 for r in revlog.revs(prev, snapfloor):
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision
Boris Feld <boris.feld@octobus.net>
parents: 39504
diff changeset
689 if r not in all_revs:
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision
Boris Feld <boris.feld@octobus.net>
parents: 39504
diff changeset
690 other = r
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision
Boris Feld <boris.feld@octobus.net>
parents: 39504
diff changeset
691 break
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision
Boris Feld <boris.feld@octobus.net>
parents: 39504
diff changeset
692 if other is not None:
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision
Boris Feld <boris.feld@octobus.net>
parents: 39504
diff changeset
693 # To avoid unfair competition, we won't use unrelated intermediate
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision
Boris Feld <boris.feld@octobus.net>
parents: 39504
diff changeset
694 # snapshot that are deeper than the ones from the parent delta
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision
Boris Feld <boris.feld@octobus.net>
parents: 39504
diff changeset
695 # chain.
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision
Boris Feld <boris.feld@octobus.net>
parents: 39504
diff changeset
696 max_depth = max(parents_snaps.keys())
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision
Boris Feld <boris.feld@octobus.net>
parents: 39504
diff changeset
697 chain = deltachain(other)
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision
Boris Feld <boris.feld@octobus.net>
parents: 39504
diff changeset
698 for idx, s in enumerate(chain):
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision
Boris Feld <boris.feld@octobus.net>
parents: 39504
diff changeset
699 if s < snapfloor:
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision
Boris Feld <boris.feld@octobus.net>
parents: 39504
diff changeset
700 continue
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision
Boris Feld <boris.feld@octobus.net>
parents: 39504
diff changeset
701 if max_depth < idx:
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision
Boris Feld <boris.feld@octobus.net>
parents: 39504
diff changeset
702 break
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision
Boris Feld <boris.feld@octobus.net>
parents: 39504
diff changeset
703 if not revlog.issnapshot(s):
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision
Boris Feld <boris.feld@octobus.net>
parents: 39504
diff changeset
704 break
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision
Boris Feld <boris.feld@octobus.net>
parents: 39504
diff changeset
705 parents_snaps[idx].add(s)
39494
e72130f58f5d snapshot: consider all snapshots in the parents' chains
Boris Feld <boris.feld@octobus.net>
parents: 39493
diff changeset
706 # Test them as possible intermediate snapshot base
e72130f58f5d snapshot: consider all snapshots in the parents' chains
Boris Feld <boris.feld@octobus.net>
parents: 39493
diff changeset
707 # We test them from highest to lowest level. High level one are more
e72130f58f5d snapshot: consider all snapshots in the parents' chains
Boris Feld <boris.feld@octobus.net>
parents: 39493
diff changeset
708 # likely to result in small delta
39495
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first
Boris Feld <boris.feld@octobus.net>
parents: 39494
diff changeset
709 floor = None
39494
e72130f58f5d snapshot: consider all snapshots in the parents' chains
Boris Feld <boris.feld@octobus.net>
parents: 39493
diff changeset
710 for idx, snaps in sorted(parents_snaps.items(), reverse=True):
39495
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first
Boris Feld <boris.feld@octobus.net>
parents: 39494
diff changeset
711 siblings = set()
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first
Boris Feld <boris.feld@octobus.net>
parents: 39494
diff changeset
712 for s in snaps:
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first
Boris Feld <boris.feld@octobus.net>
parents: 39494
diff changeset
713 siblings.update(snapshots[s])
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first
Boris Feld <boris.feld@octobus.net>
parents: 39494
diff changeset
714 # Before considering making a new intermediate snapshot, we check
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first
Boris Feld <boris.feld@octobus.net>
parents: 39494
diff changeset
715 # if an existing snapshot, children of base we consider, would be
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first
Boris Feld <boris.feld@octobus.net>
parents: 39494
diff changeset
716 # suitable.
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first
Boris Feld <boris.feld@octobus.net>
parents: 39494
diff changeset
717 #
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first
Boris Feld <boris.feld@octobus.net>
parents: 39494
diff changeset
718 # It give a change to reuse a delta chain "unrelated" to the
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first
Boris Feld <boris.feld@octobus.net>
parents: 39494
diff changeset
719 # current revision instead of starting our own. Without such
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first
Boris Feld <boris.feld@octobus.net>
parents: 39494
diff changeset
720 # re-use, topological branches would keep reopening new chains.
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first
Boris Feld <boris.feld@octobus.net>
parents: 39494
diff changeset
721 # Creating more and more snapshot as the repository grow.
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first
Boris Feld <boris.feld@octobus.net>
parents: 39494
diff changeset
722
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first
Boris Feld <boris.feld@octobus.net>
parents: 39494
diff changeset
723 if floor is not None:
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first
Boris Feld <boris.feld@octobus.net>
parents: 39494
diff changeset
724 # We only do this for siblings created after the one in our
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first
Boris Feld <boris.feld@octobus.net>
parents: 39494
diff changeset
725 # parent's delta chain. Those created before has less chances
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first
Boris Feld <boris.feld@octobus.net>
parents: 39494
diff changeset
726 # to be valid base since our ancestors had to create a new
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first
Boris Feld <boris.feld@octobus.net>
parents: 39494
diff changeset
727 # snapshot.
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first
Boris Feld <boris.feld@octobus.net>
parents: 39494
diff changeset
728 siblings = [r for r in siblings if floor < r]
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first
Boris Feld <boris.feld@octobus.net>
parents: 39494
diff changeset
729 yield tuple(sorted(siblings))
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first
Boris Feld <boris.feld@octobus.net>
parents: 39494
diff changeset
730 # then test the base from our parent's delta chain.
39494
e72130f58f5d snapshot: consider all snapshots in the parents' chains
Boris Feld <boris.feld@octobus.net>
parents: 39493
diff changeset
731 yield tuple(sorted(snaps))
39495
6a53842727c1 snapshot: consider unrelated snapshots at a similar level first
Boris Feld <boris.feld@octobus.net>
parents: 39494
diff changeset
732 floor = min(snaps)
39493
3ca144f1c8dd snapshot: search for unrelated but reusable full-snapshot
Boris Feld <boris.feld@octobus.net>
parents: 39492
diff changeset
733 # No suitable base found in the parent chain, search if any full
3ca144f1c8dd snapshot: search for unrelated but reusable full-snapshot
Boris Feld <boris.feld@octobus.net>
parents: 39492
diff changeset
734 # snapshots emitted since parent's base would be a suitable base for an
3ca144f1c8dd snapshot: search for unrelated but reusable full-snapshot
Boris Feld <boris.feld@octobus.net>
parents: 39492
diff changeset
735 # intermediate snapshot.
3ca144f1c8dd snapshot: search for unrelated but reusable full-snapshot
Boris Feld <boris.feld@octobus.net>
parents: 39492
diff changeset
736 #
3ca144f1c8dd snapshot: search for unrelated but reusable full-snapshot
Boris Feld <boris.feld@octobus.net>
parents: 39492
diff changeset
737 # It give a chance to reuse a delta chain unrelated to the current
3ca144f1c8dd snapshot: search for unrelated but reusable full-snapshot
Boris Feld <boris.feld@octobus.net>
parents: 39492
diff changeset
738 # revisions instead of starting our own. Without such re-use,
3ca144f1c8dd snapshot: search for unrelated but reusable full-snapshot
Boris Feld <boris.feld@octobus.net>
parents: 39492
diff changeset
739 # topological branches would keep reopening new full chains. Creating
3ca144f1c8dd snapshot: search for unrelated but reusable full-snapshot
Boris Feld <boris.feld@octobus.net>
parents: 39492
diff changeset
740 # more and more snapshot as the repository grow.
3ca144f1c8dd snapshot: search for unrelated but reusable full-snapshot
Boris Feld <boris.feld@octobus.net>
parents: 39492
diff changeset
741 yield tuple(snapshots[nullrev])
39492
a33f394b2bfd snapshot: try intermediate snapshot against parents' base
Boris Feld <boris.feld@octobus.net>
parents: 39488
diff changeset
742
39505
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision
Boris Feld <boris.feld@octobus.net>
parents: 39504
diff changeset
743 if not sparse:
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision
Boris Feld <boris.feld@octobus.net>
parents: 39504
diff changeset
744 # other approach failed try against prev to hopefully save us a
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision
Boris Feld <boris.feld@octobus.net>
parents: 39504
diff changeset
745 # fulltext.
c6b8eab5db19 snapshot: also consider the snapshot chain of one unrelated revision
Boris Feld <boris.feld@octobus.net>
parents: 39504
diff changeset
746 yield (prev,)
39334
507f5b1dd7c8 revlogdeltas: extract _getcandidaterevs in a function
Boris Feld <boris.feld@octobus.net>
parents: 39333
diff changeset
747
39330
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
748 class deltacomputer(object):
35738
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
749 def __init__(self, revlog):
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
750 self.revlog = revlog
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
751
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
752 def buildtext(self, revinfo, fh):
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
753 """Builds a fulltext version of a revision
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
754
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
755 revinfo: _revisioninfo instance that contains all needed info
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
756 fh: file handle to either the .i or the .d revlog file,
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
757 depending on whether it is inlined or not
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
758 """
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
759 btext = revinfo.btext
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
760 if btext[0] is not None:
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
761 return btext[0]
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
762
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
763 revlog = self.revlog
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
764 cachedelta = revinfo.cachedelta
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
765 baserev = cachedelta[0]
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
766 delta = cachedelta[1]
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
767
39331
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function
Boris Feld <boris.feld@octobus.net>
parents: 39330
diff changeset
768 fulltext = btext[0] = _textfromdelta(fh, revlog, baserev, delta,
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function
Boris Feld <boris.feld@octobus.net>
parents: 39330
diff changeset
769 revinfo.p1, revinfo.p2,
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function
Boris Feld <boris.feld@octobus.net>
parents: 39330
diff changeset
770 revinfo.flags, revinfo.node)
fd0150a3c2fe revlogdeltas: extra fulltext building in its own function
Boris Feld <boris.feld@octobus.net>
parents: 39330
diff changeset
771 return fulltext
35738
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
772
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
773 def _builddeltadiff(self, base, revinfo, fh):
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
774 revlog = self.revlog
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
775 t = self.buildtext(revinfo, fh)
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
776 if revlog.iscensored(base):
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
777 # deltas based on a censored revision must replace the
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
778 # full content in one patch, so delta works everywhere
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
779 header = mdiff.replacediffheader(revlog.rawsize(base), len(t))
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
780 delta = header + t
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
781 else:
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
782 ptext = revlog.revision(base, _df=fh, raw=True)
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
783 delta = mdiff.textdiff(ptext, t)
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
784
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
785 return delta
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
786
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
787 def _builddeltainfo(self, revinfo, base, fh):
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
788 # can we use the cached delta?
39595
a911932d5003 revlog: reuse cached delta for identical base revision (issue5975)
Boris Feld <boris.feld@octobus.net>
parents: 39594
diff changeset
789 delta = None
a911932d5003 revlog: reuse cached delta for identical base revision (issue5975)
Boris Feld <boris.feld@octobus.net>
parents: 39594
diff changeset
790 if revinfo.cachedelta:
a911932d5003 revlog: reuse cached delta for identical base revision (issue5975)
Boris Feld <boris.feld@octobus.net>
parents: 39594
diff changeset
791 cachebase, cachediff = revinfo.cachedelta
a911932d5003 revlog: reuse cached delta for identical base revision (issue5975)
Boris Feld <boris.feld@octobus.net>
parents: 39594
diff changeset
792 #check if the diff still apply
a911932d5003 revlog: reuse cached delta for identical base revision (issue5975)
Boris Feld <boris.feld@octobus.net>
parents: 39594
diff changeset
793 currentbase = cachebase
a911932d5003 revlog: reuse cached delta for identical base revision (issue5975)
Boris Feld <boris.feld@octobus.net>
parents: 39594
diff changeset
794 while (currentbase != nullrev
a911932d5003 revlog: reuse cached delta for identical base revision (issue5975)
Boris Feld <boris.feld@octobus.net>
parents: 39594
diff changeset
795 and currentbase != base
a911932d5003 revlog: reuse cached delta for identical base revision (issue5975)
Boris Feld <boris.feld@octobus.net>
parents: 39594
diff changeset
796 and self.revlog.length(currentbase) == 0):
a911932d5003 revlog: reuse cached delta for identical base revision (issue5975)
Boris Feld <boris.feld@octobus.net>
parents: 39594
diff changeset
797 currentbase = self.revlog.deltaparent(currentbase)
a911932d5003 revlog: reuse cached delta for identical base revision (issue5975)
Boris Feld <boris.feld@octobus.net>
parents: 39594
diff changeset
798 if currentbase == base:
a911932d5003 revlog: reuse cached delta for identical base revision (issue5975)
Boris Feld <boris.feld@octobus.net>
parents: 39594
diff changeset
799 delta = revinfo.cachedelta[1]
a911932d5003 revlog: reuse cached delta for identical base revision (issue5975)
Boris Feld <boris.feld@octobus.net>
parents: 39594
diff changeset
800 if delta is None:
35738
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
801 delta = self._builddeltadiff(base, revinfo, fh)
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
802 revlog = self.revlog
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
803 header, data = revlog.compress(delta)
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
804 deltalen = len(header) + len(data)
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
805 chainbase = revlog.chainbase(base)
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
806 offset = revlog.end(len(revlog) - 1)
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
807 dist = deltalen + offset - revlog.start(chainbase)
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
808 if revlog._generaldelta:
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
809 deltabase = base
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
810 else:
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
811 deltabase = chainbase
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
812 chainlen, compresseddeltalen = revlog._chaininfo(base)
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
813 chainlen += 1
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
814 compresseddeltalen += deltalen
39154
e0da43e2f71f revlog: compute snapshot depth on delta info
Boris Feld <boris.feld@octobus.net>
parents: 39152
diff changeset
815
e0da43e2f71f revlog: compute snapshot depth on delta info
Boris Feld <boris.feld@octobus.net>
parents: 39152
diff changeset
816 revlog = self.revlog
e0da43e2f71f revlog: compute snapshot depth on delta info
Boris Feld <boris.feld@octobus.net>
parents: 39152
diff changeset
817 snapshotdepth = None
e0da43e2f71f revlog: compute snapshot depth on delta info
Boris Feld <boris.feld@octobus.net>
parents: 39152
diff changeset
818 if deltabase == nullrev:
e0da43e2f71f revlog: compute snapshot depth on delta info
Boris Feld <boris.feld@octobus.net>
parents: 39152
diff changeset
819 snapshotdepth = 0
e0da43e2f71f revlog: compute snapshot depth on delta info
Boris Feld <boris.feld@octobus.net>
parents: 39152
diff changeset
820 elif revlog._sparserevlog and revlog.issnapshot(deltabase):
e0da43e2f71f revlog: compute snapshot depth on delta info
Boris Feld <boris.feld@octobus.net>
parents: 39152
diff changeset
821 # A delta chain should always be one full snapshot,
e0da43e2f71f revlog: compute snapshot depth on delta info
Boris Feld <boris.feld@octobus.net>
parents: 39152
diff changeset
822 # zero or more semi-snapshots, and zero or more deltas
e0da43e2f71f revlog: compute snapshot depth on delta info
Boris Feld <boris.feld@octobus.net>
parents: 39152
diff changeset
823 p1, p2 = revlog.rev(revinfo.p1), revlog.rev(revinfo.p2)
e0da43e2f71f revlog: compute snapshot depth on delta info
Boris Feld <boris.feld@octobus.net>
parents: 39152
diff changeset
824 if deltabase not in (p1, p2) and revlog.issnapshot(deltabase):
e0da43e2f71f revlog: compute snapshot depth on delta info
Boris Feld <boris.feld@octobus.net>
parents: 39152
diff changeset
825 snapshotdepth = len(revlog._deltachain(deltabase)[0])
e0da43e2f71f revlog: compute snapshot depth on delta info
Boris Feld <boris.feld@octobus.net>
parents: 39152
diff changeset
826
35738
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
827 return _deltainfo(dist, deltalen, (header, data), deltabase,
39154
e0da43e2f71f revlog: compute snapshot depth on delta info
Boris Feld <boris.feld@octobus.net>
parents: 39152
diff changeset
828 chainbase, chainlen, compresseddeltalen,
e0da43e2f71f revlog: compute snapshot depth on delta info
Boris Feld <boris.feld@octobus.net>
parents: 39152
diff changeset
829 snapshotdepth)
35738
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
830
39333
5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo
Boris Feld <boris.feld@octobus.net>
parents: 39332
diff changeset
831 def _fullsnapshotinfo(self, fh, revinfo):
5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo
Boris Feld <boris.feld@octobus.net>
parents: 39332
diff changeset
832 curr = len(self.revlog)
5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo
Boris Feld <boris.feld@octobus.net>
parents: 39332
diff changeset
833 rawtext = self.buildtext(revinfo, fh)
5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo
Boris Feld <boris.feld@octobus.net>
parents: 39332
diff changeset
834 data = self.revlog.compress(rawtext)
5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo
Boris Feld <boris.feld@octobus.net>
parents: 39332
diff changeset
835 compresseddeltalen = deltalen = dist = len(data[1]) + len(data[0])
5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo
Boris Feld <boris.feld@octobus.net>
parents: 39332
diff changeset
836 deltabase = chainbase = curr
5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo
Boris Feld <boris.feld@octobus.net>
parents: 39332
diff changeset
837 snapshotdepth = 0
5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo
Boris Feld <boris.feld@octobus.net>
parents: 39332
diff changeset
838 chainlen = 1
5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo
Boris Feld <boris.feld@octobus.net>
parents: 39332
diff changeset
839
5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo
Boris Feld <boris.feld@octobus.net>
parents: 39332
diff changeset
840 return _deltainfo(dist, deltalen, data, deltabase,
5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo
Boris Feld <boris.feld@octobus.net>
parents: 39332
diff changeset
841 chainbase, chainlen, compresseddeltalen,
5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo
Boris Feld <boris.feld@octobus.net>
parents: 39332
diff changeset
842 snapshotdepth)
5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo
Boris Feld <boris.feld@octobus.net>
parents: 39332
diff changeset
843
35738
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
844 def finddeltainfo(self, revinfo, fh):
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
845 """Find an acceptable delta against a candidate revision
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
846
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
847 revinfo: information about the revision (instance of _revisioninfo)
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
848 fh: file handle to either the .i or the .d revlog file,
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
849 depending on whether it is inlined or not
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
850
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
851 Returns the first acceptable candidate revision, as ordered by
39334
507f5b1dd7c8 revlogdeltas: extract _getcandidaterevs in a function
Boris Feld <boris.feld@octobus.net>
parents: 39333
diff changeset
852 _candidategroups
39333
5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo
Boris Feld <boris.feld@octobus.net>
parents: 39332
diff changeset
853
5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo
Boris Feld <boris.feld@octobus.net>
parents: 39332
diff changeset
854 If no suitable deltabase is found, we return delta info for a full
5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo
Boris Feld <boris.feld@octobus.net>
parents: 39332
diff changeset
855 snapshot.
35738
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
856 """
39085
dbb3e9e44fce revlog: do not search for delta for empty content
Boris Feld <boris.feld@octobus.net>
parents: 39084
diff changeset
857 if not revinfo.textlen:
39333
5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo
Boris Feld <boris.feld@octobus.net>
parents: 39332
diff changeset
858 return self._fullsnapshotinfo(fh, revinfo)
39085
dbb3e9e44fce revlog: do not search for delta for empty content
Boris Feld <boris.feld@octobus.net>
parents: 39084
diff changeset
859
39332
6f4b8f607a31 revlogdeltas: move special cases around raw revisions in finddeltainfo
Boris Feld <boris.feld@octobus.net>
parents: 39331
diff changeset
860 # no delta for flag processor revision (see "candelta" for why)
6f4b8f607a31 revlogdeltas: move special cases around raw revisions in finddeltainfo
Boris Feld <boris.feld@octobus.net>
parents: 39331
diff changeset
861 # not calling candelta since only one revision needs test, also to
6f4b8f607a31 revlogdeltas: move special cases around raw revisions in finddeltainfo
Boris Feld <boris.feld@octobus.net>
parents: 39331
diff changeset
862 # avoid overhead fetching flags again.
6f4b8f607a31 revlogdeltas: move special cases around raw revisions in finddeltainfo
Boris Feld <boris.feld@octobus.net>
parents: 39331
diff changeset
863 if revinfo.flags & REVIDX_RAWTEXT_CHANGING_FLAGS:
39333
5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo
Boris Feld <boris.feld@octobus.net>
parents: 39332
diff changeset
864 return self._fullsnapshotinfo(fh, revinfo)
39332
6f4b8f607a31 revlogdeltas: move special cases around raw revisions in finddeltainfo
Boris Feld <boris.feld@octobus.net>
parents: 39331
diff changeset
865
35738
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
866 cachedelta = revinfo.cachedelta
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
867 p1 = revinfo.p1
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
868 p2 = revinfo.p2
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
869 revlog = self.revlog
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
870
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
871 deltainfo = None
39335
1441eb38849f revlogdeltas: pass revision number to _candidatesgroups
Boris Feld <boris.feld@octobus.net>
parents: 39334
diff changeset
872 p1r, p2r = revlog.rev(p1), revlog.rev(p2)
39337
37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups
Boris Feld <boris.feld@octobus.net>
parents: 39336
diff changeset
873 groups = _candidategroups(self.revlog, revinfo.textlen,
37957e07138c revlogdeltas: move finddeltainfo filtering inside _candidategroups
Boris Feld <boris.feld@octobus.net>
parents: 39336
diff changeset
874 p1r, p2r, cachedelta)
39497
5b308a4e6d03 snapshot: use None as a stop value when looking for a good delta
Boris Feld <boris.feld@octobus.net>
parents: 39496
diff changeset
875 candidaterevs = next(groups)
5b308a4e6d03 snapshot: use None as a stop value when looking for a good delta
Boris Feld <boris.feld@octobus.net>
parents: 39496
diff changeset
876 while candidaterevs is not None:
35738
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
877 nominateddeltas = []
39498
04b75f3a3f2a snapshot: add refining logic at the findeltainfo level
Boris Feld <boris.feld@octobus.net>
parents: 39497
diff changeset
878 if deltainfo is not None:
04b75f3a3f2a snapshot: add refining logic at the findeltainfo level
Boris Feld <boris.feld@octobus.net>
parents: 39497
diff changeset
879 # if we already found a good delta,
04b75f3a3f2a snapshot: add refining logic at the findeltainfo level
Boris Feld <boris.feld@octobus.net>
parents: 39497
diff changeset
880 # challenge it against refined candidates
04b75f3a3f2a snapshot: add refining logic at the findeltainfo level
Boris Feld <boris.feld@octobus.net>
parents: 39497
diff changeset
881 nominateddeltas.append(deltainfo)
35738
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
882 for candidaterev in candidaterevs:
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
883 candidatedelta = self._builddeltainfo(revinfo, candidaterev, fh)
39330
655b5b465953 revlog: split functionality related to deltas computation in a new module
Boris Feld <boris.feld@octobus.net>
parents: 39329
diff changeset
884 if isgooddeltainfo(self.revlog, candidatedelta, revinfo):
35738
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
885 nominateddeltas.append(candidatedelta)
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
886 if nominateddeltas:
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
887 deltainfo = min(nominateddeltas, key=lambda x: x.deltalen)
39498
04b75f3a3f2a snapshot: add refining logic at the findeltainfo level
Boris Feld <boris.feld@octobus.net>
parents: 39497
diff changeset
888 if deltainfo is not None:
04b75f3a3f2a snapshot: add refining logic at the findeltainfo level
Boris Feld <boris.feld@octobus.net>
parents: 39497
diff changeset
889 candidaterevs = groups.send(deltainfo.base)
04b75f3a3f2a snapshot: add refining logic at the findeltainfo level
Boris Feld <boris.feld@octobus.net>
parents: 39497
diff changeset
890 else:
04b75f3a3f2a snapshot: add refining logic at the findeltainfo level
Boris Feld <boris.feld@octobus.net>
parents: 39497
diff changeset
891 candidaterevs = next(groups)
35738
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
892
39333
5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo
Boris Feld <boris.feld@octobus.net>
parents: 39332
diff changeset
893 if deltainfo is None:
5d343a24bff5 revlogdeltas: always return a delta info object in finddeltainfo
Boris Feld <boris.feld@octobus.net>
parents: 39332
diff changeset
894 deltainfo = self._fullsnapshotinfo(fh, revinfo)
35738
f90f6fd130c1 revlog: group delta computation methods under _deltacomputer object
Paul Morelle <paul.morelle@octobus.net>
parents: 35737
diff changeset
895 return deltainfo