Mercurial > public > mercurial-scm > hg-stable

annotate mercurial/revlogutils/sidedata.py @ 48575:e91aa800ae5b

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
rhg: desambiguate status without decompressing filelog if possible When status is unsure based on `stat()` and the dirstate if a file is clean or modified, we need to compare it against the filelog. This comparison can skip looking at contents if the lengths differ. This changeset optimize this further to deduce what we can about the length if the filelog without decompressing it or resolving deltas. Differential Revision: https://phab.mercurial-scm.org/D11965
author Simon Sapin <simon.sapin@octobus.net>
date Fri, 07 Jan 2022 14:40:21 +0100
parents 8bd769b5c941
children 6000f5b25c9b
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
43033
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
1 # sidedata.py - Logic around store extra data alongside revlog revisions
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
2 #
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
3 # Copyright 2019 Pierre-Yves David <pierre-yves.david@octobus.net)
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
4 #
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
5 # This software may be used and distributed according to the terms of the
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
6 # GNU General Public License version 2 or any later version.
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
7 """core code for "sidedata" support
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
8
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
9 The "sidedata" are stored alongside the revision without actually being part of
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
10 its content and not affecting its hash. It's main use cases is to cache
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
11 important information related to a changesets.
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
12
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
13 The current implementation is experimental and subject to changes. Do not rely
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
14 on it in production.
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
15
46722
3d740058b467 sidedata: move to new sidedata storage in revlogv2
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46195
diff changeset
16 Sidedata are stored in the revlog itself, thanks to a new version of the
3d740058b467 sidedata: move to new sidedata storage in revlogv2
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46195
diff changeset
17 revlog. The following format is currently used::
43033
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
18
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
19 initial header:
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
20 <number of sidedata; 2 bytes>
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
21 sidedata (repeated N times):
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
22 <sidedata-key; 2 bytes>
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
23 <sidedata-entry-length: 4 bytes>
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
24 <sidedata-content-sha1-digest: 20 bytes>
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
25 <sidedata-content; X bytes>
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
26 normal raw text:
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
27 <all bytes remaining in the rawtext>
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
28
46195
d6a9e690d620 comments: fix typos
Joerg Sonnenberger <joerg@bec.de>
parents: 45636
diff changeset
29 This is a simple and effective format. It should be enough to experiment with
43033
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
30 the concept.
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
31 """
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
32
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
33 from __future__ import absolute_import
43034
294afb982a88 sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43033
diff changeset
34
47099
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46722
diff changeset
35 import collections
43034
294afb982a88 sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43033
diff changeset
36 import struct
294afb982a88 sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43033
diff changeset
37
47099
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46722
diff changeset
38 from .. import error, requirements as requirementsmod
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46722
diff changeset
39 from ..revlogutils import constants, flagutil
44060
a61287a95dc3 core: migrate uses of hashlib.sha1 to hashutil.sha1
Augie Fackler <augie@google.com>
parents: 43554
diff changeset
40 from ..utils import hashutil
43034
294afb982a88 sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43033
diff changeset
41
43040
ba4072c0a911 sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43037
diff changeset
42 ## sidedata type constant
ba4072c0a911 sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43037
diff changeset
43 # reserve a block for testing purposes.
ba4072c0a911 sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43037
diff changeset
44 SD_TEST1 = 1
ba4072c0a911 sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43037
diff changeset
45 SD_TEST2 = 2
ba4072c0a911 sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43037
diff changeset
46 SD_TEST3 = 3
ba4072c0a911 sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43037
diff changeset
47 SD_TEST4 = 4
ba4072c0a911 sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43037
diff changeset
48 SD_TEST5 = 5
ba4072c0a911 sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43037
diff changeset
49 SD_TEST6 = 6
ba4072c0a911 sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43037
diff changeset
50 SD_TEST7 = 7
ba4072c0a911 sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43037
diff changeset
51
43142
beed7ce61681 sidedatacopies: write copies information in sidedata when applicable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43077
diff changeset
52 # key to store copies related information
beed7ce61681 sidedatacopies: write copies information in sidedata when applicable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43077
diff changeset
53 SD_P1COPIES = 8
beed7ce61681 sidedatacopies: write copies information in sidedata when applicable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43077
diff changeset
54 SD_P2COPIES = 9
beed7ce61681 sidedatacopies: write copies information in sidedata when applicable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43077
diff changeset
55 SD_FILESADDED = 10
beed7ce61681 sidedatacopies: write copies information in sidedata when applicable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43077
diff changeset
56 SD_FILESREMOVED = 11
45636
9a6b409b8ebc changing-files: rework the way we store changed files in side-data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44060
diff changeset
57 SD_FILES = 12
43142
beed7ce61681 sidedatacopies: write copies information in sidedata when applicable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43077
diff changeset
58
43040
ba4072c0a911 sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43037
diff changeset
59 # internal format constant
43554
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43142
diff changeset
60 SIDEDATA_HEADER = struct.Struct('>H')
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43142
diff changeset
61 SIDEDATA_ENTRY = struct.Struct('>HL20s')
43034
294afb982a88 sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43033
diff changeset
62
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43048
diff changeset
63
46722
3d740058b467 sidedata: move to new sidedata storage in revlogv2
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46195
diff changeset
64 def serialize_sidedata(sidedata):
43035
ea83abf95630 sidedata: add a function to write sidedata into a raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43034
diff changeset
65 sidedata = list(sidedata.items())
ea83abf95630 sidedata: add a function to write sidedata into a raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43034
diff changeset
66 sidedata.sort()
46722
3d740058b467 sidedata: move to new sidedata storage in revlogv2
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46195
diff changeset
67 buf = [SIDEDATA_HEADER.pack(len(sidedata))]
43035
ea83abf95630 sidedata: add a function to write sidedata into a raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43034
diff changeset
68 for key, value in sidedata:
44060
a61287a95dc3 core: migrate uses of hashlib.sha1 to hashutil.sha1
Augie Fackler <augie@google.com>
parents: 43554
diff changeset
69 digest = hashutil.sha1(value).digest()
46722
3d740058b467 sidedata: move to new sidedata storage in revlogv2
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46195
diff changeset
70 buf.append(SIDEDATA_ENTRY.pack(key, len(value), digest))
43035
ea83abf95630 sidedata: add a function to write sidedata into a raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43034
diff changeset
71 for key, value in sidedata:
46722
3d740058b467 sidedata: move to new sidedata storage in revlogv2
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46195
diff changeset
72 buf.append(value)
3d740058b467 sidedata: move to new sidedata storage in revlogv2
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46195
diff changeset
73 buf = b''.join(buf)
3d740058b467 sidedata: move to new sidedata storage in revlogv2
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46195
diff changeset
74 return buf
43035
ea83abf95630 sidedata: add a function to write sidedata into a raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43034
diff changeset
75
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43048
diff changeset
76
46722
3d740058b467 sidedata: move to new sidedata storage in revlogv2
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46195
diff changeset
77 def deserialize_sidedata(blob):
43034
294afb982a88 sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43033
diff changeset
78 sidedata = {}
294afb982a88 sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43033
diff changeset
79 offset = 0
46722
3d740058b467 sidedata: move to new sidedata storage in revlogv2
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46195
diff changeset
80 (nbentry,) = SIDEDATA_HEADER.unpack(blob[: SIDEDATA_HEADER.size])
43034
294afb982a88 sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43033
diff changeset
81 offset += SIDEDATA_HEADER.size
294afb982a88 sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43033
diff changeset
82 dataoffset = SIDEDATA_HEADER.size + (SIDEDATA_ENTRY.size * nbentry)
294afb982a88 sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43033
diff changeset
83 for i in range(nbentry):
294afb982a88 sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43033
diff changeset
84 nextoffset = offset + SIDEDATA_ENTRY.size
46722
3d740058b467 sidedata: move to new sidedata storage in revlogv2
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46195
diff changeset
85 key, size, storeddigest = SIDEDATA_ENTRY.unpack(blob[offset:nextoffset])
43034
294afb982a88 sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43033
diff changeset
86 offset = nextoffset
294afb982a88 sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43033
diff changeset
87 # read the data associated with that entry
294afb982a88 sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43033
diff changeset
88 nextdataoffset = dataoffset + size
46722
3d740058b467 sidedata: move to new sidedata storage in revlogv2
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46195
diff changeset
89 entrytext = bytes(blob[dataoffset:nextdataoffset])
44060
a61287a95dc3 core: migrate uses of hashlib.sha1 to hashutil.sha1
Augie Fackler <augie@google.com>
parents: 43554
diff changeset
90 readdigest = hashutil.sha1(entrytext).digest()
43034
294afb982a88 sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43033
diff changeset
91 if storeddigest != readdigest:
294afb982a88 sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43033
diff changeset
92 raise error.SidedataHashError(key, storeddigest, readdigest)
294afb982a88 sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43033
diff changeset
93 sidedata[key] = entrytext
294afb982a88 sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43033
diff changeset
94 dataoffset = nextdataoffset
46722
3d740058b467 sidedata: move to new sidedata storage in revlogv2
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46195
diff changeset
95 return sidedata
47099
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46722
diff changeset
96
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46722
diff changeset
97
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46722
diff changeset
98 def get_sidedata_helpers(repo, remote_sd_categories, pull=False):
47100
8bd769b5c941 sidedata: move documentation about sidedata helpers to sidedata module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 47099
diff changeset
99 """
8bd769b5c941 sidedata: move documentation about sidedata helpers to sidedata module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 47099
diff changeset
100 Returns a dictionary mapping revlog types to tuples of
8bd769b5c941 sidedata: move documentation about sidedata helpers to sidedata module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 47099
diff changeset
101 `(repo, computers, removers)`:
8bd769b5c941 sidedata: move documentation about sidedata helpers to sidedata module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 47099
diff changeset
102 * `repo` is used as an argument for computers
8bd769b5c941 sidedata: move documentation about sidedata helpers to sidedata module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 47099
diff changeset
103 * `computers` is a list of `(category, (keys, computer, flags)` that
8bd769b5c941 sidedata: move documentation about sidedata helpers to sidedata module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 47099
diff changeset
104 compute the missing sidedata categories that were asked:
8bd769b5c941 sidedata: move documentation about sidedata helpers to sidedata module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 47099
diff changeset
105 * `category` is the sidedata category
8bd769b5c941 sidedata: move documentation about sidedata helpers to sidedata module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 47099
diff changeset
106 * `keys` are the sidedata keys to be affected
8bd769b5c941 sidedata: move documentation about sidedata helpers to sidedata module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 47099
diff changeset
107 * `flags` is a bitmask (an integer) of flags to remove when
8bd769b5c941 sidedata: move documentation about sidedata helpers to sidedata module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 47099
diff changeset
108 removing the category.
8bd769b5c941 sidedata: move documentation about sidedata helpers to sidedata module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 47099
diff changeset
109 * `computer` is the function `(repo, store, rev, sidedata)` that
8bd769b5c941 sidedata: move documentation about sidedata helpers to sidedata module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 47099
diff changeset
110 returns a tuple of
8bd769b5c941 sidedata: move documentation about sidedata helpers to sidedata module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 47099
diff changeset
111 `(new sidedata dict, (flags to add, flags to remove))`.
8bd769b5c941 sidedata: move documentation about sidedata helpers to sidedata module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 47099
diff changeset
112 For example, it will return `({}, (0, 1 << 15))` to return no
8bd769b5c941 sidedata: move documentation about sidedata helpers to sidedata module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 47099
diff changeset
113 sidedata, with no flags to add and one flag to remove.
8bd769b5c941 sidedata: move documentation about sidedata helpers to sidedata module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 47099
diff changeset
114 * `removers` will remove the keys corresponding to the categories
8bd769b5c941 sidedata: move documentation about sidedata helpers to sidedata module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 47099
diff changeset
115 that are present, but not needed.
8bd769b5c941 sidedata: move documentation about sidedata helpers to sidedata module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 47099
diff changeset
116 If both `computers` and `removers` are empty, sidedata will simply not
8bd769b5c941 sidedata: move documentation about sidedata helpers to sidedata module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 47099
diff changeset
117 be transformed.
8bd769b5c941 sidedata: move documentation about sidedata helpers to sidedata module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 47099
diff changeset
118 """
47099
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46722
diff changeset
119 # Computers for computing sidedata on-the-fly
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46722
diff changeset
120 sd_computers = collections.defaultdict(list)
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46722
diff changeset
121 # Computers for categories to remove from sidedata
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46722
diff changeset
122 sd_removers = collections.defaultdict(list)
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46722
diff changeset
123 to_generate = remote_sd_categories - repo._wanted_sidedata
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46722
diff changeset
124 to_remove = repo._wanted_sidedata - remote_sd_categories
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46722
diff changeset
125 if pull:
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46722
diff changeset
126 to_generate, to_remove = to_remove, to_generate
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46722
diff changeset
127
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46722
diff changeset
128 for revlog_kind, computers in repo._sidedata_computers.items():
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46722
diff changeset
129 for category, computer in computers.items():
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46722
diff changeset
130 if category in to_generate:
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46722
diff changeset
131 sd_computers[revlog_kind].append(computer)
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46722
diff changeset
132 if category in to_remove:
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46722
diff changeset
133 sd_removers[revlog_kind].append(computer)
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46722
diff changeset
134
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46722
diff changeset
135 sidedata_helpers = (repo, sd_computers, sd_removers)
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46722
diff changeset
136 return sidedata_helpers
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46722
diff changeset
137
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46722
diff changeset
138
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46722
diff changeset
139 def run_sidedata_helpers(store, sidedata_helpers, sidedata, rev):
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46722
diff changeset
140 """Returns the sidedata for the given revision after running through
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46722
diff changeset
141 the given helpers.
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46722
diff changeset
142 - `store`: the revlog this applies to (changelog, manifest, or filelog
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46722
diff changeset
143 instance)
47100
8bd769b5c941 sidedata: move documentation about sidedata helpers to sidedata module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 47099
diff changeset
144 - `sidedata_helpers`: see `get_sidedata_helpers`
47099
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46722
diff changeset
145 - `sidedata`: previous sidedata at the given rev, if any
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46722
diff changeset
146 - `rev`: affected rev of `store`
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46722
diff changeset
147 """
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46722
diff changeset
148 repo, sd_computers, sd_removers = sidedata_helpers
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46722
diff changeset
149 kind = store.revlog_kind
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46722
diff changeset
150 flags_to_add = 0
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46722
diff changeset
151 flags_to_remove = 0
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46722
diff changeset
152 for _keys, sd_computer, _flags in sd_computers.get(kind, []):
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46722
diff changeset
153 sidedata, flags = sd_computer(repo, store, rev, sidedata)
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46722
diff changeset
154 flags_to_add |= flags[0]
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46722
diff changeset
155 flags_to_remove |= flags[1]
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46722
diff changeset
156 for keys, _computer, flags in sd_removers.get(kind, []):
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46722
diff changeset
157 for key in keys:
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46722
diff changeset
158 sidedata.pop(key, None)
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46722
diff changeset
159 flags_to_remove |= flags
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46722
diff changeset
160 return sidedata, (flags_to_add, flags_to_remove)
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46722
diff changeset
161
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46722
diff changeset
162
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46722
diff changeset
163 def set_sidedata_spec_for_repo(repo):
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46722
diff changeset
164 # prevent cycle metadata -> revlogutils.sidedata -> metadata
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46722
diff changeset
165 from .. import metadata
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46722
diff changeset
166
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46722
diff changeset
167 if requirementsmod.COPIESSDC_REQUIREMENT in repo.requirements:
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46722
diff changeset
168 repo.register_wanted_sidedata(SD_FILES)
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46722
diff changeset
169 repo.register_sidedata_computer(
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46722
diff changeset
170 constants.KIND_CHANGELOG,
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46722
diff changeset
171 SD_FILES,
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46722
diff changeset
172 (SD_FILES,),
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46722
diff changeset
173 metadata.copies_sidedata_computer,
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46722
diff changeset
174 flagutil.REVIDX_HASCOPIESINFO,
3aab2330b7d3 sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46722
diff changeset
175 )