Mercurial > public > mercurial-scm > hg
annotate mercurial/revlogutils/sidedata.py @ 51528:f85f23f1479b
branchcache: skip entries that are topological heads in the on disk file
In the majority of cases, topological heads are also branch heads. We have
efficient way to get the topological heads and efficient way to retrieve
their branch information. So there is little value in putting them in the branch
cache file explicitly. On the contrary, writing them explicitly tend to create
very large cache file that are inefficient to read and update.
So the branch cache v3 format is no longer including them. This changeset focus
on the format aspect and have no focus on the performance aspect. We will cover
that later.
author | Pierre-Yves David <pierre-yves.david@octobus.net> |
---|---|
date | Thu, 07 Mar 2024 10:55:22 +0100 |
parents | 6000f5b25c9b |
children | f4733654f144 |
rev | line source |
---|---|
43033
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
1 # sidedata.py - Logic around store extra data alongside revlog revisions |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
2 # |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
3 # Copyright 2019 Pierre-Yves David <pierre-yves.david@octobus.net) |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
4 # |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
5 # This software may be used and distributed according to the terms of the |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
6 # GNU General Public License version 2 or any later version. |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
7 """core code for "sidedata" support |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
8 |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
9 The "sidedata" are stored alongside the revision without actually being part of |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
10 its content and not affecting its hash. It's main use cases is to cache |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
11 important information related to a changesets. |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
12 |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
13 The current implementation is experimental and subject to changes. Do not rely |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
14 on it in production. |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
15 |
46709
3d740058b467
sidedata: move to new sidedata storage in revlogv2
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46195
diff
changeset
|
16 Sidedata are stored in the revlog itself, thanks to a new version of the |
3d740058b467
sidedata: move to new sidedata storage in revlogv2
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46195
diff
changeset
|
17 revlog. The following format is currently used:: |
43033
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
18 |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
19 initial header: |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
20 <number of sidedata; 2 bytes> |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
21 sidedata (repeated N times): |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
22 <sidedata-key; 2 bytes> |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
23 <sidedata-entry-length: 4 bytes> |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
24 <sidedata-content-sha1-digest: 20 bytes> |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
25 <sidedata-content; X bytes> |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
26 normal raw text: |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
27 <all bytes remaining in the rawtext> |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
28 |
46195 | 29 This is a simple and effective format. It should be enough to experiment with |
43033
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
30 the concept. |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
31 """ |
21025a4107d4
sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff
changeset
|
32 |
43034
294afb982a88
sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43033
diff
changeset
|
33 |
47085
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46709
diff
changeset
|
34 import collections |
43034
294afb982a88
sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43033
diff
changeset
|
35 import struct |
294afb982a88
sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43033
diff
changeset
|
36 |
47085
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46709
diff
changeset
|
37 from .. import error, requirements as requirementsmod |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46709
diff
changeset
|
38 from ..revlogutils import constants, flagutil |
44060
a61287a95dc3
core: migrate uses of hashlib.sha1 to hashutil.sha1
Augie Fackler <augie@google.com>
parents:
43506
diff
changeset
|
39 from ..utils import hashutil |
43034
294afb982a88
sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43033
diff
changeset
|
40 |
43040
ba4072c0a911
sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43037
diff
changeset
|
41 ## sidedata type constant |
ba4072c0a911
sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43037
diff
changeset
|
42 # reserve a block for testing purposes. |
ba4072c0a911
sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43037
diff
changeset
|
43 SD_TEST1 = 1 |
ba4072c0a911
sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43037
diff
changeset
|
44 SD_TEST2 = 2 |
ba4072c0a911
sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43037
diff
changeset
|
45 SD_TEST3 = 3 |
ba4072c0a911
sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43037
diff
changeset
|
46 SD_TEST4 = 4 |
ba4072c0a911
sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43037
diff
changeset
|
47 SD_TEST5 = 5 |
ba4072c0a911
sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43037
diff
changeset
|
48 SD_TEST6 = 6 |
ba4072c0a911
sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43037
diff
changeset
|
49 SD_TEST7 = 7 |
ba4072c0a911
sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43037
diff
changeset
|
50 |
43142
beed7ce61681
sidedatacopies: write copies information in sidedata when applicable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43077
diff
changeset
|
51 # key to store copies related information |
beed7ce61681
sidedatacopies: write copies information in sidedata when applicable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43077
diff
changeset
|
52 SD_P1COPIES = 8 |
beed7ce61681
sidedatacopies: write copies information in sidedata when applicable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43077
diff
changeset
|
53 SD_P2COPIES = 9 |
beed7ce61681
sidedatacopies: write copies information in sidedata when applicable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43077
diff
changeset
|
54 SD_FILESADDED = 10 |
beed7ce61681
sidedatacopies: write copies information in sidedata when applicable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43077
diff
changeset
|
55 SD_FILESREMOVED = 11 |
45634
9a6b409b8ebc
changing-files: rework the way we store changed files in side-data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
44060
diff
changeset
|
56 SD_FILES = 12 |
43142
beed7ce61681
sidedatacopies: write copies information in sidedata when applicable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43077
diff
changeset
|
57 |
43040
ba4072c0a911
sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43037
diff
changeset
|
58 # internal format constant |
43506
9f70512ae2cf
cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents:
43142
diff
changeset
|
59 SIDEDATA_HEADER = struct.Struct('>H') |
9f70512ae2cf
cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents:
43142
diff
changeset
|
60 SIDEDATA_ENTRY = struct.Struct('>HL20s') |
43034
294afb982a88
sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43033
diff
changeset
|
61 |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
43048
diff
changeset
|
62 |
46709
3d740058b467
sidedata: move to new sidedata storage in revlogv2
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46195
diff
changeset
|
63 def serialize_sidedata(sidedata): |
43035
ea83abf95630
sidedata: add a function to write sidedata into a raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43034
diff
changeset
|
64 sidedata = list(sidedata.items()) |
ea83abf95630
sidedata: add a function to write sidedata into a raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43034
diff
changeset
|
65 sidedata.sort() |
46709
3d740058b467
sidedata: move to new sidedata storage in revlogv2
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46195
diff
changeset
|
66 buf = [SIDEDATA_HEADER.pack(len(sidedata))] |
43035
ea83abf95630
sidedata: add a function to write sidedata into a raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43034
diff
changeset
|
67 for key, value in sidedata: |
44060
a61287a95dc3
core: migrate uses of hashlib.sha1 to hashutil.sha1
Augie Fackler <augie@google.com>
parents:
43506
diff
changeset
|
68 digest = hashutil.sha1(value).digest() |
46709
3d740058b467
sidedata: move to new sidedata storage in revlogv2
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46195
diff
changeset
|
69 buf.append(SIDEDATA_ENTRY.pack(key, len(value), digest)) |
43035
ea83abf95630
sidedata: add a function to write sidedata into a raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43034
diff
changeset
|
70 for key, value in sidedata: |
46709
3d740058b467
sidedata: move to new sidedata storage in revlogv2
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46195
diff
changeset
|
71 buf.append(value) |
3d740058b467
sidedata: move to new sidedata storage in revlogv2
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46195
diff
changeset
|
72 buf = b''.join(buf) |
3d740058b467
sidedata: move to new sidedata storage in revlogv2
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46195
diff
changeset
|
73 return buf |
43035
ea83abf95630
sidedata: add a function to write sidedata into a raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43034
diff
changeset
|
74 |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
43048
diff
changeset
|
75 |
46709
3d740058b467
sidedata: move to new sidedata storage in revlogv2
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46195
diff
changeset
|
76 def deserialize_sidedata(blob): |
43034
294afb982a88
sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43033
diff
changeset
|
77 sidedata = {} |
294afb982a88
sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43033
diff
changeset
|
78 offset = 0 |
46709
3d740058b467
sidedata: move to new sidedata storage in revlogv2
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46195
diff
changeset
|
79 (nbentry,) = SIDEDATA_HEADER.unpack(blob[: SIDEDATA_HEADER.size]) |
43034
294afb982a88
sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43033
diff
changeset
|
80 offset += SIDEDATA_HEADER.size |
294afb982a88
sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43033
diff
changeset
|
81 dataoffset = SIDEDATA_HEADER.size + (SIDEDATA_ENTRY.size * nbentry) |
294afb982a88
sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43033
diff
changeset
|
82 for i in range(nbentry): |
294afb982a88
sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43033
diff
changeset
|
83 nextoffset = offset + SIDEDATA_ENTRY.size |
46709
3d740058b467
sidedata: move to new sidedata storage in revlogv2
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46195
diff
changeset
|
84 key, size, storeddigest = SIDEDATA_ENTRY.unpack(blob[offset:nextoffset]) |
43034
294afb982a88
sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43033
diff
changeset
|
85 offset = nextoffset |
294afb982a88
sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43033
diff
changeset
|
86 # read the data associated with that entry |
294afb982a88
sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43033
diff
changeset
|
87 nextdataoffset = dataoffset + size |
46709
3d740058b467
sidedata: move to new sidedata storage in revlogv2
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46195
diff
changeset
|
88 entrytext = bytes(blob[dataoffset:nextdataoffset]) |
44060
a61287a95dc3
core: migrate uses of hashlib.sha1 to hashutil.sha1
Augie Fackler <augie@google.com>
parents:
43506
diff
changeset
|
89 readdigest = hashutil.sha1(entrytext).digest() |
43034
294afb982a88
sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43033
diff
changeset
|
90 if storeddigest != readdigest: |
294afb982a88
sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43033
diff
changeset
|
91 raise error.SidedataHashError(key, storeddigest, readdigest) |
294afb982a88
sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43033
diff
changeset
|
92 sidedata[key] = entrytext |
294afb982a88
sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
43033
diff
changeset
|
93 dataoffset = nextdataoffset |
46709
3d740058b467
sidedata: move to new sidedata storage in revlogv2
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46195
diff
changeset
|
94 return sidedata |
47085
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46709
diff
changeset
|
95 |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46709
diff
changeset
|
96 |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46709
diff
changeset
|
97 def get_sidedata_helpers(repo, remote_sd_categories, pull=False): |
47086
8bd769b5c941
sidedata: move documentation about sidedata helpers to sidedata module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
47085
diff
changeset
|
98 """ |
8bd769b5c941
sidedata: move documentation about sidedata helpers to sidedata module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
47085
diff
changeset
|
99 Returns a dictionary mapping revlog types to tuples of |
8bd769b5c941
sidedata: move documentation about sidedata helpers to sidedata module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
47085
diff
changeset
|
100 `(repo, computers, removers)`: |
8bd769b5c941
sidedata: move documentation about sidedata helpers to sidedata module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
47085
diff
changeset
|
101 * `repo` is used as an argument for computers |
8bd769b5c941
sidedata: move documentation about sidedata helpers to sidedata module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
47085
diff
changeset
|
102 * `computers` is a list of `(category, (keys, computer, flags)` that |
8bd769b5c941
sidedata: move documentation about sidedata helpers to sidedata module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
47085
diff
changeset
|
103 compute the missing sidedata categories that were asked: |
8bd769b5c941
sidedata: move documentation about sidedata helpers to sidedata module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
47085
diff
changeset
|
104 * `category` is the sidedata category |
8bd769b5c941
sidedata: move documentation about sidedata helpers to sidedata module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
47085
diff
changeset
|
105 * `keys` are the sidedata keys to be affected |
8bd769b5c941
sidedata: move documentation about sidedata helpers to sidedata module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
47085
diff
changeset
|
106 * `flags` is a bitmask (an integer) of flags to remove when |
8bd769b5c941
sidedata: move documentation about sidedata helpers to sidedata module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
47085
diff
changeset
|
107 removing the category. |
8bd769b5c941
sidedata: move documentation about sidedata helpers to sidedata module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
47085
diff
changeset
|
108 * `computer` is the function `(repo, store, rev, sidedata)` that |
8bd769b5c941
sidedata: move documentation about sidedata helpers to sidedata module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
47085
diff
changeset
|
109 returns a tuple of |
8bd769b5c941
sidedata: move documentation about sidedata helpers to sidedata module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
47085
diff
changeset
|
110 `(new sidedata dict, (flags to add, flags to remove))`. |
8bd769b5c941
sidedata: move documentation about sidedata helpers to sidedata module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
47085
diff
changeset
|
111 For example, it will return `({}, (0, 1 << 15))` to return no |
8bd769b5c941
sidedata: move documentation about sidedata helpers to sidedata module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
47085
diff
changeset
|
112 sidedata, with no flags to add and one flag to remove. |
8bd769b5c941
sidedata: move documentation about sidedata helpers to sidedata module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
47085
diff
changeset
|
113 * `removers` will remove the keys corresponding to the categories |
8bd769b5c941
sidedata: move documentation about sidedata helpers to sidedata module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
47085
diff
changeset
|
114 that are present, but not needed. |
8bd769b5c941
sidedata: move documentation about sidedata helpers to sidedata module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
47085
diff
changeset
|
115 If both `computers` and `removers` are empty, sidedata will simply not |
8bd769b5c941
sidedata: move documentation about sidedata helpers to sidedata module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
47085
diff
changeset
|
116 be transformed. |
8bd769b5c941
sidedata: move documentation about sidedata helpers to sidedata module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
47085
diff
changeset
|
117 """ |
47085
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46709
diff
changeset
|
118 # Computers for computing sidedata on-the-fly |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46709
diff
changeset
|
119 sd_computers = collections.defaultdict(list) |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46709
diff
changeset
|
120 # Computers for categories to remove from sidedata |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46709
diff
changeset
|
121 sd_removers = collections.defaultdict(list) |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46709
diff
changeset
|
122 to_generate = remote_sd_categories - repo._wanted_sidedata |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46709
diff
changeset
|
123 to_remove = repo._wanted_sidedata - remote_sd_categories |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46709
diff
changeset
|
124 if pull: |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46709
diff
changeset
|
125 to_generate, to_remove = to_remove, to_generate |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46709
diff
changeset
|
126 |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46709
diff
changeset
|
127 for revlog_kind, computers in repo._sidedata_computers.items(): |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46709
diff
changeset
|
128 for category, computer in computers.items(): |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46709
diff
changeset
|
129 if category in to_generate: |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46709
diff
changeset
|
130 sd_computers[revlog_kind].append(computer) |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46709
diff
changeset
|
131 if category in to_remove: |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46709
diff
changeset
|
132 sd_removers[revlog_kind].append(computer) |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46709
diff
changeset
|
133 |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46709
diff
changeset
|
134 sidedata_helpers = (repo, sd_computers, sd_removers) |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46709
diff
changeset
|
135 return sidedata_helpers |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46709
diff
changeset
|
136 |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46709
diff
changeset
|
137 |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46709
diff
changeset
|
138 def run_sidedata_helpers(store, sidedata_helpers, sidedata, rev): |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46709
diff
changeset
|
139 """Returns the sidedata for the given revision after running through |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46709
diff
changeset
|
140 the given helpers. |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46709
diff
changeset
|
141 - `store`: the revlog this applies to (changelog, manifest, or filelog |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46709
diff
changeset
|
142 instance) |
47086
8bd769b5c941
sidedata: move documentation about sidedata helpers to sidedata module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
47085
diff
changeset
|
143 - `sidedata_helpers`: see `get_sidedata_helpers` |
47085
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46709
diff
changeset
|
144 - `sidedata`: previous sidedata at the given rev, if any |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46709
diff
changeset
|
145 - `rev`: affected rev of `store` |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46709
diff
changeset
|
146 """ |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46709
diff
changeset
|
147 repo, sd_computers, sd_removers = sidedata_helpers |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46709
diff
changeset
|
148 kind = store.revlog_kind |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46709
diff
changeset
|
149 flags_to_add = 0 |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46709
diff
changeset
|
150 flags_to_remove = 0 |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46709
diff
changeset
|
151 for _keys, sd_computer, _flags in sd_computers.get(kind, []): |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46709
diff
changeset
|
152 sidedata, flags = sd_computer(repo, store, rev, sidedata) |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46709
diff
changeset
|
153 flags_to_add |= flags[0] |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46709
diff
changeset
|
154 flags_to_remove |= flags[1] |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46709
diff
changeset
|
155 for keys, _computer, flags in sd_removers.get(kind, []): |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46709
diff
changeset
|
156 for key in keys: |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46709
diff
changeset
|
157 sidedata.pop(key, None) |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46709
diff
changeset
|
158 flags_to_remove |= flags |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46709
diff
changeset
|
159 return sidedata, (flags_to_add, flags_to_remove) |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46709
diff
changeset
|
160 |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46709
diff
changeset
|
161 |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46709
diff
changeset
|
162 def set_sidedata_spec_for_repo(repo): |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46709
diff
changeset
|
163 # prevent cycle metadata -> revlogutils.sidedata -> metadata |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46709
diff
changeset
|
164 from .. import metadata |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46709
diff
changeset
|
165 |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46709
diff
changeset
|
166 if requirementsmod.COPIESSDC_REQUIREMENT in repo.requirements: |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46709
diff
changeset
|
167 repo.register_wanted_sidedata(SD_FILES) |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46709
diff
changeset
|
168 repo.register_sidedata_computer( |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46709
diff
changeset
|
169 constants.KIND_CHANGELOG, |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46709
diff
changeset
|
170 SD_FILES, |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46709
diff
changeset
|
171 (SD_FILES,), |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46709
diff
changeset
|
172 metadata.copies_sidedata_computer, |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46709
diff
changeset
|
173 flagutil.REVIDX_HASCOPIESINFO, |
3aab2330b7d3
sidedata: move sidedata-related utils to the dedicated module
Rapha?l Gom?s <rgomes@octobus.net>
parents:
46709
diff
changeset
|
174 ) |