Mercurial > public > mercurial-scm > hg

annotate tests/testlib/ext-sidedata.py @ 47084:27f1191b1305

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
sidedata: replace sidedata upgrade mechanism with the new one Note: this is split into a separate change (like some other patches in this series) because it's not easy to have all patches work 100% and this seemed easier for reviewers. When cloning or upgrading a repo, we may need to compute (or remove) sidedata. This is the same mechanism that is used in exchange, so we re-use the new system to simplify the code and fix the remaining issues (correctly dropping flags and handling partial removal, etc.). This also highlighted an issue with `test-copies-in-changeset.t` that kept sidedata categories that are not relevant anymore. They should probably be dropped entirely, but that would be for another patch. Differential Revision: https://phab.mercurial-scm.org/D10359
author Rapha?l Gom?s <rgomes@octobus.net>
date Mon, 19 Apr 2021 11:22:24 +0200
parents d55b71393907
children 3aab2330b7d3
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
43040
ba4072c0a911 sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
1 # ext-sidedata.py - small extension to test the sidedata logic
ba4072c0a911 sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
2 #
46718
ba8e508a8e69 sidedata-exchange: rewrite sidedata on-the-fly whenever possible
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46709
diff changeset
3 # Copyright 2019 Pierre-Yves David <pierre-yves.david@octobus.net>
43040
ba4072c0a911 sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
4 #
ba4072c0a911 sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
5 # This software may be used and distributed according to the terms of the
ba4072c0a911 sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
6 # GNU General Public License version 2 or any later version.
ba4072c0a911 sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
7
ba4072c0a911 sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
8 from __future__ import absolute_import
ba4072c0a911 sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
9
ba4072c0a911 sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
10 import hashlib
ba4072c0a911 sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
11 import struct
ba4072c0a911 sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
12
47012
d55b71393907 node: replace nullid and friends with nodeconstants class
Joerg Sonnenberger <joerg@bec.de>
parents: 46718
diff changeset
13 from mercurial.node import nullrev
43040
ba4072c0a911 sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
14 from mercurial import (
47084
27f1191b1305 sidedata: replace sidedata upgrade mechanism with the new one
Rapha?l Gom?s <rgomes@octobus.net>
parents: 47012
diff changeset
15 changegroup,
43040
ba4072c0a911 sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
16 extensions,
45373
d7dcc75a3eae localrepo: move requirements constant to requirements module
Pulkit Goyal <7895pulkit@gmail.com>
parents: 43134
diff changeset
17 requirements,
43040
ba4072c0a911 sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
18 revlog,
ba4072c0a911 sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
19 )
ba4072c0a911 sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
20
46046
f105c49e89cd upgrade: split actual upgrade code away from the main module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 45735
diff changeset
21 from mercurial.upgrade_utils import engine as upgrade_engine
f105c49e89cd upgrade: split actual upgrade code away from the main module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 45735
diff changeset
22
47084
27f1191b1305 sidedata: replace sidedata upgrade mechanism with the new one
Rapha?l Gom?s <rgomes@octobus.net>
parents: 47012
diff changeset
23 from mercurial.revlogutils import constants
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43042
diff changeset
24 from mercurial.revlogutils import sidedata
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43042
diff changeset
25
43040
ba4072c0a911 sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
26
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43042
diff changeset
27 def wrapaddrevision(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43042
diff changeset
28 orig, self, text, transaction, link, p1, p2, *args, **kwargs
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43042
diff changeset
29 ):
43040
ba4072c0a911 sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
30 if kwargs.get('sidedata') is None:
ba4072c0a911 sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
31 kwargs['sidedata'] = {}
ba4072c0a911 sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
32 sd = kwargs['sidedata']
ba4072c0a911 sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
33 ## let's store some arbitrary data just for testing
ba4072c0a911 sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
34 # text length
ba4072c0a911 sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
35 sd[sidedata.SD_TEST1] = struct.pack('>I', len(text))
ba4072c0a911 sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
36 # and sha2 hashes
ba4072c0a911 sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
37 sha256 = hashlib.sha256(text).digest()
ba4072c0a911 sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
38 sd[sidedata.SD_TEST2] = struct.pack('>32s', sha256)
ba4072c0a911 sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
39 return orig(self, text, transaction, link, p1, p2, *args, **kwargs)
ba4072c0a911 sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
40
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43042
diff changeset
41
46709
3d740058b467 sidedata: move to new sidedata storage in revlogv2
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46113
diff changeset
42 def wrap_revisiondata(orig, self, nodeorrev, *args, **kwargs):
3d740058b467 sidedata: move to new sidedata storage in revlogv2
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46113
diff changeset
43 text, sd = orig(self, nodeorrev, *args, **kwargs)
43134
75ad8af9c95e upgrade: allow upgrade to repository using sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43076
diff changeset
44 if getattr(self, 'sidedatanocheck', False):
46709
3d740058b467 sidedata: move to new sidedata storage in revlogv2
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46113
diff changeset
45 return text, sd
3d740058b467 sidedata: move to new sidedata storage in revlogv2
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46113
diff changeset
46 if self.version & 0xFFFF != 2:
3d740058b467 sidedata: move to new sidedata storage in revlogv2
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46113
diff changeset
47 return text, sd
47012
d55b71393907 node: replace nullid and friends with nodeconstants class
Joerg Sonnenberger <joerg@bec.de>
parents: 46718
diff changeset
48 if nodeorrev != nullrev and nodeorrev != self.nullid:
46718
ba8e508a8e69 sidedata-exchange: rewrite sidedata on-the-fly whenever possible
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46709
diff changeset
49 cat1 = sd.get(sidedata.SD_TEST1)
ba8e508a8e69 sidedata-exchange: rewrite sidedata on-the-fly whenever possible
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46709
diff changeset
50 if cat1 is not None and len(text) != struct.unpack('>I', cat1)[0]:
43042
03e769278ef3 sidedata: check that the sidedata safely roundtrip
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43040
diff changeset
51 raise RuntimeError('text size mismatch')
46718
ba8e508a8e69 sidedata-exchange: rewrite sidedata on-the-fly whenever possible
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46709
diff changeset
52 expected = sd.get(sidedata.SD_TEST2)
43042
03e769278ef3 sidedata: check that the sidedata safely roundtrip
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43040
diff changeset
53 got = hashlib.sha256(text).digest()
46718
ba8e508a8e69 sidedata-exchange: rewrite sidedata on-the-fly whenever possible
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46709
diff changeset
54 if expected is not None and got != expected:
43042
03e769278ef3 sidedata: check that the sidedata safely roundtrip
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43040
diff changeset
55 raise RuntimeError('sha256 mismatch')
46709
3d740058b467 sidedata: move to new sidedata storage in revlogv2
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46113
diff changeset
56 return text, sd
43042
03e769278ef3 sidedata: check that the sidedata safely roundtrip
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43040
diff changeset
57
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43042
diff changeset
58
47084
27f1191b1305 sidedata: replace sidedata upgrade mechanism with the new one
Rapha?l Gom?s <rgomes@octobus.net>
parents: 47012
diff changeset
59 def wrapget_sidedata_helpers(orig, srcrepo, dstrepo):
27f1191b1305 sidedata: replace sidedata upgrade mechanism with the new one
Rapha?l Gom?s <rgomes@octobus.net>
parents: 47012
diff changeset
60 repo, computers, removers = orig(srcrepo, dstrepo)
27f1191b1305 sidedata: replace sidedata upgrade mechanism with the new one
Rapha?l Gom?s <rgomes@octobus.net>
parents: 47012
diff changeset
61 assert not computers and not removers # deal with composition later
43134
75ad8af9c95e upgrade: allow upgrade to repository using sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43076
diff changeset
62 addedreqs = dstrepo.requirements - srcrepo.requirements
47084
27f1191b1305 sidedata: replace sidedata upgrade mechanism with the new one
Rapha?l Gom?s <rgomes@octobus.net>
parents: 47012
diff changeset
63
45373
d7dcc75a3eae localrepo: move requirements constant to requirements module
Pulkit Goyal <7895pulkit@gmail.com>
parents: 43134
diff changeset
64 if requirements.SIDEDATA_REQUIREMENT in addedreqs:
43134
75ad8af9c95e upgrade: allow upgrade to repository using sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43076
diff changeset
65
47084
27f1191b1305 sidedata: replace sidedata upgrade mechanism with the new one
Rapha?l Gom?s <rgomes@octobus.net>
parents: 47012
diff changeset
66 def computer(repo, revlog, rev, old_sidedata):
27f1191b1305 sidedata: replace sidedata upgrade mechanism with the new one
Rapha?l Gom?s <rgomes@octobus.net>
parents: 47012
diff changeset
67 assert not old_sidedata # not supported yet
43134
75ad8af9c95e upgrade: allow upgrade to repository using sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43076
diff changeset
68 update = {}
75ad8af9c95e upgrade: allow upgrade to repository using sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43076
diff changeset
69 revlog.sidedatanocheck = True
75ad8af9c95e upgrade: allow upgrade to repository using sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43076
diff changeset
70 try:
75ad8af9c95e upgrade: allow upgrade to repository using sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43076
diff changeset
71 text = revlog.revision(rev)
75ad8af9c95e upgrade: allow upgrade to repository using sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43076
diff changeset
72 finally:
75ad8af9c95e upgrade: allow upgrade to repository using sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43076
diff changeset
73 del revlog.sidedatanocheck
75ad8af9c95e upgrade: allow upgrade to repository using sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43076
diff changeset
74 ## let's store some arbitrary data just for testing
75ad8af9c95e upgrade: allow upgrade to repository using sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43076
diff changeset
75 # text length
75ad8af9c95e upgrade: allow upgrade to repository using sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43076
diff changeset
76 update[sidedata.SD_TEST1] = struct.pack('>I', len(text))
75ad8af9c95e upgrade: allow upgrade to repository using sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43076
diff changeset
77 # and sha2 hashes
75ad8af9c95e upgrade: allow upgrade to repository using sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43076
diff changeset
78 sha256 = hashlib.sha256(text).digest()
75ad8af9c95e upgrade: allow upgrade to repository using sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43076
diff changeset
79 update[sidedata.SD_TEST2] = struct.pack('>32s', sha256)
47084
27f1191b1305 sidedata: replace sidedata upgrade mechanism with the new one
Rapha?l Gom?s <rgomes@octobus.net>
parents: 47012
diff changeset
80 return update, (0, 0)
43134
75ad8af9c95e upgrade: allow upgrade to repository using sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43076
diff changeset
81
47084
27f1191b1305 sidedata: replace sidedata upgrade mechanism with the new one
Rapha?l Gom?s <rgomes@octobus.net>
parents: 47012
diff changeset
82 srcrepo.register_sidedata_computer(
27f1191b1305 sidedata: replace sidedata upgrade mechanism with the new one
Rapha?l Gom?s <rgomes@octobus.net>
parents: 47012
diff changeset
83 constants.KIND_CHANGELOG,
27f1191b1305 sidedata: replace sidedata upgrade mechanism with the new one
Rapha?l Gom?s <rgomes@octobus.net>
parents: 47012
diff changeset
84 b"whatever",
27f1191b1305 sidedata: replace sidedata upgrade mechanism with the new one
Rapha?l Gom?s <rgomes@octobus.net>
parents: 47012
diff changeset
85 (sidedata.SD_TEST1, sidedata.SD_TEST2),
27f1191b1305 sidedata: replace sidedata upgrade mechanism with the new one
Rapha?l Gom?s <rgomes@octobus.net>
parents: 47012
diff changeset
86 computer,
27f1191b1305 sidedata: replace sidedata upgrade mechanism with the new one
Rapha?l Gom?s <rgomes@octobus.net>
parents: 47012
diff changeset
87 0,
27f1191b1305 sidedata: replace sidedata upgrade mechanism with the new one
Rapha?l Gom?s <rgomes@octobus.net>
parents: 47012
diff changeset
88 )
27f1191b1305 sidedata: replace sidedata upgrade mechanism with the new one
Rapha?l Gom?s <rgomes@octobus.net>
parents: 47012
diff changeset
89 dstrepo.register_wanted_sidedata(b"whatever")
27f1191b1305 sidedata: replace sidedata upgrade mechanism with the new one
Rapha?l Gom?s <rgomes@octobus.net>
parents: 47012
diff changeset
90
27f1191b1305 sidedata: replace sidedata upgrade mechanism with the new one
Rapha?l Gom?s <rgomes@octobus.net>
parents: 47012
diff changeset
91 return changegroup.get_sidedata_helpers(srcrepo, dstrepo._wanted_sidedata)
43134
75ad8af9c95e upgrade: allow upgrade to repository using sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43076
diff changeset
92
75ad8af9c95e upgrade: allow upgrade to repository using sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43076
diff changeset
93
43040
ba4072c0a911 sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
94 def extsetup(ui):
ba4072c0a911 sidedata: test we can successfully write sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
95 extensions.wrapfunction(revlog.revlog, 'addrevision', wrapaddrevision)
46709
3d740058b467 sidedata: move to new sidedata storage in revlogv2
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46113
diff changeset
96 extensions.wrapfunction(revlog.revlog, '_revisiondata', wrap_revisiondata)
43134
75ad8af9c95e upgrade: allow upgrade to repository using sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43076
diff changeset
97 extensions.wrapfunction(
47084
27f1191b1305 sidedata: replace sidedata upgrade mechanism with the new one
Rapha?l Gom?s <rgomes@octobus.net>
parents: 47012
diff changeset
98 upgrade_engine, 'get_sidedata_helpers', wrapget_sidedata_helpers
43134
75ad8af9c95e upgrade: allow upgrade to repository using sidedata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43076
diff changeset
99 )
46718
ba8e508a8e69 sidedata-exchange: rewrite sidedata on-the-fly whenever possible
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46709
diff changeset
100
ba8e508a8e69 sidedata-exchange: rewrite sidedata on-the-fly whenever possible
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46709
diff changeset
101
ba8e508a8e69 sidedata-exchange: rewrite sidedata on-the-fly whenever possible
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46709
diff changeset
102 def reposetup(ui, repo):
ba8e508a8e69 sidedata-exchange: rewrite sidedata on-the-fly whenever possible
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46709
diff changeset
103 # We don't register sidedata computers because we don't care within these
ba8e508a8e69 sidedata-exchange: rewrite sidedata on-the-fly whenever possible
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46709
diff changeset
104 # tests
ba8e508a8e69 sidedata-exchange: rewrite sidedata on-the-fly whenever possible
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46709
diff changeset
105 repo.register_wanted_sidedata(sidedata.SD_TEST1)
ba8e508a8e69 sidedata-exchange: rewrite sidedata on-the-fly whenever possible
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46709
diff changeset
106 repo.register_wanted_sidedata(sidedata.SD_TEST2)