Mercurial > public > mercurial-scm > hg-stable

annotate mercurial/revlogutils/nodemap.py @ 46114:59fa3890d40a

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
node: import symbols explicitly There is no point in lazy importing mercurial.node, it is used all over the place anyway. So consistently import the used symbols directly. Fix one file using symbols indirectly via mercurial.revlog. Differential Revision: https://phab.mercurial-scm.org/D9480
author Joerg Sonnenberger <joerg@bec.de>
date Tue, 01 Dec 2020 21:54:46 +0100
parents 8ff2d8359d0f
children 8e7ce6555ea7
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
44035
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
1 # nodemap.py - nodemap related code and utilities
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
2 #
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
3 # Copyright 2019 Pierre-Yves David <pierre-yves.david@octobus.net>
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
4 # Copyright 2019 George Racinet <georges.racinet@octobus.net>
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
5 #
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
6 # This software may be used and distributed according to the terms of the
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
7 # GNU General Public License version 2 or any later version.
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
8
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
9 from __future__ import absolute_import
44350
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
10
44397
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
11 import errno
44354
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
12 import os
44355
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44354
diff changeset
13 import re
44350
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
14 import struct
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
15
44865
6493f0a567c2 nodemap: add a new mode option, with an optional "warn" value
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44740
diff changeset
16 from ..i18n import _
46114
59fa3890d40a node: import symbols explicitly
Joerg Sonnenberger <joerg@bec.de>
parents: 46090
diff changeset
17 from ..node import hex
44865
6493f0a567c2 nodemap: add a new mode option, with an optional "warn" value
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44740
diff changeset
18
44350
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
19 from .. import (
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
20 error,
44357
7f4f7ef3133e nodemap: add a optional `nodemap_add_full` method on indexes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44355
diff changeset
21 util,
44350
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
22 )
44035
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
23
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
24
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
25 class NodeMap(dict):
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
26 def __missing__(self, x):
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
27 raise error.RevlogError(b'unknown node: %s' % x)
44350
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
28
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
29
44352
6c07480d6659 nodemap: add a function to read the data from disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44351
diff changeset
30 def persisted_data(revlog):
6c07480d6659 nodemap: add a function to read the data from disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44351
diff changeset
31 """read the nodemap for a revlog from disk"""
6c07480d6659 nodemap: add a function to read the data from disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44351
diff changeset
32 if revlog.nodemap_file is None:
6c07480d6659 nodemap: add a function to read the data from disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44351
diff changeset
33 return None
44354
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
34 pdata = revlog.opener.tryread(revlog.nodemap_file)
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
35 if not pdata:
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
36 return None
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
37 offset = 0
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
38 (version,) = S_VERSION.unpack(pdata[offset : offset + S_VERSION.size])
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
39 if version != ONDISK_VERSION:
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
40 return None
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
41 offset += S_VERSION.size
44369
e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44367
diff changeset
42 headers = S_HEADER.unpack(pdata[offset : offset + S_HEADER.size])
44525
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44509
diff changeset
43 uid_size, tip_rev, data_length, data_unused, tip_node_size = headers
44354
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
44 offset += S_HEADER.size
44365
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
45 docket = NodeMapDocket(pdata[offset : offset + uid_size])
44525
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44509
diff changeset
46 offset += uid_size
44369
e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44367
diff changeset
47 docket.tip_rev = tip_rev
44525
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44509
diff changeset
48 docket.tip_node = pdata[offset : offset + tip_node_size]
44370
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
49 docket.data_length = data_length
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
50 docket.data_unused = data_unused
44354
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
51
44365
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
52 filename = _rawdata_filepath(revlog, docket)
44869
5e3c718692bb nodemap: drop the 'exp-' prefix for internal opener option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44867
diff changeset
53 use_mmap = revlog.opener.options.get(b"persistent-nodemap.mmap")
44397
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
54 try:
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
55 with revlog.opener(filename) as fd:
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
56 if use_mmap:
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
57 data = util.buffer(util.mmapread(fd, data_length))
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
58 else:
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
59 data = fd.read(data_length)
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
60 except OSError as e:
46090
8ff2d8359d0f persistent-nodemap: properly ignore non-existent `.nd` data file
Simon Sapin <simon-commits@exyr.org>
parents: 45957
diff changeset
61 if e.errno == errno.ENOENT:
8ff2d8359d0f persistent-nodemap: properly ignore non-existent `.nd` data file
Simon Sapin <simon-commits@exyr.org>
parents: 45957
diff changeset
62 return None
8ff2d8359d0f persistent-nodemap: properly ignore non-existent `.nd` data file
Simon Sapin <simon-commits@exyr.org>
parents: 45957
diff changeset
63 else:
44397
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
64 raise
44373
c7eebdb15139 nodemap: never read more than the expected data amount
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44372
diff changeset
65 if len(data) < data_length:
c7eebdb15139 nodemap: never read more than the expected data amount
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44372
diff changeset
66 return None
c7eebdb15139 nodemap: never read more than the expected data amount
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44372
diff changeset
67 return docket, data
44352
6c07480d6659 nodemap: add a function to read the data from disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44351
diff changeset
68
6c07480d6659 nodemap: add a function to read the data from disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44351
diff changeset
69
44351
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44350
diff changeset
70 def setup_persistent_nodemap(tr, revlog):
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44350
diff changeset
71 """Install whatever is needed transaction side to persist a nodemap on disk
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44350
diff changeset
72
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44350
diff changeset
73 (only actually persist the nodemap if this is relevant for this revlog)
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44350
diff changeset
74 """
44353
daad3aace942 nodemap: only use persistent nodemap for non-inlined revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44352
diff changeset
75 if revlog._inline:
daad3aace942 nodemap: only use persistent nodemap for non-inlined revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44352
diff changeset
76 return # inlined revlog are too small for this to be relevant
44351
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44350
diff changeset
77 if revlog.nodemap_file is None:
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44350
diff changeset
78 return # we do not use persistent_nodemap on this revlog
44527
448d700e0d27 nodemap: make sure the nodemap docket is updated after the changelog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44526
diff changeset
79
448d700e0d27 nodemap: make sure the nodemap docket is updated after the changelog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44526
diff changeset
80 # we need to happen after the changelog finalization, in that use "cl-"
448d700e0d27 nodemap: make sure the nodemap docket is updated after the changelog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44526
diff changeset
81 callback_id = b"nm-revlog-persistent-nodemap-%s" % revlog.nodemap_file
44351
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44350
diff changeset
82 if tr.hasfinalize(callback_id):
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44350
diff changeset
83 return # no need to register again
44526
64e2f603de9d nodemap: make sure hooks have access to an up-to-date version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44525
diff changeset
84 tr.addpending(
64e2f603de9d nodemap: make sure hooks have access to an up-to-date version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44525
diff changeset
85 callback_id, lambda tr: _persist_nodemap(tr, revlog, pending=True)
64e2f603de9d nodemap: make sure hooks have access to an up-to-date version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44525
diff changeset
86 )
44509
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44506
diff changeset
87 tr.addfinalize(callback_id, lambda tr: _persist_nodemap(tr, revlog))
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44506
diff changeset
88
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44506
diff changeset
89
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44506
diff changeset
90 class _NoTransaction(object):
45957
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 44869
diff changeset
91 """transaction like object to update the nodemap outside a transaction"""
44509
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44506
diff changeset
92
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44506
diff changeset
93 def __init__(self):
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44506
diff changeset
94 self._postclose = {}
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44506
diff changeset
95
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44506
diff changeset
96 def addpostclose(self, callback_id, callback_func):
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44506
diff changeset
97 self._postclose[callback_id] = callback_func
44351
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44350
diff changeset
98
44638
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44527
diff changeset
99 def registertmp(self, *args, **kwargs):
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44527
diff changeset
100 pass
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44527
diff changeset
101
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44527
diff changeset
102 def addbackup(self, *args, **kwargs):
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44527
diff changeset
103 pass
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44527
diff changeset
104
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44527
diff changeset
105 def add(self, *args, **kwargs):
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44527
diff changeset
106 pass
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44527
diff changeset
107
44639
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44638
diff changeset
108 def addabort(self, *args, **kwargs):
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44638
diff changeset
109 pass
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44638
diff changeset
110
44865
6493f0a567c2 nodemap: add a new mode option, with an optional "warn" value
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44740
diff changeset
111 def _report(self, *args):
6493f0a567c2 nodemap: add a new mode option, with an optional "warn" value
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44740
diff changeset
112 pass
6493f0a567c2 nodemap: add a new mode option, with an optional "warn" value
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44740
diff changeset
113
44351
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44350
diff changeset
114
44465
336ec75ed1ac nodemap: warm the persistent nodemap on disk with debugupdatecache
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44397
diff changeset
115 def update_persistent_nodemap(revlog):
336ec75ed1ac nodemap: warm the persistent nodemap on disk with debugupdatecache
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44397
diff changeset
116 """update the persistent nodemap right now
336ec75ed1ac nodemap: warm the persistent nodemap on disk with debugupdatecache
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44397
diff changeset
117
336ec75ed1ac nodemap: warm the persistent nodemap on disk with debugupdatecache
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44397
diff changeset
118 To be used for updating the nodemap on disk outside of a normal transaction
336ec75ed1ac nodemap: warm the persistent nodemap on disk with debugupdatecache
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44397
diff changeset
119 setup (eg, `debugupdatecache`).
336ec75ed1ac nodemap: warm the persistent nodemap on disk with debugupdatecache
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44397
diff changeset
120 """
44740
b4537125eb3c nodemap: skip persistent nodemap warming for revlog not using it
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44697
diff changeset
121 if revlog._inline:
b4537125eb3c nodemap: skip persistent nodemap warming for revlog not using it
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44697
diff changeset
122 return # inlined revlog are too small for this to be relevant
b4537125eb3c nodemap: skip persistent nodemap warming for revlog not using it
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44697
diff changeset
123 if revlog.nodemap_file is None:
b4537125eb3c nodemap: skip persistent nodemap warming for revlog not using it
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44697
diff changeset
124 return # we do not use persistent_nodemap on this revlog
b4537125eb3c nodemap: skip persistent nodemap warming for revlog not using it
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44697
diff changeset
125
44509
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44506
diff changeset
126 notr = _NoTransaction()
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44506
diff changeset
127 _persist_nodemap(notr, revlog)
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44506
diff changeset
128 for k in sorted(notr._postclose):
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44506
diff changeset
129 notr._postclose[k](None)
44465
336ec75ed1ac nodemap: warm the persistent nodemap on disk with debugupdatecache
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44397
diff changeset
130
336ec75ed1ac nodemap: warm the persistent nodemap on disk with debugupdatecache
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44397
diff changeset
131
44526
64e2f603de9d nodemap: make sure hooks have access to an up-to-date version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44525
diff changeset
132 def _persist_nodemap(tr, revlog, pending=False):
45957
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 44869
diff changeset
133 """Write nodemap data on disk for a given revlog"""
44351
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44350
diff changeset
134 if getattr(revlog, 'filteredrevs', ()):
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44350
diff changeset
135 raise error.ProgrammingError(
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44350
diff changeset
136 "cannot persist nodemap of a filtered changelog"
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44350
diff changeset
137 )
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44350
diff changeset
138 if revlog.nodemap_file is None:
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44350
diff changeset
139 msg = "calling persist nodemap on a revlog without the feature enableb"
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44350
diff changeset
140 raise error.ProgrammingError(msg)
44367
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
141
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
142 can_incremental = util.safehasattr(revlog.index, "nodemap_data_incremental")
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
143 ondisk_docket = revlog._nodemap_docket
44397
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
144 feed_data = util.safehasattr(revlog.index, "update_nodemap_data")
44869
5e3c718692bb nodemap: drop the 'exp-' prefix for internal opener option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44867
diff changeset
145 use_mmap = revlog.opener.options.get(b"persistent-nodemap.mmap")
5e3c718692bb nodemap: drop the 'exp-' prefix for internal opener option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44867
diff changeset
146 mode = revlog.opener.options.get(b"persistent-nodemap.mode")
44865
6493f0a567c2 nodemap: add a new mode option, with an optional "warn" value
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44740
diff changeset
147 if not can_incremental:
6493f0a567c2 nodemap: add a new mode option, with an optional "warn" value
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44740
diff changeset
148 msg = _(b"persistent nodemap in strict mode without efficient method")
6493f0a567c2 nodemap: add a new mode option, with an optional "warn" value
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44740
diff changeset
149 if mode == b'warn':
6493f0a567c2 nodemap: add a new mode option, with an optional "warn" value
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44740
diff changeset
150 tr._report(b"%s\n" % msg)
44866
6b01799e9db0 nodemap: add a new mode value, "strict"
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44865
diff changeset
151 elif mode == b'strict':
6b01799e9db0 nodemap: add a new mode value, "strict"
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44865
diff changeset
152 raise error.Abort(msg)
44367
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
153
44371
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
154 data = None
44367
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
155 # first attemp an incremental update of the data
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
156 if can_incremental and ondisk_docket is not None:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
157 target_docket = revlog._nodemap_docket.copy()
44371
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
158 (
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
159 src_docket,
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
160 data_changed_count,
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
161 data,
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
162 ) = revlog.index.nodemap_data_incremental()
44640
c70bcaf7927b nodemap: automatically "vacuum" the persistent nodemap when too sparse
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44639
diff changeset
163 new_length = target_docket.data_length + len(data)
c70bcaf7927b nodemap: automatically "vacuum" the persistent nodemap when too sparse
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44639
diff changeset
164 new_unused = target_docket.data_unused + data_changed_count
44371
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
165 if src_docket != target_docket:
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
166 data = None
44640
c70bcaf7927b nodemap: automatically "vacuum" the persistent nodemap when too sparse
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44639
diff changeset
167 elif new_length <= (new_unused * 10): # under 10% of unused data
c70bcaf7927b nodemap: automatically "vacuum" the persistent nodemap when too sparse
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44639
diff changeset
168 data = None
44371
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
169 else:
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
170 datafile = _rawdata_filepath(revlog, target_docket)
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
171 # EXP-TODO: if this is a cache, this should use a cache vfs, not a
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
172 # store vfs
44638
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44527
diff changeset
173 tr.add(datafile, target_docket.data_length)
44372
2ea6a67ff502 nodemap: write new data from the expected current data length
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44371
diff changeset
174 with revlog.opener(datafile, b'r+') as fd:
2ea6a67ff502 nodemap: write new data from the expected current data length
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44371
diff changeset
175 fd.seek(target_docket.data_length)
44371
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
176 fd.write(data)
44397
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
177 if feed_data:
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
178 if use_mmap:
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
179 fd.seek(0)
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
180 new_data = fd.read(new_length)
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
181 else:
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
182 fd.flush()
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
183 new_data = util.buffer(util.mmapread(fd, new_length))
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
184 target_docket.data_length = new_length
44640
c70bcaf7927b nodemap: automatically "vacuum" the persistent nodemap when too sparse
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44639
diff changeset
185 target_docket.data_unused = new_unused
44371
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
186
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
187 if data is None:
44367
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
188 # otherwise fallback to a full new export
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
189 target_docket = NodeMapDocket()
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
190 datafile = _rawdata_filepath(revlog, target_docket)
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
191 if util.safehasattr(revlog.index, "nodemap_data_all"):
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
192 data = revlog.index.nodemap_data_all()
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
193 else:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
194 data = persistent_data(revlog.index)
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
195 # EXP-TODO: if this is a cache, this should use a cache vfs, not a
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
196 # store vfs
44639
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44638
diff changeset
197
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44638
diff changeset
198 tryunlink = revlog.opener.tryunlink
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44638
diff changeset
199
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44638
diff changeset
200 def abortck(tr):
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44638
diff changeset
201 tryunlink(datafile)
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44638
diff changeset
202
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44638
diff changeset
203 callback_id = b"delete-%s" % datafile
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44638
diff changeset
204
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44638
diff changeset
205 # some flavor of the transaction abort does not cleanup new file, it
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44638
diff changeset
206 # simply empty them.
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44638
diff changeset
207 tr.addabort(callback_id, abortck)
44397
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
208 with revlog.opener(datafile, b'w+') as fd:
44367
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
209 fd.write(data)
44397
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
210 if feed_data:
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
211 if use_mmap:
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
212 new_data = data
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
213 else:
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
214 fd.flush()
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
215 new_data = util.buffer(util.mmapread(fd, len(data)))
44370
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
216 target_docket.data_length = len(data)
44369
e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44367
diff changeset
217 target_docket.tip_rev = revlog.tiprev()
44525
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44509
diff changeset
218 target_docket.tip_node = revlog.node(target_docket.tip_rev)
44367
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
219 # EXP-TODO: if this is a cache, this should use a cache vfs, not a
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
220 # store vfs
44526
64e2f603de9d nodemap: make sure hooks have access to an up-to-date version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44525
diff changeset
221 file_path = revlog.nodemap_file
64e2f603de9d nodemap: make sure hooks have access to an up-to-date version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44525
diff changeset
222 if pending:
64e2f603de9d nodemap: make sure hooks have access to an up-to-date version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44525
diff changeset
223 file_path += b'.a'
44638
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44527
diff changeset
224 tr.registertmp(file_path)
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44527
diff changeset
225 else:
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44527
diff changeset
226 tr.addbackup(file_path)
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44527
diff changeset
227
44526
64e2f603de9d nodemap: make sure hooks have access to an up-to-date version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44525
diff changeset
228 with revlog.opener(file_path, b'w', atomictemp=True) as fp:
44367
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
229 fp.write(target_docket.serialize())
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
230 revlog._nodemap_docket = target_docket
44397
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
231 if feed_data:
44374
6ecc34b31137 nodemap: update the index with the newly written data (when appropriate)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44373
diff changeset
232 revlog.index.update_nodemap_data(target_docket, new_data)
6ecc34b31137 nodemap: update the index with the newly written data (when appropriate)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44373
diff changeset
233
44367
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
234 # search for old index file in all cases, some older process might have
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
235 # left one behind.
44365
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
236 olds = _other_rawdata_filepath(revlog, target_docket)
44355
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44354
diff changeset
237 if olds:
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44354
diff changeset
238 realvfs = getattr(revlog, '_realopener', revlog.opener)
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44354
diff changeset
239
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44354
diff changeset
240 def cleanup(tr):
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44354
diff changeset
241 for oldfile in olds:
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44354
diff changeset
242 realvfs.tryunlink(oldfile)
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44354
diff changeset
243
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44354
diff changeset
244 callback_id = b"revlog-cleanup-nodemap-%s" % revlog.nodemap_file
44509
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44506
diff changeset
245 tr.addpostclose(callback_id, cleanup)
44354
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
246
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
247
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
248 ### Nodemap docket file
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
249 #
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
250 # The nodemap data are stored on disk using 2 files:
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
251 #
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
252 # * a raw data files containing a persistent nodemap
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
253 # (see `Nodemap Trie` section)
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
254 #
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
255 # * a small "docket" file containing medatadata
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
256 #
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
257 # While the nodemap data can be multiple tens of megabytes, the "docket" is
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
258 # small, it is easy to update it automatically or to duplicated its content
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
259 # during a transaction.
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
260 #
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
261 # Multiple raw data can exist at the same time (The currently valid one and a
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
262 # new one beind used by an in progress transaction). To accomodate this, the
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
263 # filename hosting the raw data has a variable parts. The exact filename is
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
264 # specified inside the "docket" file.
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
265 #
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
266 # The docket file contains information to find, qualify and validate the raw
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
267 # data. Its content is currently very light, but it will expand as the on disk
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
268 # nodemap gains the necessary features to be used in production.
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
269
44867
261e71752d1f nodemap: move on disk file to version 1
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44866
diff changeset
270 ONDISK_VERSION = 1
44354
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
271 S_VERSION = struct.Struct(">B")
44525
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44509
diff changeset
272 S_HEADER = struct.Struct(">BQQQQ")
44354
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
273
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
274 ID_SIZE = 8
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
275
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
276
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
277 def _make_uid():
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
278 """return a new unique identifier.
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
279
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
280 The identifier is random and composed of ascii characters."""
46114
59fa3890d40a node: import symbols explicitly
Joerg Sonnenberger <joerg@bec.de>
parents: 46090
diff changeset
281 return hex(os.urandom(ID_SIZE))
44354
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
282
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
283
44365
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
284 class NodeMapDocket(object):
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
285 """metadata associated with persistent nodemap data
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
286
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
287 The persistent data may come from disk or be on their way to disk.
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
288 """
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
289
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
290 def __init__(self, uid=None):
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
291 if uid is None:
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
292 uid = _make_uid()
44506
283fd803afa5 nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44480
diff changeset
293 # a unique identifier for the data file:
283fd803afa5 nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44480
diff changeset
294 # - When new data are appended, it is preserved.
283fd803afa5 nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44480
diff changeset
295 # - When a new data file is created, a new identifier is generated.
44365
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
296 self.uid = uid
44506
283fd803afa5 nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44480
diff changeset
297 # the tipmost revision stored in the data file. This revision and all
283fd803afa5 nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44480
diff changeset
298 # revision before it are expected to be encoded in the data file.
44369
e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44367
diff changeset
299 self.tip_rev = None
44525
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44509
diff changeset
300 # the node of that tipmost revision, if it mismatch the current index
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44509
diff changeset
301 # data the docket is not valid for the current index and should be
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44509
diff changeset
302 # discarded.
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44509
diff changeset
303 #
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44509
diff changeset
304 # note: this method is not perfect as some destructive operation could
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44509
diff changeset
305 # preserve the same tip_rev + tip_node while altering lower revision.
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44509
diff changeset
306 # However this multiple other caches have the same vulnerability (eg:
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44509
diff changeset
307 # brancmap cache).
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44509
diff changeset
308 self.tip_node = None
44506
283fd803afa5 nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44480
diff changeset
309 # the size (in bytes) of the persisted data to encode the nodemap valid
283fd803afa5 nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44480
diff changeset
310 # for `tip_rev`.
283fd803afa5 nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44480
diff changeset
311 # - data file shorter than this are corrupted,
283fd803afa5 nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44480
diff changeset
312 # - any extra data should be ignored.
44370
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
313 self.data_length = None
44506
283fd803afa5 nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44480
diff changeset
314 # the amount (in bytes) of "dead" data, still in the data file but no
283fd803afa5 nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44480
diff changeset
315 # longer used for the nodemap.
44370
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
316 self.data_unused = 0
44365
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
317
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
318 def copy(self):
44369
e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44367
diff changeset
319 new = NodeMapDocket(uid=self.uid)
e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44367
diff changeset
320 new.tip_rev = self.tip_rev
44525
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44509
diff changeset
321 new.tip_node = self.tip_node
44370
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
322 new.data_length = self.data_length
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
323 new.data_unused = self.data_unused
44369
e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44367
diff changeset
324 return new
44365
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
325
44371
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
326 def __cmp__(self, other):
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
327 if self.uid < other.uid:
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
328 return -1
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
329 if self.uid > other.uid:
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
330 return 1
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
331 elif self.data_length < other.data_length:
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
332 return -1
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
333 elif self.data_length > other.data_length:
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
334 return 1
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
335 return 0
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
336
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
337 def __eq__(self, other):
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
338 return self.uid == other.uid and self.data_length == other.data_length
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
339
44365
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
340 def serialize(self):
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
341 """return serialized bytes for a docket using the passed uid"""
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
342 data = []
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
343 data.append(S_VERSION.pack(ONDISK_VERSION))
44370
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
344 headers = (
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
345 len(self.uid),
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
346 self.tip_rev,
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
347 self.data_length,
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
348 self.data_unused,
44525
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44509
diff changeset
349 len(self.tip_node),
44370
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
350 )
44369
e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44367
diff changeset
351 data.append(S_HEADER.pack(*headers))
44365
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
352 data.append(self.uid)
44525
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44509
diff changeset
353 data.append(self.tip_node)
44365
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
354 return b''.join(data)
44354
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
355
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
356
44365
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
357 def _rawdata_filepath(revlog, docket):
44354
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
358 """The (vfs relative) nodemap's rawdata file for a given uid"""
44526
64e2f603de9d nodemap: make sure hooks have access to an up-to-date version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44525
diff changeset
359 if revlog.nodemap_file.endswith(b'.n.a'):
64e2f603de9d nodemap: make sure hooks have access to an up-to-date version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44525
diff changeset
360 prefix = revlog.nodemap_file[:-4]
64e2f603de9d nodemap: make sure hooks have access to an up-to-date version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44525
diff changeset
361 else:
64e2f603de9d nodemap: make sure hooks have access to an up-to-date version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44525
diff changeset
362 prefix = revlog.nodemap_file[:-2]
44365
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
363 return b"%s-%s.nd" % (prefix, docket.uid)
44351
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44350
diff changeset
364
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44350
diff changeset
365
44365
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
366 def _other_rawdata_filepath(revlog, docket):
44355
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44354
diff changeset
367 prefix = revlog.nodemap_file[:-2]
44480
6aee0647e026 nodemap: fix missing r-prefix on regular expression
Augie Fackler <augie@google.com>
parents: 44465
diff changeset
368 pattern = re.compile(br"(^|/)%s-[0-9a-f]+\.nd$" % prefix)
44365
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
369 new_file_path = _rawdata_filepath(revlog, docket)
44355
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44354
diff changeset
370 new_file_name = revlog.opener.basename(new_file_path)
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44354
diff changeset
371 dirpath = revlog.opener.dirname(new_file_path)
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44354
diff changeset
372 others = []
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44354
diff changeset
373 for f in revlog.opener.listdir(dirpath):
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44354
diff changeset
374 if pattern.match(f) and f != new_file_name:
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44354
diff changeset
375 others.append(f)
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44354
diff changeset
376 return others
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44354
diff changeset
377
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44354
diff changeset
378
44350
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
379 ### Nodemap Trie
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
380 #
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
381 # This is a simple reference implementation to compute and persist a nodemap
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
382 # trie. This reference implementation is write only. The python version of this
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
383 # is not expected to be actually used, since it wont provide performance
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
384 # improvement over existing non-persistent C implementation.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
385 #
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
386 # The nodemap is persisted as Trie using 4bits-address/16-entries block. each
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
387 # revision can be adressed using its node shortest prefix.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
388 #
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
389 # The trie is stored as a sequence of block. Each block contains 16 entries
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
390 # (signed 64bit integer, big endian). Each entry can be one of the following:
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
391 #
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
392 # * value >= 0 -> index of sub-block
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
393 # * value == -1 -> no value
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
394 # * value < -1 -> a revision value: rev = -(value+10)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
395 #
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
396 # The implementation focus on simplicity, not on performance. A Rust
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
397 # implementation should provide a efficient version of the same binary
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
398 # persistence. This reference python implementation is never meant to be
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
399 # extensively use in production.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
400
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
401
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
402 def persistent_data(index):
45957
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 44869
diff changeset
403 """return the persistent binary form for a nodemap for a given index"""
44350
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
404 trie = _build_trie(index)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
405 return _persist_trie(trie)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
406
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
407
44367
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
408 def update_persistent_data(index, root, max_idx, last_rev):
45957
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 44869
diff changeset
409 """return the incremental update for persistent nodemap from a given index"""
44370
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
410 changed_block, trie = _update_trie(index, root, last_rev)
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
411 return (
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
412 changed_block * S_BLOCK.size,
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
413 _persist_trie(trie, existing_idx=max_idx),
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
414 )
44367
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
415
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
416
44350
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
417 S_BLOCK = struct.Struct(">" + ("l" * 16))
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
418
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
419 NO_ENTRY = -1
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
420 # rev 0 need to be -2 because 0 is used by block, -1 is a special value.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
421 REV_OFFSET = 2
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
422
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
423
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
424 def _transform_rev(rev):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
425 """Return the number used to represent the rev in the tree.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
426
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
427 (or retrieve a rev number from such representation)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
428
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
429 Note that this is an involution, a function equal to its inverse (i.e.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
430 which gives the identity when applied to itself).
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
431 """
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
432 return -(rev + REV_OFFSET)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
433
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
434
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
435 def _to_int(hex_digit):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
436 """turn an hexadecimal digit into a proper integer"""
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
437 return int(hex_digit, 16)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
438
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
439
44358
7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44357
diff changeset
440 class Block(dict):
7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44357
diff changeset
441 """represent a block of the Trie
7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44357
diff changeset
442
7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44357
diff changeset
443 contains up to 16 entry indexed from 0 to 15"""
7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44357
diff changeset
444
44364
f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44362
diff changeset
445 def __init__(self):
f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44362
diff changeset
446 super(Block, self).__init__()
f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44362
diff changeset
447 # If this block exist on disk, here is its ID
f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44362
diff changeset
448 self.ondisk_id = None
f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44362
diff changeset
449
44359
55b12f2593c1 nodemap: move the iteratio inside the Block object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44358
diff changeset
450 def __iter__(self):
55b12f2593c1 nodemap: move the iteratio inside the Block object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44358
diff changeset
451 return iter(self.get(i) for i in range(16))
55b12f2593c1 nodemap: move the iteratio inside the Block object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44358
diff changeset
452
44358
7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44357
diff changeset
453
44350
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
454 def _build_trie(index):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
455 """build a nodemap trie
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
456
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
457 The nodemap stores revision number for each unique prefix.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
458
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
459 Each block is a dictionary with keys in `[0, 15]`. Values are either
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
460 another block or a revision number.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
461 """
44358
7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44357
diff changeset
462 root = Block()
44350
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
463 for rev in range(len(index)):
46114
59fa3890d40a node: import symbols explicitly
Joerg Sonnenberger <joerg@bec.de>
parents: 46090
diff changeset
464 current_hex = hex(index[rev][7])
59fa3890d40a node: import symbols explicitly
Joerg Sonnenberger <joerg@bec.de>
parents: 46090
diff changeset
465 _insert_into_block(index, 0, root, rev, current_hex)
44350
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
466 return root
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
467
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
468
44367
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
469 def _update_trie(index, root, last_rev):
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
470 """consume"""
44370
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
471 changed = 0
44367
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
472 for rev in range(last_rev + 1, len(index)):
46114
59fa3890d40a node: import symbols explicitly
Joerg Sonnenberger <joerg@bec.de>
parents: 46090
diff changeset
473 current_hex = hex(index[rev][7])
59fa3890d40a node: import symbols explicitly
Joerg Sonnenberger <joerg@bec.de>
parents: 46090
diff changeset
474 changed += _insert_into_block(index, 0, root, rev, current_hex)
44370
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
475 return changed, root
44367
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
476
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
477
44350
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
478 def _insert_into_block(index, level, block, current_rev, current_hex):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
479 """insert a new revision in a block
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
480
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
481 index: the index we are adding revision for
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
482 level: the depth of the current block in the trie
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
483 block: the block currently being considered
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
484 current_rev: the revision number we are adding
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
485 current_hex: the hexadecimal representation of the of that revision
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
486 """
44370
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
487 changed = 1
44367
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
488 if block.ondisk_id is not None:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
489 block.ondisk_id = None
44350
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
490 hex_digit = _to_int(current_hex[level : level + 1])
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
491 entry = block.get(hex_digit)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
492 if entry is None:
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
493 # no entry, simply store the revision number
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
494 block[hex_digit] = current_rev
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
495 elif isinstance(entry, dict):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
496 # need to recurse to an underlying block
44370
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
497 changed += _insert_into_block(
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
498 index, level + 1, entry, current_rev, current_hex
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
499 )
44350
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
500 else:
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
501 # collision with a previously unique prefix, inserting new
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
502 # vertices to fit both entry.
46114
59fa3890d40a node: import symbols explicitly
Joerg Sonnenberger <joerg@bec.de>
parents: 46090
diff changeset
503 other_hex = hex(index[entry][7])
44350
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
504 other_rev = entry
44358
7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44357
diff changeset
505 new = Block()
44350
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
506 block[hex_digit] = new
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
507 _insert_into_block(index, level + 1, new, other_rev, other_hex)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
508 _insert_into_block(index, level + 1, new, current_rev, current_hex)
44370
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
509 return changed
44350
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
510
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
511
44367
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
512 def _persist_trie(root, existing_idx=None):
44350
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
513 """turn a nodemap trie into persistent binary data
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
514
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
515 See `_build_trie` for nodemap trie structure"""
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
516 block_map = {}
44367
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
517 if existing_idx is not None:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
518 base_idx = existing_idx + 1
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
519 else:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
520 base_idx = 0
44350
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
521 chunks = []
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
522 for tn in _walk_trie(root):
44367
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
523 if tn.ondisk_id is not None:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
524 block_map[id(tn)] = tn.ondisk_id
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
525 else:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
526 block_map[id(tn)] = len(chunks) + base_idx
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
527 chunks.append(_persist_block(tn, block_map))
44350
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
528 return b''.join(chunks)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
529
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
530
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
531 def _walk_trie(block):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
532 """yield all the block in a trie
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
533
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
534 Children blocks are always yield before their parent block.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
535 """
44696
ed475420e0af nodemap: rename a variable to prevent shawoding "_"
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44640
diff changeset
536 for (__, item) in sorted(block.items()):
44350
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
537 if isinstance(item, dict):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
538 for sub_block in _walk_trie(item):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
539 yield sub_block
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
540 yield block
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
541
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
542
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
543 def _persist_block(block_node, block_map):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
544 """produce persistent binary data for a single block
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
545
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
546 Children block are assumed to be already persisted and present in
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
547 block_map.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
548 """
44359
55b12f2593c1 nodemap: move the iteratio inside the Block object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44358
diff changeset
549 data = tuple(_to_value(v, block_map) for v in block_node)
44350
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
550 return S_BLOCK.pack(*data)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
551
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
552
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
553 def _to_value(item, block_map):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
554 """persist any value as an integer"""
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
555 if item is None:
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
556 return NO_ENTRY
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
557 elif isinstance(item, dict):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
558 return block_map[id(item)]
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
559 else:
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
560 return _transform_rev(item)
44360
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44359
diff changeset
561
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44359
diff changeset
562
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44359
diff changeset
563 def parse_data(data):
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44359
diff changeset
564 """parse parse nodemap data into a nodemap Trie"""
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44359
diff changeset
565 if (len(data) % S_BLOCK.size) != 0:
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44359
diff changeset
566 msg = "nodemap data size is not a multiple of block size (%d): %d"
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44359
diff changeset
567 raise error.Abort(msg % (S_BLOCK.size, len(data)))
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44359
diff changeset
568 if not data:
44367
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
569 return Block(), None
44360
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44359
diff changeset
570 block_map = {}
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44359
diff changeset
571 new_blocks = []
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44359
diff changeset
572 for i in range(0, len(data), S_BLOCK.size):
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44359
diff changeset
573 block = Block()
44364
f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44362
diff changeset
574 block.ondisk_id = len(block_map)
f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44362
diff changeset
575 block_map[block.ondisk_id] = block
44360
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44359
diff changeset
576 block_data = data[i : i + S_BLOCK.size]
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44359
diff changeset
577 values = S_BLOCK.unpack(block_data)
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44359
diff changeset
578 new_blocks.append((block, values))
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44359
diff changeset
579 for b, values in new_blocks:
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44359
diff changeset
580 for idx, v in enumerate(values):
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44359
diff changeset
581 if v == NO_ENTRY:
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44359
diff changeset
582 continue
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44359
diff changeset
583 elif v >= 0:
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44359
diff changeset
584 b[idx] = block_map[v]
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44359
diff changeset
585 else:
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44359
diff changeset
586 b[idx] = _transform_rev(v)
44367
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
587 return block, i // S_BLOCK.size
44361
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
588
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
589
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
590 # debug utility
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
591
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
592
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
593 def check_data(ui, index, data):
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
594 """verify that the provided nodemap data are valid for the given idex"""
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
595 ret = 0
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
596 ui.status((b"revision in index: %d\n") % len(index))
44367
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
597 root, __ = parse_data(data)
44361
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
598 all_revs = set(_all_revisions(root))
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
599 ui.status((b"revision in nodemap: %d\n") % len(all_revs))
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
600 for r in range(len(index)):
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
601 if r not in all_revs:
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
602 msg = b" revision missing from nodemap: %d\n" % r
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
603 ui.write_err(msg)
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
604 ret = 1
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
605 else:
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
606 all_revs.remove(r)
46114
59fa3890d40a node: import symbols explicitly
Joerg Sonnenberger <joerg@bec.de>
parents: 46090
diff changeset
607 nm_rev = _find_node(root, hex(index[r][7]))
44362
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44361
diff changeset
608 if nm_rev is None:
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44361
diff changeset
609 msg = b" revision node does not match any entries: %d\n" % r
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44361
diff changeset
610 ui.write_err(msg)
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44361
diff changeset
611 ret = 1
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44361
diff changeset
612 elif nm_rev != r:
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44361
diff changeset
613 msg = (
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44361
diff changeset
614 b" revision node does not match the expected revision: "
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44361
diff changeset
615 b"%d != %d\n" % (r, nm_rev)
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44361
diff changeset
616 )
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44361
diff changeset
617 ui.write_err(msg)
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44361
diff changeset
618 ret = 1
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44361
diff changeset
619
44361
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
620 if all_revs:
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
621 for r in sorted(all_revs):
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
622 msg = b" extra revision in nodemap: %d\n" % r
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
623 ui.write_err(msg)
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
624 ret = 1
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
625 return ret
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
626
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
627
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
628 def _all_revisions(root):
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
629 """return all revisions stored in a Trie"""
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
630 for block in _walk_trie(root):
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
631 for v in block:
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
632 if v is None or isinstance(v, Block):
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
633 continue
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
634 yield v
44362
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44361
diff changeset
635
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44361
diff changeset
636
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44361
diff changeset
637 def _find_node(block, node):
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44361
diff changeset
638 """find the revision associated with a given node"""
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44361
diff changeset
639 entry = block.get(_to_int(node[0:1]))
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44361
diff changeset
640 if isinstance(entry, dict):
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44361
diff changeset
641 return _find_node(entry, node[1:])
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44361
diff changeset
642 return entry