Mercurial > public > mercurial-scm > hg-stable

annotate mercurial/revlogutils/nodemap.py @ 44866:6b01799e9db0

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
nodemap: add a new mode value, "strict" When "strict" is set, operation will refuse to use the slow path and abort if they would. This is useful for testing, benchmarking and server operation. Differential Revision: https://phab.mercurial-scm.org/D8415
author Pierre-Yves David <pierre-yves.david@octobus.net>
date Tue, 14 Apr 2020 03:01:52 +0200
parents 6493f0a567c2
children 261e71752d1f
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
44035
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
1 # nodemap.py - nodemap related code and utilities
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
2 #
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
3 # Copyright 2019 Pierre-Yves David <pierre-yves.david@octobus.net>
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
4 # Copyright 2019 George Racinet <georges.racinet@octobus.net>
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
5 #
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
6 # This software may be used and distributed according to the terms of the
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
7 # GNU General Public License version 2 or any later version.
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
8
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
9 from __future__ import absolute_import
44350
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
10
44397
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
11 import errno
44354
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
12 import os
44355
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44354
diff changeset
13 import re
44350
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
14 import struct
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
15
44865
6493f0a567c2 nodemap: add a new mode option, with an optional "warn" value
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44740
diff changeset
16 from ..i18n import _
6493f0a567c2 nodemap: add a new mode option, with an optional "warn" value
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44740
diff changeset
17
44350
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
18 from .. import (
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
19 error,
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
20 node as nodemod,
44357
7f4f7ef3133e nodemap: add a optional `nodemap_add_full` method on indexes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44355
diff changeset
21 util,
44350
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
22 )
44035
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
23
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
24
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
25 class NodeMap(dict):
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
26 def __missing__(self, x):
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
27 raise error.RevlogError(b'unknown node: %s' % x)
44350
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
28
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
29
44352
6c07480d6659 nodemap: add a function to read the data from disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44351
diff changeset
30 def persisted_data(revlog):
6c07480d6659 nodemap: add a function to read the data from disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44351
diff changeset
31 """read the nodemap for a revlog from disk"""
6c07480d6659 nodemap: add a function to read the data from disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44351
diff changeset
32 if revlog.nodemap_file is None:
6c07480d6659 nodemap: add a function to read the data from disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44351
diff changeset
33 return None
44354
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
34 pdata = revlog.opener.tryread(revlog.nodemap_file)
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
35 if not pdata:
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
36 return None
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
37 offset = 0
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
38 (version,) = S_VERSION.unpack(pdata[offset : offset + S_VERSION.size])
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
39 if version != ONDISK_VERSION:
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
40 return None
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
41 offset += S_VERSION.size
44369
e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44367
diff changeset
42 headers = S_HEADER.unpack(pdata[offset : offset + S_HEADER.size])
44525
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44509
diff changeset
43 uid_size, tip_rev, data_length, data_unused, tip_node_size = headers
44354
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
44 offset += S_HEADER.size
44365
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
45 docket = NodeMapDocket(pdata[offset : offset + uid_size])
44525
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44509
diff changeset
46 offset += uid_size
44369
e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44367
diff changeset
47 docket.tip_rev = tip_rev
44525
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44509
diff changeset
48 docket.tip_node = pdata[offset : offset + tip_node_size]
44370
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
49 docket.data_length = data_length
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
50 docket.data_unused = data_unused
44354
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
51
44365
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
52 filename = _rawdata_filepath(revlog, docket)
44697
0792ad55e533 nodemap: access the mmap opener option using bytes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44696
diff changeset
53 use_mmap = revlog.opener.options.get(b"exp-persistent-nodemap.mmap")
44397
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
54 try:
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
55 with revlog.opener(filename) as fd:
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
56 if use_mmap:
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
57 data = util.buffer(util.mmapread(fd, data_length))
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
58 else:
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
59 data = fd.read(data_length)
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
60 except OSError as e:
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
61 if e.errno != errno.ENOENT:
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
62 raise
44373
c7eebdb15139 nodemap: never read more than the expected data amount
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44372
diff changeset
63 if len(data) < data_length:
c7eebdb15139 nodemap: never read more than the expected data amount
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44372
diff changeset
64 return None
c7eebdb15139 nodemap: never read more than the expected data amount
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44372
diff changeset
65 return docket, data
44352
6c07480d6659 nodemap: add a function to read the data from disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44351
diff changeset
66
6c07480d6659 nodemap: add a function to read the data from disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44351
diff changeset
67
44351
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44350
diff changeset
68 def setup_persistent_nodemap(tr, revlog):
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44350
diff changeset
69 """Install whatever is needed transaction side to persist a nodemap on disk
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44350
diff changeset
70
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44350
diff changeset
71 (only actually persist the nodemap if this is relevant for this revlog)
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44350
diff changeset
72 """
44353
daad3aace942 nodemap: only use persistent nodemap for non-inlined revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44352
diff changeset
73 if revlog._inline:
daad3aace942 nodemap: only use persistent nodemap for non-inlined revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44352
diff changeset
74 return # inlined revlog are too small for this to be relevant
44351
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44350
diff changeset
75 if revlog.nodemap_file is None:
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44350
diff changeset
76 return # we do not use persistent_nodemap on this revlog
44527
448d700e0d27 nodemap: make sure the nodemap docket is updated after the changelog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44526
diff changeset
77
448d700e0d27 nodemap: make sure the nodemap docket is updated after the changelog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44526
diff changeset
78 # we need to happen after the changelog finalization, in that use "cl-"
448d700e0d27 nodemap: make sure the nodemap docket is updated after the changelog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44526
diff changeset
79 callback_id = b"nm-revlog-persistent-nodemap-%s" % revlog.nodemap_file
44351
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44350
diff changeset
80 if tr.hasfinalize(callback_id):
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44350
diff changeset
81 return # no need to register again
44526
64e2f603de9d nodemap: make sure hooks have access to an up-to-date version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44525
diff changeset
82 tr.addpending(
64e2f603de9d nodemap: make sure hooks have access to an up-to-date version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44525
diff changeset
83 callback_id, lambda tr: _persist_nodemap(tr, revlog, pending=True)
64e2f603de9d nodemap: make sure hooks have access to an up-to-date version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44525
diff changeset
84 )
44509
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44506
diff changeset
85 tr.addfinalize(callback_id, lambda tr: _persist_nodemap(tr, revlog))
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44506
diff changeset
86
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44506
diff changeset
87
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44506
diff changeset
88 class _NoTransaction(object):
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44506
diff changeset
89 """transaction like object to update the nodemap outside a transaction
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44506
diff changeset
90 """
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44506
diff changeset
91
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44506
diff changeset
92 def __init__(self):
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44506
diff changeset
93 self._postclose = {}
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44506
diff changeset
94
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44506
diff changeset
95 def addpostclose(self, callback_id, callback_func):
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44506
diff changeset
96 self._postclose[callback_id] = callback_func
44351
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44350
diff changeset
97
44638
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44527
diff changeset
98 def registertmp(self, *args, **kwargs):
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44527
diff changeset
99 pass
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44527
diff changeset
100
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44527
diff changeset
101 def addbackup(self, *args, **kwargs):
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44527
diff changeset
102 pass
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44527
diff changeset
103
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44527
diff changeset
104 def add(self, *args, **kwargs):
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44527
diff changeset
105 pass
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44527
diff changeset
106
44639
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44638
diff changeset
107 def addabort(self, *args, **kwargs):
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44638
diff changeset
108 pass
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44638
diff changeset
109
44865
6493f0a567c2 nodemap: add a new mode option, with an optional "warn" value
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44740
diff changeset
110 def _report(self, *args):
6493f0a567c2 nodemap: add a new mode option, with an optional "warn" value
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44740
diff changeset
111 pass
6493f0a567c2 nodemap: add a new mode option, with an optional "warn" value
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44740
diff changeset
112
44351
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44350
diff changeset
113
44465
336ec75ed1ac nodemap: warm the persistent nodemap on disk with debugupdatecache
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44397
diff changeset
114 def update_persistent_nodemap(revlog):
336ec75ed1ac nodemap: warm the persistent nodemap on disk with debugupdatecache
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44397
diff changeset
115 """update the persistent nodemap right now
336ec75ed1ac nodemap: warm the persistent nodemap on disk with debugupdatecache
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44397
diff changeset
116
336ec75ed1ac nodemap: warm the persistent nodemap on disk with debugupdatecache
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44397
diff changeset
117 To be used for updating the nodemap on disk outside of a normal transaction
336ec75ed1ac nodemap: warm the persistent nodemap on disk with debugupdatecache
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44397
diff changeset
118 setup (eg, `debugupdatecache`).
336ec75ed1ac nodemap: warm the persistent nodemap on disk with debugupdatecache
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44397
diff changeset
119 """
44740
b4537125eb3c nodemap: skip persistent nodemap warming for revlog not using it
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44697
diff changeset
120 if revlog._inline:
b4537125eb3c nodemap: skip persistent nodemap warming for revlog not using it
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44697
diff changeset
121 return # inlined revlog are too small for this to be relevant
b4537125eb3c nodemap: skip persistent nodemap warming for revlog not using it
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44697
diff changeset
122 if revlog.nodemap_file is None:
b4537125eb3c nodemap: skip persistent nodemap warming for revlog not using it
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44697
diff changeset
123 return # we do not use persistent_nodemap on this revlog
b4537125eb3c nodemap: skip persistent nodemap warming for revlog not using it
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44697
diff changeset
124
44509
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44506
diff changeset
125 notr = _NoTransaction()
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44506
diff changeset
126 _persist_nodemap(notr, revlog)
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44506
diff changeset
127 for k in sorted(notr._postclose):
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44506
diff changeset
128 notr._postclose[k](None)
44465
336ec75ed1ac nodemap: warm the persistent nodemap on disk with debugupdatecache
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44397
diff changeset
129
336ec75ed1ac nodemap: warm the persistent nodemap on disk with debugupdatecache
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44397
diff changeset
130
44526
64e2f603de9d nodemap: make sure hooks have access to an up-to-date version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44525
diff changeset
131 def _persist_nodemap(tr, revlog, pending=False):
44351
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44350
diff changeset
132 """Write nodemap data on disk for a given revlog
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44350
diff changeset
133 """
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44350
diff changeset
134 if getattr(revlog, 'filteredrevs', ()):
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44350
diff changeset
135 raise error.ProgrammingError(
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44350
diff changeset
136 "cannot persist nodemap of a filtered changelog"
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44350
diff changeset
137 )
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44350
diff changeset
138 if revlog.nodemap_file is None:
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44350
diff changeset
139 msg = "calling persist nodemap on a revlog without the feature enableb"
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44350
diff changeset
140 raise error.ProgrammingError(msg)
44367
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
141
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
142 can_incremental = util.safehasattr(revlog.index, "nodemap_data_incremental")
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
143 ondisk_docket = revlog._nodemap_docket
44397
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
144 feed_data = util.safehasattr(revlog.index, "update_nodemap_data")
44697
0792ad55e533 nodemap: access the mmap opener option using bytes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44696
diff changeset
145 use_mmap = revlog.opener.options.get(b"exp-persistent-nodemap.mmap")
44865
6493f0a567c2 nodemap: add a new mode option, with an optional "warn" value
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44740
diff changeset
146 mode = revlog.opener.options.get(b"exp-persistent-nodemap.mode")
6493f0a567c2 nodemap: add a new mode option, with an optional "warn" value
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44740
diff changeset
147 if not can_incremental:
6493f0a567c2 nodemap: add a new mode option, with an optional "warn" value
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44740
diff changeset
148 msg = _(b"persistent nodemap in strict mode without efficient method")
6493f0a567c2 nodemap: add a new mode option, with an optional "warn" value
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44740
diff changeset
149 if mode == b'warn':
6493f0a567c2 nodemap: add a new mode option, with an optional "warn" value
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44740
diff changeset
150 tr._report(b"%s\n" % msg)
44866
6b01799e9db0 nodemap: add a new mode value, "strict"
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44865
diff changeset
151 elif mode == b'strict':
6b01799e9db0 nodemap: add a new mode value, "strict"
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44865
diff changeset
152 raise error.Abort(msg)
44367
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
153
44371
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
154 data = None
44367
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
155 # first attemp an incremental update of the data
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
156 if can_incremental and ondisk_docket is not None:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
157 target_docket = revlog._nodemap_docket.copy()
44371
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
158 (
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
159 src_docket,
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
160 data_changed_count,
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
161 data,
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
162 ) = revlog.index.nodemap_data_incremental()
44640
c70bcaf7927b nodemap: automatically "vacuum" the persistent nodemap when too sparse
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44639
diff changeset
163 new_length = target_docket.data_length + len(data)
c70bcaf7927b nodemap: automatically "vacuum" the persistent nodemap when too sparse
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44639
diff changeset
164 new_unused = target_docket.data_unused + data_changed_count
44371
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
165 if src_docket != target_docket:
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
166 data = None
44640
c70bcaf7927b nodemap: automatically "vacuum" the persistent nodemap when too sparse
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44639
diff changeset
167 elif new_length <= (new_unused * 10): # under 10% of unused data
c70bcaf7927b nodemap: automatically "vacuum" the persistent nodemap when too sparse
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44639
diff changeset
168 data = None
44371
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
169 else:
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
170 datafile = _rawdata_filepath(revlog, target_docket)
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
171 # EXP-TODO: if this is a cache, this should use a cache vfs, not a
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
172 # store vfs
44638
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44527
diff changeset
173 tr.add(datafile, target_docket.data_length)
44372
2ea6a67ff502 nodemap: write new data from the expected current data length
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44371
diff changeset
174 with revlog.opener(datafile, b'r+') as fd:
2ea6a67ff502 nodemap: write new data from the expected current data length
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44371
diff changeset
175 fd.seek(target_docket.data_length)
44371
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
176 fd.write(data)
44397
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
177 if feed_data:
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
178 if use_mmap:
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
179 fd.seek(0)
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
180 new_data = fd.read(new_length)
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
181 else:
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
182 fd.flush()
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
183 new_data = util.buffer(util.mmapread(fd, new_length))
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
184 target_docket.data_length = new_length
44640
c70bcaf7927b nodemap: automatically "vacuum" the persistent nodemap when too sparse
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44639
diff changeset
185 target_docket.data_unused = new_unused
44371
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
186
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
187 if data is None:
44367
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
188 # otherwise fallback to a full new export
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
189 target_docket = NodeMapDocket()
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
190 datafile = _rawdata_filepath(revlog, target_docket)
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
191 if util.safehasattr(revlog.index, "nodemap_data_all"):
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
192 data = revlog.index.nodemap_data_all()
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
193 else:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
194 data = persistent_data(revlog.index)
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
195 # EXP-TODO: if this is a cache, this should use a cache vfs, not a
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
196 # store vfs
44639
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44638
diff changeset
197
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44638
diff changeset
198 tryunlink = revlog.opener.tryunlink
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44638
diff changeset
199
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44638
diff changeset
200 def abortck(tr):
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44638
diff changeset
201 tryunlink(datafile)
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44638
diff changeset
202
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44638
diff changeset
203 callback_id = b"delete-%s" % datafile
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44638
diff changeset
204
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44638
diff changeset
205 # some flavor of the transaction abort does not cleanup new file, it
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44638
diff changeset
206 # simply empty them.
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44638
diff changeset
207 tr.addabort(callback_id, abortck)
44397
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
208 with revlog.opener(datafile, b'w+') as fd:
44367
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
209 fd.write(data)
44397
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
210 if feed_data:
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
211 if use_mmap:
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
212 new_data = data
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
213 else:
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
214 fd.flush()
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
215 new_data = util.buffer(util.mmapread(fd, len(data)))
44370
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
216 target_docket.data_length = len(data)
44369
e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44367
diff changeset
217 target_docket.tip_rev = revlog.tiprev()
44525
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44509
diff changeset
218 target_docket.tip_node = revlog.node(target_docket.tip_rev)
44367
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
219 # EXP-TODO: if this is a cache, this should use a cache vfs, not a
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
220 # store vfs
44526
64e2f603de9d nodemap: make sure hooks have access to an up-to-date version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44525
diff changeset
221 file_path = revlog.nodemap_file
64e2f603de9d nodemap: make sure hooks have access to an up-to-date version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44525
diff changeset
222 if pending:
64e2f603de9d nodemap: make sure hooks have access to an up-to-date version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44525
diff changeset
223 file_path += b'.a'
44638
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44527
diff changeset
224 tr.registertmp(file_path)
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44527
diff changeset
225 else:
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44527
diff changeset
226 tr.addbackup(file_path)
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44527
diff changeset
227
44526
64e2f603de9d nodemap: make sure hooks have access to an up-to-date version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44525
diff changeset
228 with revlog.opener(file_path, b'w', atomictemp=True) as fp:
44367
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
229 fp.write(target_docket.serialize())
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
230 revlog._nodemap_docket = target_docket
44397
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
231 if feed_data:
44374
6ecc34b31137 nodemap: update the index with the newly written data (when appropriate)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44373
diff changeset
232 revlog.index.update_nodemap_data(target_docket, new_data)
6ecc34b31137 nodemap: update the index with the newly written data (when appropriate)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44373
diff changeset
233
44367
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
234 # search for old index file in all cases, some older process might have
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
235 # left one behind.
44365
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
236 olds = _other_rawdata_filepath(revlog, target_docket)
44355
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44354
diff changeset
237 if olds:
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44354
diff changeset
238 realvfs = getattr(revlog, '_realopener', revlog.opener)
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44354
diff changeset
239
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44354
diff changeset
240 def cleanup(tr):
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44354
diff changeset
241 for oldfile in olds:
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44354
diff changeset
242 realvfs.tryunlink(oldfile)
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44354
diff changeset
243
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44354
diff changeset
244 callback_id = b"revlog-cleanup-nodemap-%s" % revlog.nodemap_file
44509
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44506
diff changeset
245 tr.addpostclose(callback_id, cleanup)
44354
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
246
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
247
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
248 ### Nodemap docket file
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
249 #
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
250 # The nodemap data are stored on disk using 2 files:
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
251 #
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
252 # * a raw data files containing a persistent nodemap
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
253 # (see `Nodemap Trie` section)
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
254 #
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
255 # * a small "docket" file containing medatadata
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
256 #
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
257 # While the nodemap data can be multiple tens of megabytes, the "docket" is
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
258 # small, it is easy to update it automatically or to duplicated its content
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
259 # during a transaction.
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
260 #
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
261 # Multiple raw data can exist at the same time (The currently valid one and a
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
262 # new one beind used by an in progress transaction). To accomodate this, the
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
263 # filename hosting the raw data has a variable parts. The exact filename is
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
264 # specified inside the "docket" file.
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
265 #
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
266 # The docket file contains information to find, qualify and validate the raw
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
267 # data. Its content is currently very light, but it will expand as the on disk
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
268 # nodemap gains the necessary features to be used in production.
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
269
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
270 # version 0 is experimental, no BC garantee, do no use outside of tests.
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
271 ONDISK_VERSION = 0
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
272 S_VERSION = struct.Struct(">B")
44525
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44509
diff changeset
273 S_HEADER = struct.Struct(">BQQQQ")
44354
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
274
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
275 ID_SIZE = 8
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
276
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
277
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
278 def _make_uid():
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
279 """return a new unique identifier.
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
280
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
281 The identifier is random and composed of ascii characters."""
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
282 return nodemod.hex(os.urandom(ID_SIZE))
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
283
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
284
44365
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
285 class NodeMapDocket(object):
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
286 """metadata associated with persistent nodemap data
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
287
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
288 The persistent data may come from disk or be on their way to disk.
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
289 """
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
290
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
291 def __init__(self, uid=None):
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
292 if uid is None:
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
293 uid = _make_uid()
44506
283fd803afa5 nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44480
diff changeset
294 # a unique identifier for the data file:
283fd803afa5 nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44480
diff changeset
295 # - When new data are appended, it is preserved.
283fd803afa5 nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44480
diff changeset
296 # - When a new data file is created, a new identifier is generated.
44365
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
297 self.uid = uid
44506
283fd803afa5 nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44480
diff changeset
298 # the tipmost revision stored in the data file. This revision and all
283fd803afa5 nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44480
diff changeset
299 # revision before it are expected to be encoded in the data file.
44369
e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44367
diff changeset
300 self.tip_rev = None
44525
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44509
diff changeset
301 # the node of that tipmost revision, if it mismatch the current index
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44509
diff changeset
302 # data the docket is not valid for the current index and should be
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44509
diff changeset
303 # discarded.
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44509
diff changeset
304 #
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44509
diff changeset
305 # note: this method is not perfect as some destructive operation could
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44509
diff changeset
306 # preserve the same tip_rev + tip_node while altering lower revision.
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44509
diff changeset
307 # However this multiple other caches have the same vulnerability (eg:
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44509
diff changeset
308 # brancmap cache).
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44509
diff changeset
309 self.tip_node = None
44506
283fd803afa5 nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44480
diff changeset
310 # the size (in bytes) of the persisted data to encode the nodemap valid
283fd803afa5 nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44480
diff changeset
311 # for `tip_rev`.
283fd803afa5 nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44480
diff changeset
312 # - data file shorter than this are corrupted,
283fd803afa5 nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44480
diff changeset
313 # - any extra data should be ignored.
44370
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
314 self.data_length = None
44506
283fd803afa5 nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44480
diff changeset
315 # the amount (in bytes) of "dead" data, still in the data file but no
283fd803afa5 nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44480
diff changeset
316 # longer used for the nodemap.
44370
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
317 self.data_unused = 0
44365
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
318
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
319 def copy(self):
44369
e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44367
diff changeset
320 new = NodeMapDocket(uid=self.uid)
e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44367
diff changeset
321 new.tip_rev = self.tip_rev
44525
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44509
diff changeset
322 new.tip_node = self.tip_node
44370
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
323 new.data_length = self.data_length
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
324 new.data_unused = self.data_unused
44369
e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44367
diff changeset
325 return new
44365
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
326
44371
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
327 def __cmp__(self, other):
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
328 if self.uid < other.uid:
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
329 return -1
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
330 if self.uid > other.uid:
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
331 return 1
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
332 elif self.data_length < other.data_length:
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
333 return -1
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
334 elif self.data_length > other.data_length:
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
335 return 1
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
336 return 0
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
337
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
338 def __eq__(self, other):
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
339 return self.uid == other.uid and self.data_length == other.data_length
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
340
44365
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
341 def serialize(self):
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
342 """return serialized bytes for a docket using the passed uid"""
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
343 data = []
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
344 data.append(S_VERSION.pack(ONDISK_VERSION))
44370
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
345 headers = (
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
346 len(self.uid),
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
347 self.tip_rev,
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
348 self.data_length,
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
349 self.data_unused,
44525
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44509
diff changeset
350 len(self.tip_node),
44370
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
351 )
44369
e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44367
diff changeset
352 data.append(S_HEADER.pack(*headers))
44365
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
353 data.append(self.uid)
44525
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44509
diff changeset
354 data.append(self.tip_node)
44365
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
355 return b''.join(data)
44354
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
356
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
357
44365
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
358 def _rawdata_filepath(revlog, docket):
44354
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
359 """The (vfs relative) nodemap's rawdata file for a given uid"""
44526
64e2f603de9d nodemap: make sure hooks have access to an up-to-date version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44525
diff changeset
360 if revlog.nodemap_file.endswith(b'.n.a'):
64e2f603de9d nodemap: make sure hooks have access to an up-to-date version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44525
diff changeset
361 prefix = revlog.nodemap_file[:-4]
64e2f603de9d nodemap: make sure hooks have access to an up-to-date version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44525
diff changeset
362 else:
64e2f603de9d nodemap: make sure hooks have access to an up-to-date version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44525
diff changeset
363 prefix = revlog.nodemap_file[:-2]
44365
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
364 return b"%s-%s.nd" % (prefix, docket.uid)
44351
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44350
diff changeset
365
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44350
diff changeset
366
44365
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
367 def _other_rawdata_filepath(revlog, docket):
44355
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44354
diff changeset
368 prefix = revlog.nodemap_file[:-2]
44480
6aee0647e026 nodemap: fix missing r-prefix on regular expression
Augie Fackler <augie@google.com>
parents: 44465
diff changeset
369 pattern = re.compile(br"(^|/)%s-[0-9a-f]+\.nd$" % prefix)
44365
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
370 new_file_path = _rawdata_filepath(revlog, docket)
44355
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44354
diff changeset
371 new_file_name = revlog.opener.basename(new_file_path)
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44354
diff changeset
372 dirpath = revlog.opener.dirname(new_file_path)
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44354
diff changeset
373 others = []
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44354
diff changeset
374 for f in revlog.opener.listdir(dirpath):
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44354
diff changeset
375 if pattern.match(f) and f != new_file_name:
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44354
diff changeset
376 others.append(f)
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44354
diff changeset
377 return others
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44354
diff changeset
378
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44354
diff changeset
379
44350
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
380 ### Nodemap Trie
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
381 #
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
382 # This is a simple reference implementation to compute and persist a nodemap
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
383 # trie. This reference implementation is write only. The python version of this
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
384 # is not expected to be actually used, since it wont provide performance
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
385 # improvement over existing non-persistent C implementation.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
386 #
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
387 # The nodemap is persisted as Trie using 4bits-address/16-entries block. each
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
388 # revision can be adressed using its node shortest prefix.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
389 #
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
390 # The trie is stored as a sequence of block. Each block contains 16 entries
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
391 # (signed 64bit integer, big endian). Each entry can be one of the following:
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
392 #
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
393 # * value >= 0 -> index of sub-block
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
394 # * value == -1 -> no value
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
395 # * value < -1 -> a revision value: rev = -(value+10)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
396 #
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
397 # The implementation focus on simplicity, not on performance. A Rust
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
398 # implementation should provide a efficient version of the same binary
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
399 # persistence. This reference python implementation is never meant to be
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
400 # extensively use in production.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
401
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
402
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
403 def persistent_data(index):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
404 """return the persistent binary form for a nodemap for a given index
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
405 """
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
406 trie = _build_trie(index)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
407 return _persist_trie(trie)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
408
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
409
44367
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
410 def update_persistent_data(index, root, max_idx, last_rev):
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
411 """return the incremental update for persistent nodemap from a given index
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
412 """
44370
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
413 changed_block, trie = _update_trie(index, root, last_rev)
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
414 return (
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
415 changed_block * S_BLOCK.size,
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
416 _persist_trie(trie, existing_idx=max_idx),
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
417 )
44367
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
418
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
419
44350
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
420 S_BLOCK = struct.Struct(">" + ("l" * 16))
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
421
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
422 NO_ENTRY = -1
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
423 # rev 0 need to be -2 because 0 is used by block, -1 is a special value.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
424 REV_OFFSET = 2
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
425
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
426
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
427 def _transform_rev(rev):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
428 """Return the number used to represent the rev in the tree.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
429
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
430 (or retrieve a rev number from such representation)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
431
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
432 Note that this is an involution, a function equal to its inverse (i.e.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
433 which gives the identity when applied to itself).
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
434 """
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
435 return -(rev + REV_OFFSET)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
436
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
437
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
438 def _to_int(hex_digit):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
439 """turn an hexadecimal digit into a proper integer"""
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
440 return int(hex_digit, 16)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
441
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
442
44358
7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44357
diff changeset
443 class Block(dict):
7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44357
diff changeset
444 """represent a block of the Trie
7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44357
diff changeset
445
7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44357
diff changeset
446 contains up to 16 entry indexed from 0 to 15"""
7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44357
diff changeset
447
44364
f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44362
diff changeset
448 def __init__(self):
f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44362
diff changeset
449 super(Block, self).__init__()
f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44362
diff changeset
450 # If this block exist on disk, here is its ID
f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44362
diff changeset
451 self.ondisk_id = None
f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44362
diff changeset
452
44359
55b12f2593c1 nodemap: move the iteratio inside the Block object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44358
diff changeset
453 def __iter__(self):
55b12f2593c1 nodemap: move the iteratio inside the Block object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44358
diff changeset
454 return iter(self.get(i) for i in range(16))
55b12f2593c1 nodemap: move the iteratio inside the Block object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44358
diff changeset
455
44358
7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44357
diff changeset
456
44350
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
457 def _build_trie(index):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
458 """build a nodemap trie
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
459
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
460 The nodemap stores revision number for each unique prefix.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
461
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
462 Each block is a dictionary with keys in `[0, 15]`. Values are either
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
463 another block or a revision number.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
464 """
44358
7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44357
diff changeset
465 root = Block()
44350
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
466 for rev in range(len(index)):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
467 hex = nodemod.hex(index[rev][7])
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
468 _insert_into_block(index, 0, root, rev, hex)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
469 return root
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
470
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
471
44367
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
472 def _update_trie(index, root, last_rev):
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
473 """consume"""
44370
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
474 changed = 0
44367
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
475 for rev in range(last_rev + 1, len(index)):
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
476 hex = nodemod.hex(index[rev][7])
44370
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
477 changed += _insert_into_block(index, 0, root, rev, hex)
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
478 return changed, root
44367
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
479
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
480
44350
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
481 def _insert_into_block(index, level, block, current_rev, current_hex):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
482 """insert a new revision in a block
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
483
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
484 index: the index we are adding revision for
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
485 level: the depth of the current block in the trie
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
486 block: the block currently being considered
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
487 current_rev: the revision number we are adding
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
488 current_hex: the hexadecimal representation of the of that revision
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
489 """
44370
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
490 changed = 1
44367
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
491 if block.ondisk_id is not None:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
492 block.ondisk_id = None
44350
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
493 hex_digit = _to_int(current_hex[level : level + 1])
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
494 entry = block.get(hex_digit)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
495 if entry is None:
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
496 # no entry, simply store the revision number
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
497 block[hex_digit] = current_rev
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
498 elif isinstance(entry, dict):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
499 # need to recurse to an underlying block
44370
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
500 changed += _insert_into_block(
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
501 index, level + 1, entry, current_rev, current_hex
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
502 )
44350
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
503 else:
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
504 # collision with a previously unique prefix, inserting new
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
505 # vertices to fit both entry.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
506 other_hex = nodemod.hex(index[entry][7])
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
507 other_rev = entry
44358
7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44357
diff changeset
508 new = Block()
44350
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
509 block[hex_digit] = new
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
510 _insert_into_block(index, level + 1, new, other_rev, other_hex)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
511 _insert_into_block(index, level + 1, new, current_rev, current_hex)
44370
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
512 return changed
44350
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
513
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
514
44367
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
515 def _persist_trie(root, existing_idx=None):
44350
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
516 """turn a nodemap trie into persistent binary data
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
517
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
518 See `_build_trie` for nodemap trie structure"""
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
519 block_map = {}
44367
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
520 if existing_idx is not None:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
521 base_idx = existing_idx + 1
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
522 else:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
523 base_idx = 0
44350
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
524 chunks = []
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
525 for tn in _walk_trie(root):
44367
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
526 if tn.ondisk_id is not None:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
527 block_map[id(tn)] = tn.ondisk_id
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
528 else:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
529 block_map[id(tn)] = len(chunks) + base_idx
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
530 chunks.append(_persist_block(tn, block_map))
44350
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
531 return b''.join(chunks)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
532
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
533
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
534 def _walk_trie(block):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
535 """yield all the block in a trie
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
536
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
537 Children blocks are always yield before their parent block.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
538 """
44696
ed475420e0af nodemap: rename a variable to prevent shawoding "_"
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44640
diff changeset
539 for (__, item) in sorted(block.items()):
44350
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
540 if isinstance(item, dict):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
541 for sub_block in _walk_trie(item):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
542 yield sub_block
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
543 yield block
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
544
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
545
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
546 def _persist_block(block_node, block_map):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
547 """produce persistent binary data for a single block
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
548
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
549 Children block are assumed to be already persisted and present in
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
550 block_map.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
551 """
44359
55b12f2593c1 nodemap: move the iteratio inside the Block object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44358
diff changeset
552 data = tuple(_to_value(v, block_map) for v in block_node)
44350
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
553 return S_BLOCK.pack(*data)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
554
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
555
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
556 def _to_value(item, block_map):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
557 """persist any value as an integer"""
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
558 if item is None:
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
559 return NO_ENTRY
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
560 elif isinstance(item, dict):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
561 return block_map[id(item)]
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
562 else:
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
563 return _transform_rev(item)
44360
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44359
diff changeset
564
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44359
diff changeset
565
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44359
diff changeset
566 def parse_data(data):
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44359
diff changeset
567 """parse parse nodemap data into a nodemap Trie"""
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44359
diff changeset
568 if (len(data) % S_BLOCK.size) != 0:
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44359
diff changeset
569 msg = "nodemap data size is not a multiple of block size (%d): %d"
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44359
diff changeset
570 raise error.Abort(msg % (S_BLOCK.size, len(data)))
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44359
diff changeset
571 if not data:
44367
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
572 return Block(), None
44360
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44359
diff changeset
573 block_map = {}
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44359
diff changeset
574 new_blocks = []
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44359
diff changeset
575 for i in range(0, len(data), S_BLOCK.size):
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44359
diff changeset
576 block = Block()
44364
f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44362
diff changeset
577 block.ondisk_id = len(block_map)
f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44362
diff changeset
578 block_map[block.ondisk_id] = block
44360
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44359
diff changeset
579 block_data = data[i : i + S_BLOCK.size]
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44359
diff changeset
580 values = S_BLOCK.unpack(block_data)
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44359
diff changeset
581 new_blocks.append((block, values))
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44359
diff changeset
582 for b, values in new_blocks:
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44359
diff changeset
583 for idx, v in enumerate(values):
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44359
diff changeset
584 if v == NO_ENTRY:
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44359
diff changeset
585 continue
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44359
diff changeset
586 elif v >= 0:
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44359
diff changeset
587 b[idx] = block_map[v]
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44359
diff changeset
588 else:
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44359
diff changeset
589 b[idx] = _transform_rev(v)
44367
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
590 return block, i // S_BLOCK.size
44361
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
591
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
592
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
593 # debug utility
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
594
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
595
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
596 def check_data(ui, index, data):
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
597 """verify that the provided nodemap data are valid for the given idex"""
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
598 ret = 0
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
599 ui.status((b"revision in index: %d\n") % len(index))
44367
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
600 root, __ = parse_data(data)
44361
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
601 all_revs = set(_all_revisions(root))
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
602 ui.status((b"revision in nodemap: %d\n") % len(all_revs))
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
603 for r in range(len(index)):
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
604 if r not in all_revs:
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
605 msg = b" revision missing from nodemap: %d\n" % r
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
606 ui.write_err(msg)
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
607 ret = 1
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
608 else:
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
609 all_revs.remove(r)
44362
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44361
diff changeset
610 nm_rev = _find_node(root, nodemod.hex(index[r][7]))
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44361
diff changeset
611 if nm_rev is None:
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44361
diff changeset
612 msg = b" revision node does not match any entries: %d\n" % r
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44361
diff changeset
613 ui.write_err(msg)
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44361
diff changeset
614 ret = 1
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44361
diff changeset
615 elif nm_rev != r:
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44361
diff changeset
616 msg = (
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44361
diff changeset
617 b" revision node does not match the expected revision: "
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44361
diff changeset
618 b"%d != %d\n" % (r, nm_rev)
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44361
diff changeset
619 )
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44361
diff changeset
620 ui.write_err(msg)
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44361
diff changeset
621 ret = 1
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44361
diff changeset
622
44361
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
623 if all_revs:
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
624 for r in sorted(all_revs):
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
625 msg = b" extra revision in nodemap: %d\n" % r
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
626 ui.write_err(msg)
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
627 ret = 1
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
628 return ret
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
629
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
630
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
631 def _all_revisions(root):
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
632 """return all revisions stored in a Trie"""
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
633 for block in _walk_trie(root):
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
634 for v in block:
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
635 if v is None or isinstance(v, Block):
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
636 continue
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
637 yield v
44362
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44361
diff changeset
638
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44361
diff changeset
639
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44361
diff changeset
640 def _find_node(block, node):
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44361
diff changeset
641 """find the revision associated with a given node"""
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44361
diff changeset
642 entry = block.get(_to_int(node[0:1]))
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44361
diff changeset
643 if isinstance(entry, dict):
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44361
diff changeset
644 return _find_node(entry, node[1:])
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44361
diff changeset
645 return entry