Mercurial > public > mercurial-scm > hg-stable

annotate mercurial/revlogutils/nodemap.py @ 44740:b4537125eb3c stable

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
nodemap: skip persistent nodemap warming for revlog not using it Before this patch, the usual checking (especially, inline-ess) were not performed when warming the cache through `hg debugupdatecache`. This is now fixed. Differential Revision: https://phab.mercurial-scm.org/D8408
author Pierre-Yves David <pierre-yves.david@octobus.net>
date Mon, 13 Apr 2020 18:04:55 +0200
parents 0792ad55e533
children 6493f0a567c2
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
44035
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
1 # nodemap.py - nodemap related code and utilities
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
2 #
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
3 # Copyright 2019 Pierre-Yves David <pierre-yves.david@octobus.net>
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
4 # Copyright 2019 George Racinet <georges.racinet@octobus.net>
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
5 #
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
6 # This software may be used and distributed according to the terms of the
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
7 # GNU General Public License version 2 or any later version.
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
8
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
9 from __future__ import absolute_import
44350
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
10
44397
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
11 import errno
44354
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
12 import os
44355
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44354
diff changeset
13 import re
44350
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
14 import struct
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
15
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
16 from .. import (
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
17 error,
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
18 node as nodemod,
44357
7f4f7ef3133e nodemap: add a optional `nodemap_add_full` method on indexes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44355
diff changeset
19 util,
44350
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
20 )
44035
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
21
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
22
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
23 class NodeMap(dict):
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
24 def __missing__(self, x):
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
25 raise error.RevlogError(b'unknown node: %s' % x)
44350
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
26
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
27
44352
6c07480d6659 nodemap: add a function to read the data from disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44351
diff changeset
28 def persisted_data(revlog):
6c07480d6659 nodemap: add a function to read the data from disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44351
diff changeset
29 """read the nodemap for a revlog from disk"""
6c07480d6659 nodemap: add a function to read the data from disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44351
diff changeset
30 if revlog.nodemap_file is None:
6c07480d6659 nodemap: add a function to read the data from disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44351
diff changeset
31 return None
44354
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
32 pdata = revlog.opener.tryread(revlog.nodemap_file)
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
33 if not pdata:
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
34 return None
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
35 offset = 0
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
36 (version,) = S_VERSION.unpack(pdata[offset : offset + S_VERSION.size])
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
37 if version != ONDISK_VERSION:
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
38 return None
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
39 offset += S_VERSION.size
44369
e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44367
diff changeset
40 headers = S_HEADER.unpack(pdata[offset : offset + S_HEADER.size])
44525
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44509
diff changeset
41 uid_size, tip_rev, data_length, data_unused, tip_node_size = headers
44354
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
42 offset += S_HEADER.size
44365
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
43 docket = NodeMapDocket(pdata[offset : offset + uid_size])
44525
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44509
diff changeset
44 offset += uid_size
44369
e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44367
diff changeset
45 docket.tip_rev = tip_rev
44525
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44509
diff changeset
46 docket.tip_node = pdata[offset : offset + tip_node_size]
44370
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
47 docket.data_length = data_length
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
48 docket.data_unused = data_unused
44354
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
49
44365
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
50 filename = _rawdata_filepath(revlog, docket)
44697
0792ad55e533 nodemap: access the mmap opener option using bytes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44696
diff changeset
51 use_mmap = revlog.opener.options.get(b"exp-persistent-nodemap.mmap")
44397
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
52 try:
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
53 with revlog.opener(filename) as fd:
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
54 if use_mmap:
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
55 data = util.buffer(util.mmapread(fd, data_length))
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
56 else:
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
57 data = fd.read(data_length)
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
58 except OSError as e:
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
59 if e.errno != errno.ENOENT:
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
60 raise
44373
c7eebdb15139 nodemap: never read more than the expected data amount
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44372
diff changeset
61 if len(data) < data_length:
c7eebdb15139 nodemap: never read more than the expected data amount
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44372
diff changeset
62 return None
c7eebdb15139 nodemap: never read more than the expected data amount
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44372
diff changeset
63 return docket, data
44352
6c07480d6659 nodemap: add a function to read the data from disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44351
diff changeset
64
6c07480d6659 nodemap: add a function to read the data from disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44351
diff changeset
65
44351
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44350
diff changeset
66 def setup_persistent_nodemap(tr, revlog):
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44350
diff changeset
67 """Install whatever is needed transaction side to persist a nodemap on disk
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44350
diff changeset
68
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44350
diff changeset
69 (only actually persist the nodemap if this is relevant for this revlog)
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44350
diff changeset
70 """
44353
daad3aace942 nodemap: only use persistent nodemap for non-inlined revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44352
diff changeset
71 if revlog._inline:
daad3aace942 nodemap: only use persistent nodemap for non-inlined revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44352
diff changeset
72 return # inlined revlog are too small for this to be relevant
44351
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44350
diff changeset
73 if revlog.nodemap_file is None:
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44350
diff changeset
74 return # we do not use persistent_nodemap on this revlog
44527
448d700e0d27 nodemap: make sure the nodemap docket is updated after the changelog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44526
diff changeset
75
448d700e0d27 nodemap: make sure the nodemap docket is updated after the changelog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44526
diff changeset
76 # we need to happen after the changelog finalization, in that use "cl-"
448d700e0d27 nodemap: make sure the nodemap docket is updated after the changelog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44526
diff changeset
77 callback_id = b"nm-revlog-persistent-nodemap-%s" % revlog.nodemap_file
44351
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44350
diff changeset
78 if tr.hasfinalize(callback_id):
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44350
diff changeset
79 return # no need to register again
44526
64e2f603de9d nodemap: make sure hooks have access to an up-to-date version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44525
diff changeset
80 tr.addpending(
64e2f603de9d nodemap: make sure hooks have access to an up-to-date version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44525
diff changeset
81 callback_id, lambda tr: _persist_nodemap(tr, revlog, pending=True)
64e2f603de9d nodemap: make sure hooks have access to an up-to-date version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44525
diff changeset
82 )
44509
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44506
diff changeset
83 tr.addfinalize(callback_id, lambda tr: _persist_nodemap(tr, revlog))
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44506
diff changeset
84
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44506
diff changeset
85
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44506
diff changeset
86 class _NoTransaction(object):
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44506
diff changeset
87 """transaction like object to update the nodemap outside a transaction
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44506
diff changeset
88 """
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44506
diff changeset
89
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44506
diff changeset
90 def __init__(self):
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44506
diff changeset
91 self._postclose = {}
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44506
diff changeset
92
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44506
diff changeset
93 def addpostclose(self, callback_id, callback_func):
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44506
diff changeset
94 self._postclose[callback_id] = callback_func
44351
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44350
diff changeset
95
44638
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44527
diff changeset
96 def registertmp(self, *args, **kwargs):
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44527
diff changeset
97 pass
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44527
diff changeset
98
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44527
diff changeset
99 def addbackup(self, *args, **kwargs):
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44527
diff changeset
100 pass
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44527
diff changeset
101
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44527
diff changeset
102 def add(self, *args, **kwargs):
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44527
diff changeset
103 pass
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44527
diff changeset
104
44639
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44638
diff changeset
105 def addabort(self, *args, **kwargs):
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44638
diff changeset
106 pass
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44638
diff changeset
107
44351
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44350
diff changeset
108
44465
336ec75ed1ac nodemap: warm the persistent nodemap on disk with debugupdatecache
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44397
diff changeset
109 def update_persistent_nodemap(revlog):
336ec75ed1ac nodemap: warm the persistent nodemap on disk with debugupdatecache
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44397
diff changeset
110 """update the persistent nodemap right now
336ec75ed1ac nodemap: warm the persistent nodemap on disk with debugupdatecache
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44397
diff changeset
111
336ec75ed1ac nodemap: warm the persistent nodemap on disk with debugupdatecache
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44397
diff changeset
112 To be used for updating the nodemap on disk outside of a normal transaction
336ec75ed1ac nodemap: warm the persistent nodemap on disk with debugupdatecache
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44397
diff changeset
113 setup (eg, `debugupdatecache`).
336ec75ed1ac nodemap: warm the persistent nodemap on disk with debugupdatecache
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44397
diff changeset
114 """
44740
b4537125eb3c nodemap: skip persistent nodemap warming for revlog not using it
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44697
diff changeset
115 if revlog._inline:
b4537125eb3c nodemap: skip persistent nodemap warming for revlog not using it
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44697
diff changeset
116 return # inlined revlog are too small for this to be relevant
b4537125eb3c nodemap: skip persistent nodemap warming for revlog not using it
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44697
diff changeset
117 if revlog.nodemap_file is None:
b4537125eb3c nodemap: skip persistent nodemap warming for revlog not using it
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44697
diff changeset
118 return # we do not use persistent_nodemap on this revlog
b4537125eb3c nodemap: skip persistent nodemap warming for revlog not using it
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44697
diff changeset
119
44509
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44506
diff changeset
120 notr = _NoTransaction()
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44506
diff changeset
121 _persist_nodemap(notr, revlog)
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44506
diff changeset
122 for k in sorted(notr._postclose):
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44506
diff changeset
123 notr._postclose[k](None)
44465
336ec75ed1ac nodemap: warm the persistent nodemap on disk with debugupdatecache
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44397
diff changeset
124
336ec75ed1ac nodemap: warm the persistent nodemap on disk with debugupdatecache
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44397
diff changeset
125
44526
64e2f603de9d nodemap: make sure hooks have access to an up-to-date version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44525
diff changeset
126 def _persist_nodemap(tr, revlog, pending=False):
44351
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44350
diff changeset
127 """Write nodemap data on disk for a given revlog
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44350
diff changeset
128 """
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44350
diff changeset
129 if getattr(revlog, 'filteredrevs', ()):
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44350
diff changeset
130 raise error.ProgrammingError(
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44350
diff changeset
131 "cannot persist nodemap of a filtered changelog"
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44350
diff changeset
132 )
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44350
diff changeset
133 if revlog.nodemap_file is None:
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44350
diff changeset
134 msg = "calling persist nodemap on a revlog without the feature enableb"
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44350
diff changeset
135 raise error.ProgrammingError(msg)
44367
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
136
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
137 can_incremental = util.safehasattr(revlog.index, "nodemap_data_incremental")
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
138 ondisk_docket = revlog._nodemap_docket
44397
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
139 feed_data = util.safehasattr(revlog.index, "update_nodemap_data")
44697
0792ad55e533 nodemap: access the mmap opener option using bytes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44696
diff changeset
140 use_mmap = revlog.opener.options.get(b"exp-persistent-nodemap.mmap")
44367
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
141
44371
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
142 data = None
44367
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
143 # first attemp an incremental update of the data
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
144 if can_incremental and ondisk_docket is not None:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
145 target_docket = revlog._nodemap_docket.copy()
44371
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
146 (
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
147 src_docket,
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
148 data_changed_count,
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
149 data,
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
150 ) = revlog.index.nodemap_data_incremental()
44640
c70bcaf7927b nodemap: automatically "vacuum" the persistent nodemap when too sparse
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44639
diff changeset
151 new_length = target_docket.data_length + len(data)
c70bcaf7927b nodemap: automatically "vacuum" the persistent nodemap when too sparse
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44639
diff changeset
152 new_unused = target_docket.data_unused + data_changed_count
44371
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
153 if src_docket != target_docket:
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
154 data = None
44640
c70bcaf7927b nodemap: automatically "vacuum" the persistent nodemap when too sparse
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44639
diff changeset
155 elif new_length <= (new_unused * 10): # under 10% of unused data
c70bcaf7927b nodemap: automatically "vacuum" the persistent nodemap when too sparse
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44639
diff changeset
156 data = None
44371
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
157 else:
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
158 datafile = _rawdata_filepath(revlog, target_docket)
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
159 # EXP-TODO: if this is a cache, this should use a cache vfs, not a
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
160 # store vfs
44638
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44527
diff changeset
161 tr.add(datafile, target_docket.data_length)
44372
2ea6a67ff502 nodemap: write new data from the expected current data length
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44371
diff changeset
162 with revlog.opener(datafile, b'r+') as fd:
2ea6a67ff502 nodemap: write new data from the expected current data length
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44371
diff changeset
163 fd.seek(target_docket.data_length)
44371
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
164 fd.write(data)
44397
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
165 if feed_data:
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
166 if use_mmap:
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
167 fd.seek(0)
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
168 new_data = fd.read(new_length)
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
169 else:
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
170 fd.flush()
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
171 new_data = util.buffer(util.mmapread(fd, new_length))
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
172 target_docket.data_length = new_length
44640
c70bcaf7927b nodemap: automatically "vacuum" the persistent nodemap when too sparse
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44639
diff changeset
173 target_docket.data_unused = new_unused
44371
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
174
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
175 if data is None:
44367
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
176 # otherwise fallback to a full new export
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
177 target_docket = NodeMapDocket()
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
178 datafile = _rawdata_filepath(revlog, target_docket)
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
179 if util.safehasattr(revlog.index, "nodemap_data_all"):
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
180 data = revlog.index.nodemap_data_all()
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
181 else:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
182 data = persistent_data(revlog.index)
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
183 # EXP-TODO: if this is a cache, this should use a cache vfs, not a
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
184 # store vfs
44639
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44638
diff changeset
185
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44638
diff changeset
186 tryunlink = revlog.opener.tryunlink
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44638
diff changeset
187
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44638
diff changeset
188 def abortck(tr):
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44638
diff changeset
189 tryunlink(datafile)
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44638
diff changeset
190
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44638
diff changeset
191 callback_id = b"delete-%s" % datafile
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44638
diff changeset
192
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44638
diff changeset
193 # some flavor of the transaction abort does not cleanup new file, it
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44638
diff changeset
194 # simply empty them.
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44638
diff changeset
195 tr.addabort(callback_id, abortck)
44397
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
196 with revlog.opener(datafile, b'w+') as fd:
44367
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
197 fd.write(data)
44397
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
198 if feed_data:
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
199 if use_mmap:
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
200 new_data = data
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
201 else:
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
202 fd.flush()
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
203 new_data = util.buffer(util.mmapread(fd, len(data)))
44370
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
204 target_docket.data_length = len(data)
44369
e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44367
diff changeset
205 target_docket.tip_rev = revlog.tiprev()
44525
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44509
diff changeset
206 target_docket.tip_node = revlog.node(target_docket.tip_rev)
44367
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
207 # EXP-TODO: if this is a cache, this should use a cache vfs, not a
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
208 # store vfs
44526
64e2f603de9d nodemap: make sure hooks have access to an up-to-date version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44525
diff changeset
209 file_path = revlog.nodemap_file
64e2f603de9d nodemap: make sure hooks have access to an up-to-date version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44525
diff changeset
210 if pending:
64e2f603de9d nodemap: make sure hooks have access to an up-to-date version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44525
diff changeset
211 file_path += b'.a'
44638
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44527
diff changeset
212 tr.registertmp(file_path)
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44527
diff changeset
213 else:
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44527
diff changeset
214 tr.addbackup(file_path)
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44527
diff changeset
215
44526
64e2f603de9d nodemap: make sure hooks have access to an up-to-date version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44525
diff changeset
216 with revlog.opener(file_path, b'w', atomictemp=True) as fp:
44367
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
217 fp.write(target_docket.serialize())
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
218 revlog._nodemap_docket = target_docket
44397
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44374
diff changeset
219 if feed_data:
44374
6ecc34b31137 nodemap: update the index with the newly written data (when appropriate)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44373
diff changeset
220 revlog.index.update_nodemap_data(target_docket, new_data)
6ecc34b31137 nodemap: update the index with the newly written data (when appropriate)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44373
diff changeset
221
44367
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
222 # search for old index file in all cases, some older process might have
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
223 # left one behind.
44365
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
224 olds = _other_rawdata_filepath(revlog, target_docket)
44355
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44354
diff changeset
225 if olds:
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44354
diff changeset
226 realvfs = getattr(revlog, '_realopener', revlog.opener)
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44354
diff changeset
227
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44354
diff changeset
228 def cleanup(tr):
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44354
diff changeset
229 for oldfile in olds:
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44354
diff changeset
230 realvfs.tryunlink(oldfile)
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44354
diff changeset
231
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44354
diff changeset
232 callback_id = b"revlog-cleanup-nodemap-%s" % revlog.nodemap_file
44509
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44506
diff changeset
233 tr.addpostclose(callback_id, cleanup)
44354
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
234
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
235
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
236 ### Nodemap docket file
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
237 #
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
238 # The nodemap data are stored on disk using 2 files:
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
239 #
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
240 # * a raw data files containing a persistent nodemap
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
241 # (see `Nodemap Trie` section)
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
242 #
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
243 # * a small "docket" file containing medatadata
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
244 #
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
245 # While the nodemap data can be multiple tens of megabytes, the "docket" is
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
246 # small, it is easy to update it automatically or to duplicated its content
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
247 # during a transaction.
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
248 #
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
249 # Multiple raw data can exist at the same time (The currently valid one and a
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
250 # new one beind used by an in progress transaction). To accomodate this, the
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
251 # filename hosting the raw data has a variable parts. The exact filename is
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
252 # specified inside the "docket" file.
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
253 #
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
254 # The docket file contains information to find, qualify and validate the raw
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
255 # data. Its content is currently very light, but it will expand as the on disk
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
256 # nodemap gains the necessary features to be used in production.
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
257
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
258 # version 0 is experimental, no BC garantee, do no use outside of tests.
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
259 ONDISK_VERSION = 0
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
260 S_VERSION = struct.Struct(">B")
44525
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44509
diff changeset
261 S_HEADER = struct.Struct(">BQQQQ")
44354
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
262
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
263 ID_SIZE = 8
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
264
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
265
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
266 def _make_uid():
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
267 """return a new unique identifier.
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
268
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
269 The identifier is random and composed of ascii characters."""
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
270 return nodemod.hex(os.urandom(ID_SIZE))
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
271
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
272
44365
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
273 class NodeMapDocket(object):
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
274 """metadata associated with persistent nodemap data
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
275
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
276 The persistent data may come from disk or be on their way to disk.
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
277 """
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
278
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
279 def __init__(self, uid=None):
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
280 if uid is None:
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
281 uid = _make_uid()
44506
283fd803afa5 nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44480
diff changeset
282 # a unique identifier for the data file:
283fd803afa5 nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44480
diff changeset
283 # - When new data are appended, it is preserved.
283fd803afa5 nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44480
diff changeset
284 # - When a new data file is created, a new identifier is generated.
44365
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
285 self.uid = uid
44506
283fd803afa5 nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44480
diff changeset
286 # the tipmost revision stored in the data file. This revision and all
283fd803afa5 nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44480
diff changeset
287 # revision before it are expected to be encoded in the data file.
44369
e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44367
diff changeset
288 self.tip_rev = None
44525
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44509
diff changeset
289 # the node of that tipmost revision, if it mismatch the current index
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44509
diff changeset
290 # data the docket is not valid for the current index and should be
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44509
diff changeset
291 # discarded.
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44509
diff changeset
292 #
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44509
diff changeset
293 # note: this method is not perfect as some destructive operation could
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44509
diff changeset
294 # preserve the same tip_rev + tip_node while altering lower revision.
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44509
diff changeset
295 # However this multiple other caches have the same vulnerability (eg:
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44509
diff changeset
296 # brancmap cache).
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44509
diff changeset
297 self.tip_node = None
44506
283fd803afa5 nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44480
diff changeset
298 # the size (in bytes) of the persisted data to encode the nodemap valid
283fd803afa5 nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44480
diff changeset
299 # for `tip_rev`.
283fd803afa5 nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44480
diff changeset
300 # - data file shorter than this are corrupted,
283fd803afa5 nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44480
diff changeset
301 # - any extra data should be ignored.
44370
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
302 self.data_length = None
44506
283fd803afa5 nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44480
diff changeset
303 # the amount (in bytes) of "dead" data, still in the data file but no
283fd803afa5 nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44480
diff changeset
304 # longer used for the nodemap.
44370
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
305 self.data_unused = 0
44365
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
306
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
307 def copy(self):
44369
e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44367
diff changeset
308 new = NodeMapDocket(uid=self.uid)
e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44367
diff changeset
309 new.tip_rev = self.tip_rev
44525
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44509
diff changeset
310 new.tip_node = self.tip_node
44370
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
311 new.data_length = self.data_length
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
312 new.data_unused = self.data_unused
44369
e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44367
diff changeset
313 return new
44365
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
314
44371
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
315 def __cmp__(self, other):
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
316 if self.uid < other.uid:
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
317 return -1
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
318 if self.uid > other.uid:
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
319 return 1
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
320 elif self.data_length < other.data_length:
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
321 return -1
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
322 elif self.data_length > other.data_length:
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
323 return 1
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
324 return 0
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
325
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
326 def __eq__(self, other):
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
327 return self.uid == other.uid and self.data_length == other.data_length
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44370
diff changeset
328
44365
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
329 def serialize(self):
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
330 """return serialized bytes for a docket using the passed uid"""
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
331 data = []
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
332 data.append(S_VERSION.pack(ONDISK_VERSION))
44370
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
333 headers = (
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
334 len(self.uid),
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
335 self.tip_rev,
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
336 self.data_length,
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
337 self.data_unused,
44525
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44509
diff changeset
338 len(self.tip_node),
44370
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
339 )
44369
e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44367
diff changeset
340 data.append(S_HEADER.pack(*headers))
44365
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
341 data.append(self.uid)
44525
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44509
diff changeset
342 data.append(self.tip_node)
44365
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
343 return b''.join(data)
44354
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
344
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
345
44365
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
346 def _rawdata_filepath(revlog, docket):
44354
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44353
diff changeset
347 """The (vfs relative) nodemap's rawdata file for a given uid"""
44526
64e2f603de9d nodemap: make sure hooks have access to an up-to-date version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44525
diff changeset
348 if revlog.nodemap_file.endswith(b'.n.a'):
64e2f603de9d nodemap: make sure hooks have access to an up-to-date version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44525
diff changeset
349 prefix = revlog.nodemap_file[:-4]
64e2f603de9d nodemap: make sure hooks have access to an up-to-date version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44525
diff changeset
350 else:
64e2f603de9d nodemap: make sure hooks have access to an up-to-date version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44525
diff changeset
351 prefix = revlog.nodemap_file[:-2]
44365
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
352 return b"%s-%s.nd" % (prefix, docket.uid)
44351
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44350
diff changeset
353
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44350
diff changeset
354
44365
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
355 def _other_rawdata_filepath(revlog, docket):
44355
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44354
diff changeset
356 prefix = revlog.nodemap_file[:-2]
44480
6aee0647e026 nodemap: fix missing r-prefix on regular expression
Augie Fackler <augie@google.com>
parents: 44465
diff changeset
357 pattern = re.compile(br"(^|/)%s-[0-9a-f]+\.nd$" % prefix)
44365
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44364
diff changeset
358 new_file_path = _rawdata_filepath(revlog, docket)
44355
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44354
diff changeset
359 new_file_name = revlog.opener.basename(new_file_path)
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44354
diff changeset
360 dirpath = revlog.opener.dirname(new_file_path)
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44354
diff changeset
361 others = []
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44354
diff changeset
362 for f in revlog.opener.listdir(dirpath):
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44354
diff changeset
363 if pattern.match(f) and f != new_file_name:
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44354
diff changeset
364 others.append(f)
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44354
diff changeset
365 return others
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44354
diff changeset
366
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44354
diff changeset
367
44350
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
368 ### Nodemap Trie
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
369 #
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
370 # This is a simple reference implementation to compute and persist a nodemap
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
371 # trie. This reference implementation is write only. The python version of this
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
372 # is not expected to be actually used, since it wont provide performance
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
373 # improvement over existing non-persistent C implementation.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
374 #
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
375 # The nodemap is persisted as Trie using 4bits-address/16-entries block. each
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
376 # revision can be adressed using its node shortest prefix.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
377 #
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
378 # The trie is stored as a sequence of block. Each block contains 16 entries
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
379 # (signed 64bit integer, big endian). Each entry can be one of the following:
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
380 #
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
381 # * value >= 0 -> index of sub-block
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
382 # * value == -1 -> no value
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
383 # * value < -1 -> a revision value: rev = -(value+10)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
384 #
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
385 # The implementation focus on simplicity, not on performance. A Rust
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
386 # implementation should provide a efficient version of the same binary
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
387 # persistence. This reference python implementation is never meant to be
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
388 # extensively use in production.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
389
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
390
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
391 def persistent_data(index):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
392 """return the persistent binary form for a nodemap for a given index
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
393 """
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
394 trie = _build_trie(index)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
395 return _persist_trie(trie)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
396
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
397
44367
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
398 def update_persistent_data(index, root, max_idx, last_rev):
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
399 """return the incremental update for persistent nodemap from a given index
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
400 """
44370
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
401 changed_block, trie = _update_trie(index, root, last_rev)
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
402 return (
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
403 changed_block * S_BLOCK.size,
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
404 _persist_trie(trie, existing_idx=max_idx),
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
405 )
44367
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
406
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
407
44350
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
408 S_BLOCK = struct.Struct(">" + ("l" * 16))
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
409
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
410 NO_ENTRY = -1
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
411 # rev 0 need to be -2 because 0 is used by block, -1 is a special value.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
412 REV_OFFSET = 2
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
413
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
414
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
415 def _transform_rev(rev):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
416 """Return the number used to represent the rev in the tree.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
417
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
418 (or retrieve a rev number from such representation)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
419
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
420 Note that this is an involution, a function equal to its inverse (i.e.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
421 which gives the identity when applied to itself).
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
422 """
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
423 return -(rev + REV_OFFSET)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
424
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
425
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
426 def _to_int(hex_digit):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
427 """turn an hexadecimal digit into a proper integer"""
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
428 return int(hex_digit, 16)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
429
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
430
44358
7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44357
diff changeset
431 class Block(dict):
7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44357
diff changeset
432 """represent a block of the Trie
7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44357
diff changeset
433
7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44357
diff changeset
434 contains up to 16 entry indexed from 0 to 15"""
7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44357
diff changeset
435
44364
f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44362
diff changeset
436 def __init__(self):
f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44362
diff changeset
437 super(Block, self).__init__()
f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44362
diff changeset
438 # If this block exist on disk, here is its ID
f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44362
diff changeset
439 self.ondisk_id = None
f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44362
diff changeset
440
44359
55b12f2593c1 nodemap: move the iteratio inside the Block object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44358
diff changeset
441 def __iter__(self):
55b12f2593c1 nodemap: move the iteratio inside the Block object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44358
diff changeset
442 return iter(self.get(i) for i in range(16))
55b12f2593c1 nodemap: move the iteratio inside the Block object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44358
diff changeset
443
44358
7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44357
diff changeset
444
44350
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
445 def _build_trie(index):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
446 """build a nodemap trie
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
447
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
448 The nodemap stores revision number for each unique prefix.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
449
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
450 Each block is a dictionary with keys in `[0, 15]`. Values are either
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
451 another block or a revision number.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
452 """
44358
7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44357
diff changeset
453 root = Block()
44350
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
454 for rev in range(len(index)):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
455 hex = nodemod.hex(index[rev][7])
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
456 _insert_into_block(index, 0, root, rev, hex)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
457 return root
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
458
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
459
44367
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
460 def _update_trie(index, root, last_rev):
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
461 """consume"""
44370
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
462 changed = 0
44367
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
463 for rev in range(last_rev + 1, len(index)):
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
464 hex = nodemod.hex(index[rev][7])
44370
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
465 changed += _insert_into_block(index, 0, root, rev, hex)
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
466 return changed, root
44367
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
467
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
468
44350
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
469 def _insert_into_block(index, level, block, current_rev, current_hex):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
470 """insert a new revision in a block
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
471
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
472 index: the index we are adding revision for
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
473 level: the depth of the current block in the trie
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
474 block: the block currently being considered
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
475 current_rev: the revision number we are adding
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
476 current_hex: the hexadecimal representation of the of that revision
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
477 """
44370
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
478 changed = 1
44367
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
479 if block.ondisk_id is not None:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
480 block.ondisk_id = None
44350
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
481 hex_digit = _to_int(current_hex[level : level + 1])
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
482 entry = block.get(hex_digit)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
483 if entry is None:
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
484 # no entry, simply store the revision number
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
485 block[hex_digit] = current_rev
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
486 elif isinstance(entry, dict):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
487 # need to recurse to an underlying block
44370
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
488 changed += _insert_into_block(
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
489 index, level + 1, entry, current_rev, current_hex
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
490 )
44350
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
491 else:
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
492 # collision with a previously unique prefix, inserting new
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
493 # vertices to fit both entry.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
494 other_hex = nodemod.hex(index[entry][7])
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
495 other_rev = entry
44358
7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44357
diff changeset
496 new = Block()
44350
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
497 block[hex_digit] = new
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
498 _insert_into_block(index, level + 1, new, other_rev, other_hex)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
499 _insert_into_block(index, level + 1, new, current_rev, current_hex)
44370
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44369
diff changeset
500 return changed
44350
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
501
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
502
44367
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
503 def _persist_trie(root, existing_idx=None):
44350
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
504 """turn a nodemap trie into persistent binary data
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
505
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
506 See `_build_trie` for nodemap trie structure"""
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
507 block_map = {}
44367
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
508 if existing_idx is not None:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
509 base_idx = existing_idx + 1
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
510 else:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
511 base_idx = 0
44350
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
512 chunks = []
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
513 for tn in _walk_trie(root):
44367
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
514 if tn.ondisk_id is not None:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
515 block_map[id(tn)] = tn.ondisk_id
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
516 else:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
517 block_map[id(tn)] = len(chunks) + base_idx
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
518 chunks.append(_persist_block(tn, block_map))
44350
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
519 return b''.join(chunks)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
520
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
521
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
522 def _walk_trie(block):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
523 """yield all the block in a trie
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
524
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
525 Children blocks are always yield before their parent block.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
526 """
44696
ed475420e0af nodemap: rename a variable to prevent shawoding "_"
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44640
diff changeset
527 for (__, item) in sorted(block.items()):
44350
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
528 if isinstance(item, dict):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
529 for sub_block in _walk_trie(item):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
530 yield sub_block
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
531 yield block
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
532
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
533
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
534 def _persist_block(block_node, block_map):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
535 """produce persistent binary data for a single block
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
536
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
537 Children block are assumed to be already persisted and present in
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
538 block_map.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
539 """
44359
55b12f2593c1 nodemap: move the iteratio inside the Block object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44358
diff changeset
540 data = tuple(_to_value(v, block_map) for v in block_node)
44350
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
541 return S_BLOCK.pack(*data)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
542
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
543
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
544 def _to_value(item, block_map):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
545 """persist any value as an integer"""
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
546 if item is None:
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
547 return NO_ENTRY
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
548 elif isinstance(item, dict):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
549 return block_map[id(item)]
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
550 else:
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44035
diff changeset
551 return _transform_rev(item)
44360
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44359
diff changeset
552
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44359
diff changeset
553
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44359
diff changeset
554 def parse_data(data):
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44359
diff changeset
555 """parse parse nodemap data into a nodemap Trie"""
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44359
diff changeset
556 if (len(data) % S_BLOCK.size) != 0:
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44359
diff changeset
557 msg = "nodemap data size is not a multiple of block size (%d): %d"
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44359
diff changeset
558 raise error.Abort(msg % (S_BLOCK.size, len(data)))
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44359
diff changeset
559 if not data:
44367
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
560 return Block(), None
44360
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44359
diff changeset
561 block_map = {}
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44359
diff changeset
562 new_blocks = []
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44359
diff changeset
563 for i in range(0, len(data), S_BLOCK.size):
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44359
diff changeset
564 block = Block()
44364
f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44362
diff changeset
565 block.ondisk_id = len(block_map)
f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44362
diff changeset
566 block_map[block.ondisk_id] = block
44360
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44359
diff changeset
567 block_data = data[i : i + S_BLOCK.size]
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44359
diff changeset
568 values = S_BLOCK.unpack(block_data)
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44359
diff changeset
569 new_blocks.append((block, values))
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44359
diff changeset
570 for b, values in new_blocks:
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44359
diff changeset
571 for idx, v in enumerate(values):
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44359
diff changeset
572 if v == NO_ENTRY:
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44359
diff changeset
573 continue
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44359
diff changeset
574 elif v >= 0:
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44359
diff changeset
575 b[idx] = block_map[v]
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44359
diff changeset
576 else:
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44359
diff changeset
577 b[idx] = _transform_rev(v)
44367
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
578 return block, i // S_BLOCK.size
44361
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
579
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
580
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
581 # debug utility
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
582
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
583
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
584 def check_data(ui, index, data):
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
585 """verify that the provided nodemap data are valid for the given idex"""
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
586 ret = 0
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
587 ui.status((b"revision in index: %d\n") % len(index))
44367
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44366
diff changeset
588 root, __ = parse_data(data)
44361
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
589 all_revs = set(_all_revisions(root))
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
590 ui.status((b"revision in nodemap: %d\n") % len(all_revs))
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
591 for r in range(len(index)):
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
592 if r not in all_revs:
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
593 msg = b" revision missing from nodemap: %d\n" % r
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
594 ui.write_err(msg)
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
595 ret = 1
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
596 else:
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
597 all_revs.remove(r)
44362
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44361
diff changeset
598 nm_rev = _find_node(root, nodemod.hex(index[r][7]))
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44361
diff changeset
599 if nm_rev is None:
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44361
diff changeset
600 msg = b" revision node does not match any entries: %d\n" % r
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44361
diff changeset
601 ui.write_err(msg)
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44361
diff changeset
602 ret = 1
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44361
diff changeset
603 elif nm_rev != r:
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44361
diff changeset
604 msg = (
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44361
diff changeset
605 b" revision node does not match the expected revision: "
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44361
diff changeset
606 b"%d != %d\n" % (r, nm_rev)
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44361
diff changeset
607 )
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44361
diff changeset
608 ui.write_err(msg)
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44361
diff changeset
609 ret = 1
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44361
diff changeset
610
44361
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
611 if all_revs:
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
612 for r in sorted(all_revs):
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
613 msg = b" extra revision in nodemap: %d\n" % r
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
614 ui.write_err(msg)
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
615 ret = 1
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
616 return ret
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
617
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
618
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
619 def _all_revisions(root):
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
620 """return all revisions stored in a Trie"""
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
621 for block in _walk_trie(root):
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
622 for v in block:
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
623 if v is None or isinstance(v, Block):
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
624 continue
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44360
diff changeset
625 yield v
44362
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44361
diff changeset
626
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44361
diff changeset
627
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44361
diff changeset
628 def _find_node(block, node):
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44361
diff changeset
629 """find the revision associated with a given node"""
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44361
diff changeset
630 entry = block.get(_to_int(node[0:1]))
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44361
diff changeset
631 if isinstance(entry, dict):
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44361
diff changeset
632 return _find_node(entry, node[1:])
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44361
diff changeset
633 return entry