mercurial-scm/hg-stable: mercurial/revlogutils/nodemap.py annotate

annotate mercurial/revlogutils/nodemap.py @ 49037:642e31cb55f0

py3: use class X: instead of class X(object): The inheritance from object is implied in Python 3. So this should be equivalent. This change was generated via an automated search and replace. So there may have been some accidental changes. Differential Revision: https://phab.mercurial-scm.org/D12352

author	Gregory Szorc <gregory.szorc@gmail.com>
date	Mon, 21 Feb 2022 13:08:28 -0700
parents	6000f5b25c9b
children	2e726c934fcd

rev	line source
44035 ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	1 # nodemap.py - nodemap related code and utilities
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	2 #
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	3 # Copyright 2019 Pierre-Yves David <pierre-yves.david@octobus.net>
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	4 # Copyright 2019 George Racinet <georges.racinet@octobus.net>
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	5 #
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	6 # This software may be used and distributed according to the terms of the
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	7 # GNU General Public License version 2 or any later version.
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	8
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	9
44397 f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44374 diff changeset	10 import errno
44355 563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	11 import re
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	12 import struct
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	13
46114 59fa3890d40a node: import symbols explicitly Joerg Sonnenberger <joerg@bec.de> parents: 46090 diff changeset	14 from ..node import hex
44865 6493f0a567c2 nodemap: add a new mode option, with an optional "warn" value Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44740 diff changeset	15
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	16 from .. import (
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	17 error,
48707 de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48704 diff changeset	18 requirements,
44357 7f4f7ef3133e nodemap: add a optional `nodemap_add_full` method on indexes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44355 diff changeset	19 util,
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	20 )
47323 7ea39d633cf3 docket: move the uid logic in the `revlogutils.docket` module Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47322 diff changeset	21 from . import docket as docket_mod
44035 ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	22
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	23
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	24 class NodeMap(dict):
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	25 def __missing__(self, x):
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	26 raise error.RevlogError(b'unknown node: %s' % x)
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	27
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	28
47887 52018f8ef020 persistent-nodemap: introduce a test to highlight possible race Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47457 diff changeset	29 def test_race_hook_1():
52018f8ef020 persistent-nodemap: introduce a test to highlight possible race Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47457 diff changeset	30 """hook point for test
52018f8ef020 persistent-nodemap: introduce a test to highlight possible race Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47457 diff changeset	31
52018f8ef020 persistent-nodemap: introduce a test to highlight possible race Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47457 diff changeset	32 This let tests to have things happens between the docket reading and the
52018f8ef020 persistent-nodemap: introduce a test to highlight possible race Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47457 diff changeset	33 data reading"""
52018f8ef020 persistent-nodemap: introduce a test to highlight possible race Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47457 diff changeset	34 pass
52018f8ef020 persistent-nodemap: introduce a test to highlight possible race Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47457 diff changeset	35
52018f8ef020 persistent-nodemap: introduce a test to highlight possible race Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47457 diff changeset	36
48707 de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48704 diff changeset	37 def post_stream_cleanup(repo):
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48704 diff changeset	38 """The stream clone might needs to remove some file if persisten nodemap
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48704 diff changeset	39 was dropped while stream cloning
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48704 diff changeset	40 """
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48704 diff changeset	41 if requirements.REVLOGV1_REQUIREMENT not in repo.requirements:
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48704 diff changeset	42 return
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48704 diff changeset	43 if requirements.NODEMAP_REQUIREMENT in repo.requirements:
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48704 diff changeset	44 return
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48704 diff changeset	45 unfi = repo.unfiltered()
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48704 diff changeset	46 delete_nodemap(None, unfi, unfi.changelog)
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48704 diff changeset	47 delete_nodemap(None, repo, unfi.manifestlog._rootstore._revlog)
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48704 diff changeset	48
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48704 diff changeset	49
44352 6c07480d6659 nodemap: add a function to read the data from disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44351 diff changeset	50 def persisted_data(revlog):
6c07480d6659 nodemap: add a function to read the data from disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44351 diff changeset	51 """read the nodemap for a revlog from disk"""
47178 04f2f94836a5 revlog: rename `nodemap_file` to `_nodemap_file` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47166 diff changeset	52 if revlog._nodemap_file is None:
44352 6c07480d6659 nodemap: add a function to read the data from disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44351 diff changeset	53 return None
47178 04f2f94836a5 revlog: rename `nodemap_file` to `_nodemap_file` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47166 diff changeset	54 pdata = revlog.opener.tryread(revlog._nodemap_file)
44354 2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	55 if not pdata:
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	56 return None
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	57 offset = 0
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	58 (version,) = S_VERSION.unpack(pdata[offset : offset + S_VERSION.size])
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	59 if version != ONDISK_VERSION:
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	60 return None
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	61 offset += S_VERSION.size
44369 e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44367 diff changeset	62 headers = S_HEADER.unpack(pdata[offset : offset + S_HEADER.size])
44525 6c906eaedd0d nodemap: track the tip_node for validation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44509 diff changeset	63 uid_size, tip_rev, data_length, data_unused, tip_node_size = headers
44354 2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	64 offset += S_HEADER.size
44365 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	65 docket = NodeMapDocket(pdata[offset : offset + uid_size])
44525 6c906eaedd0d nodemap: track the tip_node for validation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44509 diff changeset	66 offset += uid_size
44369 e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44367 diff changeset	67 docket.tip_rev = tip_rev
44525 6c906eaedd0d nodemap: track the tip_node for validation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44509 diff changeset	68 docket.tip_node = pdata[offset : offset + tip_node_size]
44370 8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	69 docket.data_length = data_length
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	70 docket.data_unused = data_unused
44354 2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	71
44365 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	72 filename = _rawdata_filepath(revlog, docket)
44869 5e3c718692bb nodemap: drop the 'exp-' prefix for internal opener option Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44867 diff changeset	73 use_mmap = revlog.opener.options.get(b"persistent-nodemap.mmap")
47887 52018f8ef020 persistent-nodemap: introduce a test to highlight possible race Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47457 diff changeset	74
52018f8ef020 persistent-nodemap: introduce a test to highlight possible race Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47457 diff changeset	75 test_race_hook_1()
44397 f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44374 diff changeset	76 try:
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44374 diff changeset	77 with revlog.opener(filename) as fd:
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44374 diff changeset	78 if use_mmap:
46971 a3720569a43f nodemap: deal with data mmap error Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46796 diff changeset	79 try:
a3720569a43f nodemap: deal with data mmap error Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46796 diff changeset	80 data = util.buffer(util.mmapread(fd, data_length))
a3720569a43f nodemap: deal with data mmap error Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46796 diff changeset	81 except ValueError:
a3720569a43f nodemap: deal with data mmap error Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46796 diff changeset	82 # raised when the read file is too small
a3720569a43f nodemap: deal with data mmap error Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46796 diff changeset	83 data = b''
44397 f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44374 diff changeset	84 else:
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44374 diff changeset	85 data = fd.read(data_length)
46321 d32e7ed81f4f persistent-nodemap: catch the right exception on python Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46312 diff changeset	86 except (IOError, OSError) as e:
46090 8ff2d8359d0f persistent-nodemap: properly ignore non-existent `.nd` data file Simon Sapin <simon-commits@exyr.org> parents: 45957 diff changeset	87 if e.errno == errno.ENOENT:
8ff2d8359d0f persistent-nodemap: properly ignore non-existent `.nd` data file Simon Sapin <simon-commits@exyr.org> parents: 45957 diff changeset	88 return None
8ff2d8359d0f persistent-nodemap: properly ignore non-existent `.nd` data file Simon Sapin <simon-commits@exyr.org> parents: 45957 diff changeset	89 else:
44397 f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44374 diff changeset	90 raise
44373 c7eebdb15139 nodemap: never read more than the expected data amount Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44372 diff changeset	91 if len(data) < data_length:
c7eebdb15139 nodemap: never read more than the expected data amount Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44372 diff changeset	92 return None
c7eebdb15139 nodemap: never read more than the expected data amount Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44372 diff changeset	93 return docket, data
44352 6c07480d6659 nodemap: add a function to read the data from disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44351 diff changeset	94
6c07480d6659 nodemap: add a function to read the data from disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44351 diff changeset	95
44351 5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	96 def setup_persistent_nodemap(tr, revlog):
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	97 """Install whatever is needed transaction side to persist a nodemap on disk
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	98
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	99 (only actually persist the nodemap if this is relevant for this revlog)
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	100 """
44353 daad3aace942 nodemap: only use persistent nodemap for non-inlined revlog Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44352 diff changeset	101 if revlog._inline:
daad3aace942 nodemap: only use persistent nodemap for non-inlined revlog Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44352 diff changeset	102 return # inlined revlog are too small for this to be relevant
47178 04f2f94836a5 revlog: rename `nodemap_file` to `_nodemap_file` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47166 diff changeset	103 if revlog._nodemap_file is None:
44351 5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	104 return # we do not use persistent_nodemap on this revlog
44527 448d700e0d27 nodemap: make sure the nodemap docket is updated after the changelog Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44526 diff changeset	105
448d700e0d27 nodemap: make sure the nodemap docket is updated after the changelog Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44526 diff changeset	106 # we need to happen after the changelog finalization, in that use "cl-"
47178 04f2f94836a5 revlog: rename `nodemap_file` to `_nodemap_file` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47166 diff changeset	107 callback_id = b"nm-revlog-persistent-nodemap-%s" % revlog._nodemap_file
44351 5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	108 if tr.hasfinalize(callback_id):
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	109 return # no need to register again
44526 64e2f603de9d nodemap: make sure hooks have access to an up-to-date version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44525 diff changeset	110 tr.addpending(
46529 83f037acf71a nodemap: make `_persist_nodemap` a public function Pulkit Goyal <7895pulkit@gmail.com> parents: 46321 diff changeset	111 callback_id, lambda tr: persist_nodemap(tr, revlog, pending=True)
44526 64e2f603de9d nodemap: make sure hooks have access to an up-to-date version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44525 diff changeset	112 )
46529 83f037acf71a nodemap: make `_persist_nodemap` a public function Pulkit Goyal <7895pulkit@gmail.com> parents: 46321 diff changeset	113 tr.addfinalize(callback_id, lambda tr: persist_nodemap(tr, revlog))
44509 3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44506 diff changeset	114
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44506 diff changeset	115
49037 642e31cb55f0 py3: use class X: instead of class X(object): Gregory Szorc <gregory.szorc@gmail.com> parents: 48966 diff changeset	116 class _NoTransaction:
45957 89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44869 diff changeset	117 """transaction like object to update the nodemap outside a transaction"""
44509 3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44506 diff changeset	118
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44506 diff changeset	119 def __init__(self):
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44506 diff changeset	120 self._postclose = {}
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44506 diff changeset	121
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44506 diff changeset	122 def addpostclose(self, callback_id, callback_func):
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44506 diff changeset	123 self._postclose[callback_id] = callback_func
44351 5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	124
44638 01b0805534bb nodemap: make sure on disk change get rolled back with the transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44527 diff changeset	125 def registertmp(self, args, *kwargs):
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44527 diff changeset	126 pass
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44527 diff changeset	127
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44527 diff changeset	128 def addbackup(self, args, *kwargs):
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44527 diff changeset	129 pass
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44527 diff changeset	130
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44527 diff changeset	131 def add(self, args, *kwargs):
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44527 diff changeset	132 pass
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44527 diff changeset	133
44639 99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44638 diff changeset	134 def addabort(self, args, *kwargs):
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44638 diff changeset	135 pass
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44638 diff changeset	136
44865 6493f0a567c2 nodemap: add a new mode option, with an optional "warn" value Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44740 diff changeset	137 def _report(self, *args):
6493f0a567c2 nodemap: add a new mode option, with an optional "warn" value Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44740 diff changeset	138 pass
6493f0a567c2 nodemap: add a new mode option, with an optional "warn" value Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44740 diff changeset	139
44351 5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	140
44465 336ec75ed1ac nodemap: warm the persistent nodemap on disk with debugupdatecache Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44397 diff changeset	141 def update_persistent_nodemap(revlog):
336ec75ed1ac nodemap: warm the persistent nodemap on disk with debugupdatecache Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44397 diff changeset	142 """update the persistent nodemap right now
336ec75ed1ac nodemap: warm the persistent nodemap on disk with debugupdatecache Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44397 diff changeset	143
336ec75ed1ac nodemap: warm the persistent nodemap on disk with debugupdatecache Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44397 diff changeset	144 To be used for updating the nodemap on disk outside of a normal transaction
336ec75ed1ac nodemap: warm the persistent nodemap on disk with debugupdatecache Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44397 diff changeset	145 setup (eg, `debugupdatecache`).
336ec75ed1ac nodemap: warm the persistent nodemap on disk with debugupdatecache Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44397 diff changeset	146 """
44740 b4537125eb3c nodemap: skip persistent nodemap warming for revlog not using it Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44697 diff changeset	147 if revlog._inline:
b4537125eb3c nodemap: skip persistent nodemap warming for revlog not using it Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44697 diff changeset	148 return # inlined revlog are too small for this to be relevant
47178 04f2f94836a5 revlog: rename `nodemap_file` to `_nodemap_file` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47166 diff changeset	149 if revlog._nodemap_file is None:
44740 b4537125eb3c nodemap: skip persistent nodemap warming for revlog not using it Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44697 diff changeset	150 return # we do not use persistent_nodemap on this revlog
b4537125eb3c nodemap: skip persistent nodemap warming for revlog not using it Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44697 diff changeset	151
44509 3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44506 diff changeset	152 notr = _NoTransaction()
46529 83f037acf71a nodemap: make `_persist_nodemap` a public function Pulkit Goyal <7895pulkit@gmail.com> parents: 46321 diff changeset	153 persist_nodemap(notr, revlog)
44509 3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44506 diff changeset	154 for k in sorted(notr._postclose):
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44506 diff changeset	155 notr._postclose[k](None)
44465 336ec75ed1ac nodemap: warm the persistent nodemap on disk with debugupdatecache Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44397 diff changeset	156
336ec75ed1ac nodemap: warm the persistent nodemap on disk with debugupdatecache Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44397 diff changeset	157
46576 67b5fafd3a46 upgrade: speed up when we have only nodemap to downgrade Pulkit Goyal <7895pulkit@gmail.com> parents: 46533 diff changeset	158 def delete_nodemap(tr, repo, revlog):
47457 f70ca39d0ab8 nodemap: fix some comment formatting Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47323 diff changeset	159 """Delete nodemap data on disk for a given revlog"""
48704 d55c4472bbb6 persistent-nodemap: properly delete all nodemap files on downgrade Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48335 diff changeset	160 prefix = revlog.radix
d55c4472bbb6 persistent-nodemap: properly delete all nodemap files on downgrade Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48335 diff changeset	161 pattern = re.compile(br"(^\|/)%s(-[0-9a-f]+\.nd\|\.n(\.a)?)$" % prefix)
d55c4472bbb6 persistent-nodemap: properly delete all nodemap files on downgrade Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48335 diff changeset	162 dirpath = revlog.opener.dirname(revlog._indexfile)
d55c4472bbb6 persistent-nodemap: properly delete all nodemap files on downgrade Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48335 diff changeset	163 for f in revlog.opener.listdir(dirpath):
d55c4472bbb6 persistent-nodemap: properly delete all nodemap files on downgrade Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48335 diff changeset	164 if pattern.match(f):
d55c4472bbb6 persistent-nodemap: properly delete all nodemap files on downgrade Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48335 diff changeset	165 repo.svfs.tryunlink(f)
46576 67b5fafd3a46 upgrade: speed up when we have only nodemap to downgrade Pulkit Goyal <7895pulkit@gmail.com> parents: 46533 diff changeset	166
67b5fafd3a46 upgrade: speed up when we have only nodemap to downgrade Pulkit Goyal <7895pulkit@gmail.com> parents: 46533 diff changeset	167
46533 98e39f04d60e upgrade: implement partial upgrade for upgrading persistent-nodemap Pulkit Goyal <7895pulkit@gmail.com> parents: 46531 diff changeset	168 def persist_nodemap(tr, revlog, pending=False, force=False):
45957 89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44869 diff changeset	169 """Write nodemap data on disk for a given revlog"""
44351 5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	170 if getattr(revlog, 'filteredrevs', ()):
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	171 raise error.ProgrammingError(
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	172 "cannot persist nodemap of a filtered changelog"
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	173 )
47178 04f2f94836a5 revlog: rename `nodemap_file` to `_nodemap_file` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47166 diff changeset	174 if revlog._nodemap_file is None:
46533 98e39f04d60e upgrade: implement partial upgrade for upgrading persistent-nodemap Pulkit Goyal <7895pulkit@gmail.com> parents: 46531 diff changeset	175 if force:
47178 04f2f94836a5 revlog: rename `nodemap_file` to `_nodemap_file` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47166 diff changeset	176 revlog._nodemap_file = get_nodemap_file(revlog)
46533 98e39f04d60e upgrade: implement partial upgrade for upgrading persistent-nodemap Pulkit Goyal <7895pulkit@gmail.com> parents: 46531 diff changeset	177 else:
98e39f04d60e upgrade: implement partial upgrade for upgrading persistent-nodemap Pulkit Goyal <7895pulkit@gmail.com> parents: 46531 diff changeset	178 msg = "calling persist nodemap on a revlog without the feature enabled"
98e39f04d60e upgrade: implement partial upgrade for upgrading persistent-nodemap Pulkit Goyal <7895pulkit@gmail.com> parents: 46531 diff changeset	179 raise error.ProgrammingError(msg)
44367 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	180
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	181 can_incremental = util.safehasattr(revlog.index, "nodemap_data_incremental")
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	182 ondisk_docket = revlog._nodemap_docket
44397 f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44374 diff changeset	183 feed_data = util.safehasattr(revlog.index, "update_nodemap_data")
44869 5e3c718692bb nodemap: drop the 'exp-' prefix for internal opener option Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44867 diff changeset	184 use_mmap = revlog.opener.options.get(b"persistent-nodemap.mmap")
44367 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	185
44371 1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	186 data = None
44367 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	187 # first attemp an incremental update of the data
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	188 if can_incremental and ondisk_docket is not None:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	189 target_docket = revlog._nodemap_docket.copy()
44371 1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	190 (
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	191 src_docket,
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	192 data_changed_count,
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	193 data,
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	194 ) = revlog.index.nodemap_data_incremental()
44640 c70bcaf7927b nodemap: automatically "vacuum" the persistent nodemap when too sparse Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44639 diff changeset	195 new_length = target_docket.data_length + len(data)
c70bcaf7927b nodemap: automatically "vacuum" the persistent nodemap when too sparse Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44639 diff changeset	196 new_unused = target_docket.data_unused + data_changed_count
44371 1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	197 if src_docket != target_docket:
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	198 data = None
44640 c70bcaf7927b nodemap: automatically "vacuum" the persistent nodemap when too sparse Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44639 diff changeset	199 elif new_length <= (new_unused * 10): # under 10% of unused data
c70bcaf7927b nodemap: automatically "vacuum" the persistent nodemap when too sparse Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44639 diff changeset	200 data = None
44371 1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	201 else:
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	202 datafile = _rawdata_filepath(revlog, target_docket)
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	203 # EXP-TODO: if this is a cache, this should use a cache vfs, not a
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	204 # store vfs
44638 01b0805534bb nodemap: make sure on disk change get rolled back with the transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44527 diff changeset	205 tr.add(datafile, target_docket.data_length)
44372 2ea6a67ff502 nodemap: write new data from the expected current data length Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44371 diff changeset	206 with revlog.opener(datafile, b'r+') as fd:
2ea6a67ff502 nodemap: write new data from the expected current data length Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44371 diff changeset	207 fd.seek(target_docket.data_length)
44371 1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	208 fd.write(data)
44397 f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44374 diff changeset	209 if feed_data:
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44374 diff changeset	210 if use_mmap:
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44374 diff changeset	211 fd.seek(0)
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44374 diff changeset	212 new_data = fd.read(new_length)
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44374 diff changeset	213 else:
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44374 diff changeset	214 fd.flush()
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44374 diff changeset	215 new_data = util.buffer(util.mmapread(fd, new_length))
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44374 diff changeset	216 target_docket.data_length = new_length
44640 c70bcaf7927b nodemap: automatically "vacuum" the persistent nodemap when too sparse Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44639 diff changeset	217 target_docket.data_unused = new_unused
44371 1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	218
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	219 if data is None:
44367 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	220 # otherwise fallback to a full new export
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	221 target_docket = NodeMapDocket()
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	222 datafile = _rawdata_filepath(revlog, target_docket)
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	223 if util.safehasattr(revlog.index, "nodemap_data_all"):
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	224 data = revlog.index.nodemap_data_all()
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	225 else:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	226 data = persistent_data(revlog.index)
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	227 # EXP-TODO: if this is a cache, this should use a cache vfs, not a
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	228 # store vfs
44639 99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44638 diff changeset	229
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44638 diff changeset	230 tryunlink = revlog.opener.tryunlink
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44638 diff changeset	231
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44638 diff changeset	232 def abortck(tr):
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44638 diff changeset	233 tryunlink(datafile)
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44638 diff changeset	234
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44638 diff changeset	235 callback_id = b"delete-%s" % datafile
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44638 diff changeset	236
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44638 diff changeset	237 # some flavor of the transaction abort does not cleanup new file, it
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44638 diff changeset	238 # simply empty them.
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44638 diff changeset	239 tr.addabort(callback_id, abortck)
44397 f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44374 diff changeset	240 with revlog.opener(datafile, b'w+') as fd:
44367 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	241 fd.write(data)
44397 f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44374 diff changeset	242 if feed_data:
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44374 diff changeset	243 if use_mmap:
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44374 diff changeset	244 new_data = data
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44374 diff changeset	245 else:
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44374 diff changeset	246 fd.flush()
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44374 diff changeset	247 new_data = util.buffer(util.mmapread(fd, len(data)))
44370 8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	248 target_docket.data_length = len(data)
44369 e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44367 diff changeset	249 target_docket.tip_rev = revlog.tiprev()
44525 6c906eaedd0d nodemap: track the tip_node for validation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44509 diff changeset	250 target_docket.tip_node = revlog.node(target_docket.tip_rev)
44367 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	251 # EXP-TODO: if this is a cache, this should use a cache vfs, not a
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	252 # store vfs
47178 04f2f94836a5 revlog: rename `nodemap_file` to `_nodemap_file` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47166 diff changeset	253 file_path = revlog._nodemap_file
44526 64e2f603de9d nodemap: make sure hooks have access to an up-to-date version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44525 diff changeset	254 if pending:
64e2f603de9d nodemap: make sure hooks have access to an up-to-date version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44525 diff changeset	255 file_path += b'.a'
44638 01b0805534bb nodemap: make sure on disk change get rolled back with the transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44527 diff changeset	256 tr.registertmp(file_path)
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44527 diff changeset	257 else:
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44527 diff changeset	258 tr.addbackup(file_path)
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44527 diff changeset	259
44526 64e2f603de9d nodemap: make sure hooks have access to an up-to-date version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44525 diff changeset	260 with revlog.opener(file_path, b'w', atomictemp=True) as fp:
44367 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	261 fp.write(target_docket.serialize())
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	262 revlog._nodemap_docket = target_docket
44397 f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44374 diff changeset	263 if feed_data:
44374 6ecc34b31137 nodemap: update the index with the newly written data (when appropriate) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44373 diff changeset	264 revlog.index.update_nodemap_data(target_docket, new_data)
6ecc34b31137 nodemap: update the index with the newly written data (when appropriate) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44373 diff changeset	265
44367 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	266 # search for old index file in all cases, some older process might have
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	267 # left one behind.
44365 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	268 olds = _other_rawdata_filepath(revlog, target_docket)
44355 563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	269 if olds:
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	270 realvfs = getattr(revlog, '_realopener', revlog.opener)
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	271
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	272 def cleanup(tr):
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	273 for oldfile in olds:
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	274 realvfs.tryunlink(oldfile)
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	275
47178 04f2f94836a5 revlog: rename `nodemap_file` to `_nodemap_file` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47166 diff changeset	276 callback_id = b"revlog-cleanup-nodemap-%s" % revlog._nodemap_file
44509 3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44506 diff changeset	277 tr.addpostclose(callback_id, cleanup)
44354 2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	278
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	279
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	280 ### Nodemap docket file
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	281 #
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	282 # The nodemap data are stored on disk using 2 files:
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	283 #
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	284 # * a raw data files containing a persistent nodemap
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	285 # (see `Nodemap Trie` section)
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	286 #
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	287 # * a small "docket" file containing medatadata
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	288 #
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	289 # While the nodemap data can be multiple tens of megabytes, the "docket" is
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	290 # small, it is easy to update it automatically or to duplicated its content
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	291 # during a transaction.
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	292 #
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	293 # Multiple raw data can exist at the same time (The currently valid one and a
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	294 # new one beind used by an in progress transaction). To accomodate this, the
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	295 # filename hosting the raw data has a variable parts. The exact filename is
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	296 # specified inside the "docket" file.
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	297 #
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	298 # The docket file contains information to find, qualify and validate the raw
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	299 # data. Its content is currently very light, but it will expand as the on disk
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	300 # nodemap gains the necessary features to be used in production.
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	301
44867 261e71752d1f nodemap: move on disk file to version 1 Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44866 diff changeset	302 ONDISK_VERSION = 1
44354 2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	303 S_VERSION = struct.Struct(">B")
44525 6c906eaedd0d nodemap: track the tip_node for validation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44509 diff changeset	304 S_HEADER = struct.Struct(">BQQQQ")
44354 2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	305
47321 9a3aa54774ff persistent-nodemap: add a way to make the picked uid predictable Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47251 diff changeset	306
49037 642e31cb55f0 py3: use class X: instead of class X(object): Gregory Szorc <gregory.szorc@gmail.com> parents: 48966 diff changeset	307 class NodeMapDocket:
44365 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	308 """metadata associated with persistent nodemap data
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	309
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	310 The persistent data may come from disk or be on their way to disk.
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	311 """
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	312
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	313 def __init__(self, uid=None):
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	314 if uid is None:
47323 7ea39d633cf3 docket: move the uid logic in the `revlogutils.docket` module Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47322 diff changeset	315 uid = docket_mod.make_uid()
44506 283fd803afa5 nodemap: document the docket attributes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44480 diff changeset	316 # a unique identifier for the data file:
283fd803afa5 nodemap: document the docket attributes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44480 diff changeset	317 # - When new data are appended, it is preserved.
283fd803afa5 nodemap: document the docket attributes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44480 diff changeset	318 # - When a new data file is created, a new identifier is generated.
44365 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	319 self.uid = uid
44506 283fd803afa5 nodemap: document the docket attributes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44480 diff changeset	320 # the tipmost revision stored in the data file. This revision and all
283fd803afa5 nodemap: document the docket attributes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44480 diff changeset	321 # revision before it are expected to be encoded in the data file.
44369 e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44367 diff changeset	322 self.tip_rev = None
44525 6c906eaedd0d nodemap: track the tip_node for validation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44509 diff changeset	323 # the node of that tipmost revision, if it mismatch the current index
6c906eaedd0d nodemap: track the tip_node for validation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44509 diff changeset	324 # data the docket is not valid for the current index and should be
6c906eaedd0d nodemap: track the tip_node for validation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44509 diff changeset	325 # discarded.
6c906eaedd0d nodemap: track the tip_node for validation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44509 diff changeset	326 #
6c906eaedd0d nodemap: track the tip_node for validation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44509 diff changeset	327 # note: this method is not perfect as some destructive operation could
6c906eaedd0d nodemap: track the tip_node for validation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44509 diff changeset	328 # preserve the same tip_rev + tip_node while altering lower revision.
6c906eaedd0d nodemap: track the tip_node for validation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44509 diff changeset	329 # However this multiple other caches have the same vulnerability (eg:
6c906eaedd0d nodemap: track the tip_node for validation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44509 diff changeset	330 # brancmap cache).
6c906eaedd0d nodemap: track the tip_node for validation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44509 diff changeset	331 self.tip_node = None
44506 283fd803afa5 nodemap: document the docket attributes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44480 diff changeset	332 # the size (in bytes) of the persisted data to encode the nodemap valid
283fd803afa5 nodemap: document the docket attributes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44480 diff changeset	333 # for `tip_rev`.
283fd803afa5 nodemap: document the docket attributes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44480 diff changeset	334 # - data file shorter than this are corrupted,
283fd803afa5 nodemap: document the docket attributes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44480 diff changeset	335 # - any extra data should be ignored.
44370 8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	336 self.data_length = None
44506 283fd803afa5 nodemap: document the docket attributes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44480 diff changeset	337 # the amount (in bytes) of "dead" data, still in the data file but no
283fd803afa5 nodemap: document the docket attributes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44480 diff changeset	338 # longer used for the nodemap.
44370 8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	339 self.data_unused = 0
44365 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	340
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	341 def copy(self):
44369 e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44367 diff changeset	342 new = NodeMapDocket(uid=self.uid)
e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44367 diff changeset	343 new.tip_rev = self.tip_rev
44525 6c906eaedd0d nodemap: track the tip_node for validation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44509 diff changeset	344 new.tip_node = self.tip_node
44370 8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	345 new.data_length = self.data_length
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	346 new.data_unused = self.data_unused
44369 e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44367 diff changeset	347 return new
44365 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	348
44371 1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	349 def __cmp__(self, other):
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	350 if self.uid < other.uid:
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	351 return -1
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	352 if self.uid > other.uid:
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	353 return 1
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	354 elif self.data_length < other.data_length:
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	355 return -1
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	356 elif self.data_length > other.data_length:
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	357 return 1
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	358 return 0
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	359
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	360 def __eq__(self, other):
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	361 return self.uid == other.uid and self.data_length == other.data_length
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	362
44365 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	363 def serialize(self):
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	364 """return serialized bytes for a docket using the passed uid"""
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	365 data = []
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	366 data.append(S_VERSION.pack(ONDISK_VERSION))
44370 8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	367 headers = (
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	368 len(self.uid),
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	369 self.tip_rev,
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	370 self.data_length,
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	371 self.data_unused,
44525 6c906eaedd0d nodemap: track the tip_node for validation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44509 diff changeset	372 len(self.tip_node),
44370 8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	373 )
44369 e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44367 diff changeset	374 data.append(S_HEADER.pack(*headers))
44365 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	375 data.append(self.uid)
44525 6c906eaedd0d nodemap: track the tip_node for validation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44509 diff changeset	376 data.append(self.tip_node)
44365 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	377 return b''.join(data)
44354 2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	378
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	379
44365 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	380 def _rawdata_filepath(revlog, docket):
44354 2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	381 """The (vfs relative) nodemap's rawdata file for a given uid"""
47166 09338a2d5c14 revlog: also use radix when computing nodemap data file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47165 diff changeset	382 prefix = revlog.radix
44365 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	383 return b"%s-%s.nd" % (prefix, docket.uid)
44351 5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	384
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	385
44365 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	386 def _other_rawdata_filepath(revlog, docket):
47166 09338a2d5c14 revlog: also use radix when computing nodemap data file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47165 diff changeset	387 prefix = revlog.radix
44480 6aee0647e026 nodemap: fix missing r-prefix on regular expression Augie Fackler <augie@google.com> parents: 44465 diff changeset	388 pattern = re.compile(br"(^\|/)%s-[0-9a-f]+\.nd$" % prefix)
44365 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	389 new_file_path = _rawdata_filepath(revlog, docket)
44355 563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	390 new_file_name = revlog.opener.basename(new_file_path)
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	391 dirpath = revlog.opener.dirname(new_file_path)
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	392 others = []
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	393 for f in revlog.opener.listdir(dirpath):
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	394 if pattern.match(f) and f != new_file_name:
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	395 others.append(f)
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	396 return others
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	397
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	398
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	399 ### Nodemap Trie
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	400 #
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	401 # This is a simple reference implementation to compute and persist a nodemap
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	402 # trie. This reference implementation is write only. The python version of this
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	403 # is not expected to be actually used, since it wont provide performance
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	404 # improvement over existing non-persistent C implementation.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	405 #
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	406 # The nodemap is persisted as Trie using 4bits-address/16-entries block. each
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	407 # revision can be adressed using its node shortest prefix.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	408 #
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	409 # The trie is stored as a sequence of block. Each block contains 16 entries
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	410 # (signed 64bit integer, big endian). Each entry can be one of the following:
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	411 #
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	412 # * value >= 0 -> index of sub-block
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	413 # * value == -1 -> no value
46199 8e7ce6555ea7 nodemap: match comment to actual code Joerg Sonnenberger <joerg@bec.de> parents: 46114 diff changeset	414 # * value < -1 -> encoded revision: rev = -(value+2)
8e7ce6555ea7 nodemap: match comment to actual code Joerg Sonnenberger <joerg@bec.de> parents: 46114 diff changeset	415 #
8e7ce6555ea7 nodemap: match comment to actual code Joerg Sonnenberger <joerg@bec.de> parents: 46114 diff changeset	416 # See REV_OFFSET and _transform_rev below.
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	417 #
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	418 # The implementation focus on simplicity, not on performance. A Rust
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	419 # implementation should provide a efficient version of the same binary
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	420 # persistence. This reference python implementation is never meant to be
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	421 # extensively use in production.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	422
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	423
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	424 def persistent_data(index):
45957 89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44869 diff changeset	425 """return the persistent binary form for a nodemap for a given index"""
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	426 trie = _build_trie(index)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	427 return _persist_trie(trie)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	428
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	429
44367 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	430 def update_persistent_data(index, root, max_idx, last_rev):
45957 89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44869 diff changeset	431 """return the incremental update for persistent nodemap from a given index"""
44370 8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	432 changed_block, trie = _update_trie(index, root, last_rev)
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	433 return (
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	434 changed_block * S_BLOCK.size,
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	435 _persist_trie(trie, existing_idx=max_idx),
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	436 )
44367 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	437
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	438
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	439 S_BLOCK = struct.Struct(">" + ("l" * 16))
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	440
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	441 NO_ENTRY = -1
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	442 # rev 0 need to be -2 because 0 is used by block, -1 is a special value.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	443 REV_OFFSET = 2
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	444
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	445
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	446 def _transform_rev(rev):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	447 """Return the number used to represent the rev in the tree.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	448
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	449 (or retrieve a rev number from such representation)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	450
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	451 Note that this is an involution, a function equal to its inverse (i.e.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	452 which gives the identity when applied to itself).
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	453 """
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	454 return -(rev + REV_OFFSET)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	455
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	456
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	457 def _to_int(hex_digit):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	458 """turn an hexadecimal digit into a proper integer"""
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	459 return int(hex_digit, 16)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	460
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	461
44358 7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44357 diff changeset	462 class Block(dict):
7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44357 diff changeset	463 """represent a block of the Trie
7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44357 diff changeset	464
7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44357 diff changeset	465 contains up to 16 entry indexed from 0 to 15"""
7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44357 diff changeset	466
44364 f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44362 diff changeset	467 def __init__(self):
f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44362 diff changeset	468 super(Block, self).__init__()
f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44362 diff changeset	469 # If this block exist on disk, here is its ID
f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44362 diff changeset	470 self.ondisk_id = None
f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44362 diff changeset	471
44359 55b12f2593c1 nodemap: move the iteratio inside the Block object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44358 diff changeset	472 def __iter__(self):
55b12f2593c1 nodemap: move the iteratio inside the Block object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44358 diff changeset	473 return iter(self.get(i) for i in range(16))
55b12f2593c1 nodemap: move the iteratio inside the Block object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44358 diff changeset	474
44358 7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44357 diff changeset	475
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	476 def _build_trie(index):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	477 """build a nodemap trie
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	478
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	479 The nodemap stores revision number for each unique prefix.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	480
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	481 Each block is a dictionary with keys in `[0, 15]`. Values are either
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	482 another block or a revision number.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	483 """
44358 7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44357 diff changeset	484 root = Block()
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	485 for rev in range(len(index)):
46114 59fa3890d40a node: import symbols explicitly Joerg Sonnenberger <joerg@bec.de> parents: 46090 diff changeset	486 current_hex = hex(index[rev][7])
59fa3890d40a node: import symbols explicitly Joerg Sonnenberger <joerg@bec.de> parents: 46090 diff changeset	487 _insert_into_block(index, 0, root, rev, current_hex)
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	488 return root
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	489
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	490
44367 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	491 def _update_trie(index, root, last_rev):
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	492 """consume"""
44370 8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	493 changed = 0
44367 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	494 for rev in range(last_rev + 1, len(index)):
46114 59fa3890d40a node: import symbols explicitly Joerg Sonnenberger <joerg@bec.de> parents: 46090 diff changeset	495 current_hex = hex(index[rev][7])
59fa3890d40a node: import symbols explicitly Joerg Sonnenberger <joerg@bec.de> parents: 46090 diff changeset	496 changed += _insert_into_block(index, 0, root, rev, current_hex)
44370 8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	497 return changed, root
44367 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	498
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	499
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	500 def _insert_into_block(index, level, block, current_rev, current_hex):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	501 """insert a new revision in a block
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	502
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	503 index: the index we are adding revision for
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	504 level: the depth of the current block in the trie
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	505 block: the block currently being considered
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	506 current_rev: the revision number we are adding
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	507 current_hex: the hexadecimal representation of the of that revision
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	508 """
44370 8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	509 changed = 1
44367 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	510 if block.ondisk_id is not None:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	511 block.ondisk_id = None
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	512 hex_digit = _to_int(current_hex[level : level + 1])
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	513 entry = block.get(hex_digit)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	514 if entry is None:
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	515 # no entry, simply store the revision number
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	516 block[hex_digit] = current_rev
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	517 elif isinstance(entry, dict):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	518 # need to recurse to an underlying block
44370 8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	519 changed += _insert_into_block(
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	520 index, level + 1, entry, current_rev, current_hex
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	521 )
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	522 else:
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	523 # collision with a previously unique prefix, inserting new
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	524 # vertices to fit both entry.
46114 59fa3890d40a node: import symbols explicitly Joerg Sonnenberger <joerg@bec.de> parents: 46090 diff changeset	525 other_hex = hex(index[entry][7])
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	526 other_rev = entry
44358 7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44357 diff changeset	527 new = Block()
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	528 block[hex_digit] = new
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	529 _insert_into_block(index, level + 1, new, other_rev, other_hex)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	530 _insert_into_block(index, level + 1, new, current_rev, current_hex)
44370 8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	531 return changed
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	532
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	533
44367 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	534 def _persist_trie(root, existing_idx=None):
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	535 """turn a nodemap trie into persistent binary data
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	536
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	537 See `_build_trie` for nodemap trie structure"""
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	538 block_map = {}
44367 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	539 if existing_idx is not None:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	540 base_idx = existing_idx + 1
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	541 else:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	542 base_idx = 0
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	543 chunks = []
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	544 for tn in _walk_trie(root):
44367 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	545 if tn.ondisk_id is not None:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	546 block_map[id(tn)] = tn.ondisk_id
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	547 else:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	548 block_map[id(tn)] = len(chunks) + base_idx
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	549 chunks.append(_persist_block(tn, block_map))
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	550 return b''.join(chunks)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	551
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	552
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	553 def _walk_trie(block):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	554 """yield all the block in a trie
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	555
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	556 Children blocks are always yield before their parent block.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	557 """
44696 ed475420e0af nodemap: rename a variable to prevent shawoding "_" Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44640 diff changeset	558 for (__, item) in sorted(block.items()):
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	559 if isinstance(item, dict):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	560 for sub_block in _walk_trie(item):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	561 yield sub_block
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	562 yield block
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	563
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	564
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	565 def _persist_block(block_node, block_map):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	566 """produce persistent binary data for a single block
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	567
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	568 Children block are assumed to be already persisted and present in
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	569 block_map.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	570 """
44359 55b12f2593c1 nodemap: move the iteratio inside the Block object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44358 diff changeset	571 data = tuple(_to_value(v, block_map) for v in block_node)
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	572 return S_BLOCK.pack(*data)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	573
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	574
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	575 def _to_value(item, block_map):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	576 """persist any value as an integer"""
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	577 if item is None:
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	578 return NO_ENTRY
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	579 elif isinstance(item, dict):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	580 return block_map[id(item)]
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	581 else:
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	582 return _transform_rev(item)
44360 78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	583
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	584
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	585 def parse_data(data):
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	586 """parse parse nodemap data into a nodemap Trie"""
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	587 if (len(data) % S_BLOCK.size) != 0:
46391 98c816e89fac nodemap: convert error message to bytes Matt Harbison <matt_harbison@yahoo.com> parents: 46321 diff changeset	588 msg = b"nodemap data size is not a multiple of block size (%d): %d"
44360 78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	589 raise error.Abort(msg % (S_BLOCK.size, len(data)))
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	590 if not data:
44367 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	591 return Block(), None
44360 78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	592 block_map = {}
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	593 new_blocks = []
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	594 for i in range(0, len(data), S_BLOCK.size):
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	595 block = Block()
44364 f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44362 diff changeset	596 block.ondisk_id = len(block_map)
f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44362 diff changeset	597 block_map[block.ondisk_id] = block
44360 78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	598 block_data = data[i : i + S_BLOCK.size]
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	599 values = S_BLOCK.unpack(block_data)
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	600 new_blocks.append((block, values))
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	601 for b, values in new_blocks:
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	602 for idx, v in enumerate(values):
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	603 if v == NO_ENTRY:
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	604 continue
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	605 elif v >= 0:
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	606 b[idx] = block_map[v]
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	607 else:
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	608 b[idx] = _transform_rev(v)
44367 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	609 return block, i // S_BLOCK.size
44361 20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	610
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	611
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	612 # debug utility
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	613
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	614
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	615 def check_data(ui, index, data):
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	616 """verify that the provided nodemap data are valid for the given idex"""
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	617 ret = 0
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	618 ui.status((b"revision in index: %d\n") % len(index))
44367 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	619 root, __ = parse_data(data)
44361 20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	620 all_revs = set(_all_revisions(root))
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	621 ui.status((b"revision in nodemap: %d\n") % len(all_revs))
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	622 for r in range(len(index)):
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	623 if r not in all_revs:
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	624 msg = b" revision missing from nodemap: %d\n" % r
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	625 ui.write_err(msg)
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	626 ret = 1
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	627 else:
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	628 all_revs.remove(r)
46114 59fa3890d40a node: import symbols explicitly Joerg Sonnenberger <joerg@bec.de> parents: 46090 diff changeset	629 nm_rev = _find_node(root, hex(index[r][7]))
44362 d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	630 if nm_rev is None:
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	631 msg = b" revision node does not match any entries: %d\n" % r
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	632 ui.write_err(msg)
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	633 ret = 1
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	634 elif nm_rev != r:
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	635 msg = (
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	636 b" revision node does not match the expected revision: "
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	637 b"%d != %d\n" % (r, nm_rev)
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	638 )
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	639 ui.write_err(msg)
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	640 ret = 1
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	641
44361 20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	642 if all_revs:
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	643 for r in sorted(all_revs):
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	644 msg = b" extra revision in nodemap: %d\n" % r
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	645 ui.write_err(msg)
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	646 ret = 1
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	647 return ret
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	648
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	649
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	650 def _all_revisions(root):
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	651 """return all revisions stored in a Trie"""
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	652 for block in _walk_trie(root):
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	653 for v in block:
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	654 if v is None or isinstance(v, Block):
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	655 continue
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	656 yield v
44362 d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	657
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	658
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	659 def _find_node(block, node):
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	660 """find the revision associated with a given node"""
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	661 entry = block.get(_to_int(node[0:1]))
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	662 if isinstance(entry, dict):
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	663 return _find_node(entry, node[1:])
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	664 return entry
46530 835aafb2543f revlog: refactor logic to compute nodemap file in separate function Pulkit Goyal <7895pulkit@gmail.com> parents: 46529 diff changeset	665
835aafb2543f revlog: refactor logic to compute nodemap file in separate function Pulkit Goyal <7895pulkit@gmail.com> parents: 46529 diff changeset	666
47165 24be247a13b4 revlog: stop usage of `_indexfile` to computing nodemap path Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47162 diff changeset	667 def get_nodemap_file(revlog):
47251 4f38ada3fc26 revlog: move the `trypending` logic from the `changelog` to the `revlog` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47178 diff changeset	668 if revlog._trypending:
47165 24be247a13b4 revlog: stop usage of `_indexfile` to computing nodemap path Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47162 diff changeset	669 pending_path = revlog.radix + b".n.a"
24be247a13b4 revlog: stop usage of `_indexfile` to computing nodemap path Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47162 diff changeset	670 if revlog.opener.exists(pending_path):
46530 835aafb2543f revlog: refactor logic to compute nodemap file in separate function Pulkit Goyal <7895pulkit@gmail.com> parents: 46529 diff changeset	671 return pending_path
47165 24be247a13b4 revlog: stop usage of `_indexfile` to computing nodemap path Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47162 diff changeset	672 return revlog.radix + b".n"

Mercurial > public > mercurial-scm > hg-stable

annotate mercurial/revlogutils/nodemap.py @ 49037:642e31cb55f0