mercurial-scm/hg: mercurial/revlogutils/nodemap.py annotate

annotate mercurial/revlogutils/nodemap.py @ 46472:98e39f04d60e

upgrade: implement partial upgrade for upgrading persistent-nodemap Upgrading repositories to use persistent nodemap should be fast and easy as it requires only two things: 1) Updating the requirements 2) Writing a persistent-nodemap on disk For both of the steps above, we don't need to edit existing revlogs. This patch makes upgrade only do the above mentioned two steps if we are only upgarding to use persistent-nodemap feature. Since `nodemap.persist_nodemap()` assumes that there exists a nodemap file for the given revlog if we are trying to call it, this patch adds `force` argument to create a file if does not exist which is true in our upgrade case. The test changes demonstrate that we no longer write nodemap files for manifest after upgrade which I think is desirable. Differential Revision: https://phab.mercurial-scm.org/D9936

author	Pulkit Goyal <7895pulkit@gmail.com>
date	Mon, 01 Feb 2021 00:02:00 +0530
parents	dadb4db55661
children	67b5fafd3a46

rev	line source
44034 ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	1 # nodemap.py - nodemap related code and utilities
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	2 #
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	3 # Copyright 2019 Pierre-Yves David <pierre-yves.david@octobus.net>
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	4 # Copyright 2019 George Racinet <georges.racinet@octobus.net>
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	5 #
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	6 # This software may be used and distributed according to the terms of the
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	7 # GNU General Public License version 2 or any later version.
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	8
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	9 from __future__ import absolute_import
44307 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	10
44363 f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44340 diff changeset	11 import errno
44311 2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	12 import os
44312 563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44311 diff changeset	13 import re
44307 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	14 import struct
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	15
46113 59fa3890d40a node: import symbols explicitly Joerg Sonnenberger <joerg@bec.de> parents: 46089 diff changeset	16 from ..node import hex
44788 6493f0a567c2 nodemap: add a new mode option, with an optional "warn" value Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44745 diff changeset	17
44307 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	18 from .. import (
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	19 error,
44314 7f4f7ef3133e nodemap: add a optional `nodemap_add_full` method on indexes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44312 diff changeset	20 util,
44307 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	21 )
44034 ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	22
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	23
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	24 class NodeMap(dict):
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	25 def __missing__(self, x):
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	26 raise error.RevlogError(b'unknown node: %s' % x)
44307 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	27
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	28
44309 6c07480d6659 nodemap: add a function to read the data from disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44308 diff changeset	29 def persisted_data(revlog):
6c07480d6659 nodemap: add a function to read the data from disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44308 diff changeset	30 """read the nodemap for a revlog from disk"""
6c07480d6659 nodemap: add a function to read the data from disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44308 diff changeset	31 if revlog.nodemap_file is None:
6c07480d6659 nodemap: add a function to read the data from disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44308 diff changeset	32 return None
44311 2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	33 pdata = revlog.opener.tryread(revlog.nodemap_file)
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	34 if not pdata:
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	35 return None
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	36 offset = 0
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	37 (version,) = S_VERSION.unpack(pdata[offset : offset + S_VERSION.size])
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	38 if version != ONDISK_VERSION:
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	39 return None
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	40 offset += S_VERSION.size
44335 e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44333 diff changeset	41 headers = S_HEADER.unpack(pdata[offset : offset + S_HEADER.size])
44515 6c906eaedd0d nodemap: track the tip_node for validation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44497 diff changeset	42 uid_size, tip_rev, data_length, data_unused, tip_node_size = headers
44311 2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	43 offset += S_HEADER.size
44322 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44321 diff changeset	44 docket = NodeMapDocket(pdata[offset : offset + uid_size])
44515 6c906eaedd0d nodemap: track the tip_node for validation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44497 diff changeset	45 offset += uid_size
44335 e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44333 diff changeset	46 docket.tip_rev = tip_rev
44515 6c906eaedd0d nodemap: track the tip_node for validation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44497 diff changeset	47 docket.tip_node = pdata[offset : offset + tip_node_size]
44336 8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44335 diff changeset	48 docket.data_length = data_length
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44335 diff changeset	49 docket.data_unused = data_unused
44311 2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	50
44322 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44321 diff changeset	51 filename = _rawdata_filepath(revlog, docket)
44792 5e3c718692bb nodemap: drop the 'exp-' prefix for internal opener option Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44790 diff changeset	52 use_mmap = revlog.opener.options.get(b"persistent-nodemap.mmap")
44363 f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44340 diff changeset	53 try:
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44340 diff changeset	54 with revlog.opener(filename) as fd:
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44340 diff changeset	55 if use_mmap:
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44340 diff changeset	56 data = util.buffer(util.mmapread(fd, data_length))
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44340 diff changeset	57 else:
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44340 diff changeset	58 data = fd.read(data_length)
46321 d32e7ed81f4f persistent-nodemap: catch the right exception on python Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46312 diff changeset	59 except (IOError, OSError) as e:
46089 8ff2d8359d0f persistent-nodemap: properly ignore non-existent `.nd` data file Simon Sapin <simon-commits@exyr.org> parents: 45942 diff changeset	60 if e.errno == errno.ENOENT:
8ff2d8359d0f persistent-nodemap: properly ignore non-existent `.nd` data file Simon Sapin <simon-commits@exyr.org> parents: 45942 diff changeset	61 return None
8ff2d8359d0f persistent-nodemap: properly ignore non-existent `.nd` data file Simon Sapin <simon-commits@exyr.org> parents: 45942 diff changeset	62 else:
44363 f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44340 diff changeset	63 raise
44339 c7eebdb15139 nodemap: never read more than the expected data amount Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44338 diff changeset	64 if len(data) < data_length:
c7eebdb15139 nodemap: never read more than the expected data amount Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44338 diff changeset	65 return None
c7eebdb15139 nodemap: never read more than the expected data amount Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44338 diff changeset	66 return docket, data
44309 6c07480d6659 nodemap: add a function to read the data from disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44308 diff changeset	67
6c07480d6659 nodemap: add a function to read the data from disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44308 diff changeset	68
44308 5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44307 diff changeset	69 def setup_persistent_nodemap(tr, revlog):
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44307 diff changeset	70 """Install whatever is needed transaction side to persist a nodemap on disk
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44307 diff changeset	71
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44307 diff changeset	72 (only actually persist the nodemap if this is relevant for this revlog)
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44307 diff changeset	73 """
44310 daad3aace942 nodemap: only use persistent nodemap for non-inlined revlog Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44309 diff changeset	74 if revlog._inline:
daad3aace942 nodemap: only use persistent nodemap for non-inlined revlog Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44309 diff changeset	75 return # inlined revlog are too small for this to be relevant
44308 5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44307 diff changeset	76 if revlog.nodemap_file is None:
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44307 diff changeset	77 return # we do not use persistent_nodemap on this revlog
44517 448d700e0d27 nodemap: make sure the nodemap docket is updated after the changelog Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44516 diff changeset	78
448d700e0d27 nodemap: make sure the nodemap docket is updated after the changelog Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44516 diff changeset	79 # we need to happen after the changelog finalization, in that use "cl-"
448d700e0d27 nodemap: make sure the nodemap docket is updated after the changelog Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44516 diff changeset	80 callback_id = b"nm-revlog-persistent-nodemap-%s" % revlog.nodemap_file
44308 5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44307 diff changeset	81 if tr.hasfinalize(callback_id):
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44307 diff changeset	82 return # no need to register again
44516 64e2f603de9d nodemap: make sure hooks have access to an up-to-date version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44515 diff changeset	83 tr.addpending(
46468 83f037acf71a nodemap: make `_persist_nodemap` a public function Pulkit Goyal <7895pulkit@gmail.com> parents: 46321 diff changeset	84 callback_id, lambda tr: persist_nodemap(tr, revlog, pending=True)
44516 64e2f603de9d nodemap: make sure hooks have access to an up-to-date version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44515 diff changeset	85 )
46468 83f037acf71a nodemap: make `_persist_nodemap` a public function Pulkit Goyal <7895pulkit@gmail.com> parents: 46321 diff changeset	86 tr.addfinalize(callback_id, lambda tr: persist_nodemap(tr, revlog))
44497 3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44494 diff changeset	87
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44494 diff changeset	88
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44494 diff changeset	89 class _NoTransaction(object):
45942 89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44792 diff changeset	90 """transaction like object to update the nodemap outside a transaction"""
44497 3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44494 diff changeset	91
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44494 diff changeset	92 def __init__(self):
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44494 diff changeset	93 self._postclose = {}
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44494 diff changeset	94
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44494 diff changeset	95 def addpostclose(self, callback_id, callback_func):
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44494 diff changeset	96 self._postclose[callback_id] = callback_func
44308 5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44307 diff changeset	97
44634 01b0805534bb nodemap: make sure on disk change get rolled back with the transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44517 diff changeset	98 def registertmp(self, args, *kwargs):
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44517 diff changeset	99 pass
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44517 diff changeset	100
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44517 diff changeset	101 def addbackup(self, args, *kwargs):
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44517 diff changeset	102 pass
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44517 diff changeset	103
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44517 diff changeset	104 def add(self, args, *kwargs):
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44517 diff changeset	105 pass
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44517 diff changeset	106
44635 99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44634 diff changeset	107 def addabort(self, args, *kwargs):
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44634 diff changeset	108 pass
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44634 diff changeset	109
44788 6493f0a567c2 nodemap: add a new mode option, with an optional "warn" value Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44745 diff changeset	110 def _report(self, *args):
6493f0a567c2 nodemap: add a new mode option, with an optional "warn" value Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44745 diff changeset	111 pass
6493f0a567c2 nodemap: add a new mode option, with an optional "warn" value Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44745 diff changeset	112
44308 5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44307 diff changeset	113
44445 336ec75ed1ac nodemap: warm the persistent nodemap on disk with debugupdatecache Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44363 diff changeset	114 def update_persistent_nodemap(revlog):
336ec75ed1ac nodemap: warm the persistent nodemap on disk with debugupdatecache Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44363 diff changeset	115 """update the persistent nodemap right now
336ec75ed1ac nodemap: warm the persistent nodemap on disk with debugupdatecache Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44363 diff changeset	116
336ec75ed1ac nodemap: warm the persistent nodemap on disk with debugupdatecache Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44363 diff changeset	117 To be used for updating the nodemap on disk outside of a normal transaction
336ec75ed1ac nodemap: warm the persistent nodemap on disk with debugupdatecache Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44363 diff changeset	118 setup (eg, `debugupdatecache`).
336ec75ed1ac nodemap: warm the persistent nodemap on disk with debugupdatecache Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44363 diff changeset	119 """
44745 b4537125eb3c nodemap: skip persistent nodemap warming for revlog not using it Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44697 diff changeset	120 if revlog._inline:
b4537125eb3c nodemap: skip persistent nodemap warming for revlog not using it Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44697 diff changeset	121 return # inlined revlog are too small for this to be relevant
b4537125eb3c nodemap: skip persistent nodemap warming for revlog not using it Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44697 diff changeset	122 if revlog.nodemap_file is None:
b4537125eb3c nodemap: skip persistent nodemap warming for revlog not using it Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44697 diff changeset	123 return # we do not use persistent_nodemap on this revlog
b4537125eb3c nodemap: skip persistent nodemap warming for revlog not using it Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44697 diff changeset	124
44497 3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44494 diff changeset	125 notr = _NoTransaction()
46468 83f037acf71a nodemap: make `_persist_nodemap` a public function Pulkit Goyal <7895pulkit@gmail.com> parents: 46321 diff changeset	126 persist_nodemap(notr, revlog)
44497 3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44494 diff changeset	127 for k in sorted(notr._postclose):
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44494 diff changeset	128 notr._postclose[k](None)
44445 336ec75ed1ac nodemap: warm the persistent nodemap on disk with debugupdatecache Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44363 diff changeset	129
336ec75ed1ac nodemap: warm the persistent nodemap on disk with debugupdatecache Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44363 diff changeset	130
46472 98e39f04d60e upgrade: implement partial upgrade for upgrading persistent-nodemap Pulkit Goyal <7895pulkit@gmail.com> parents: 46470 diff changeset	131 def persist_nodemap(tr, revlog, pending=False, force=False):
45942 89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44792 diff changeset	132 """Write nodemap data on disk for a given revlog"""
44308 5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44307 diff changeset	133 if getattr(revlog, 'filteredrevs', ()):
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44307 diff changeset	134 raise error.ProgrammingError(
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44307 diff changeset	135 "cannot persist nodemap of a filtered changelog"
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44307 diff changeset	136 )
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44307 diff changeset	137 if revlog.nodemap_file is None:
46472 98e39f04d60e upgrade: implement partial upgrade for upgrading persistent-nodemap Pulkit Goyal <7895pulkit@gmail.com> parents: 46470 diff changeset	138 if force:
98e39f04d60e upgrade: implement partial upgrade for upgrading persistent-nodemap Pulkit Goyal <7895pulkit@gmail.com> parents: 46470 diff changeset	139 revlog.nodemap_file = get_nodemap_file(
98e39f04d60e upgrade: implement partial upgrade for upgrading persistent-nodemap Pulkit Goyal <7895pulkit@gmail.com> parents: 46470 diff changeset	140 revlog.opener, revlog.indexfile
98e39f04d60e upgrade: implement partial upgrade for upgrading persistent-nodemap Pulkit Goyal <7895pulkit@gmail.com> parents: 46470 diff changeset	141 )
98e39f04d60e upgrade: implement partial upgrade for upgrading persistent-nodemap Pulkit Goyal <7895pulkit@gmail.com> parents: 46470 diff changeset	142 else:
98e39f04d60e upgrade: implement partial upgrade for upgrading persistent-nodemap Pulkit Goyal <7895pulkit@gmail.com> parents: 46470 diff changeset	143 msg = "calling persist nodemap on a revlog without the feature enabled"
98e39f04d60e upgrade: implement partial upgrade for upgrading persistent-nodemap Pulkit Goyal <7895pulkit@gmail.com> parents: 46470 diff changeset	144 raise error.ProgrammingError(msg)
44333 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	145
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	146 can_incremental = util.safehasattr(revlog.index, "nodemap_data_incremental")
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	147 ondisk_docket = revlog._nodemap_docket
44363 f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44340 diff changeset	148 feed_data = util.safehasattr(revlog.index, "update_nodemap_data")
44792 5e3c718692bb nodemap: drop the 'exp-' prefix for internal opener option Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44790 diff changeset	149 use_mmap = revlog.opener.options.get(b"persistent-nodemap.mmap")
44333 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	150
44337 1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44336 diff changeset	151 data = None
44333 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	152 # first attemp an incremental update of the data
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	153 if can_incremental and ondisk_docket is not None:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	154 target_docket = revlog._nodemap_docket.copy()
44337 1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44336 diff changeset	155 (
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44336 diff changeset	156 src_docket,
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44336 diff changeset	157 data_changed_count,
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44336 diff changeset	158 data,
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44336 diff changeset	159 ) = revlog.index.nodemap_data_incremental()
44636 c70bcaf7927b nodemap: automatically "vacuum" the persistent nodemap when too sparse Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44635 diff changeset	160 new_length = target_docket.data_length + len(data)
c70bcaf7927b nodemap: automatically "vacuum" the persistent nodemap when too sparse Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44635 diff changeset	161 new_unused = target_docket.data_unused + data_changed_count
44337 1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44336 diff changeset	162 if src_docket != target_docket:
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44336 diff changeset	163 data = None
44636 c70bcaf7927b nodemap: automatically "vacuum" the persistent nodemap when too sparse Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44635 diff changeset	164 elif new_length <= (new_unused * 10): # under 10% of unused data
c70bcaf7927b nodemap: automatically "vacuum" the persistent nodemap when too sparse Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44635 diff changeset	165 data = None
44337 1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44336 diff changeset	166 else:
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44336 diff changeset	167 datafile = _rawdata_filepath(revlog, target_docket)
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44336 diff changeset	168 # EXP-TODO: if this is a cache, this should use a cache vfs, not a
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44336 diff changeset	169 # store vfs
44634 01b0805534bb nodemap: make sure on disk change get rolled back with the transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44517 diff changeset	170 tr.add(datafile, target_docket.data_length)
44338 2ea6a67ff502 nodemap: write new data from the expected current data length Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44337 diff changeset	171 with revlog.opener(datafile, b'r+') as fd:
2ea6a67ff502 nodemap: write new data from the expected current data length Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44337 diff changeset	172 fd.seek(target_docket.data_length)
44337 1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44336 diff changeset	173 fd.write(data)
44363 f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44340 diff changeset	174 if feed_data:
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44340 diff changeset	175 if use_mmap:
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44340 diff changeset	176 fd.seek(0)
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44340 diff changeset	177 new_data = fd.read(new_length)
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44340 diff changeset	178 else:
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44340 diff changeset	179 fd.flush()
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44340 diff changeset	180 new_data = util.buffer(util.mmapread(fd, new_length))
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44340 diff changeset	181 target_docket.data_length = new_length
44636 c70bcaf7927b nodemap: automatically "vacuum" the persistent nodemap when too sparse Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44635 diff changeset	182 target_docket.data_unused = new_unused
44337 1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44336 diff changeset	183
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44336 diff changeset	184 if data is None:
44333 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	185 # otherwise fallback to a full new export
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	186 target_docket = NodeMapDocket()
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	187 datafile = _rawdata_filepath(revlog, target_docket)
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	188 if util.safehasattr(revlog.index, "nodemap_data_all"):
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	189 data = revlog.index.nodemap_data_all()
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	190 else:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	191 data = persistent_data(revlog.index)
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	192 # EXP-TODO: if this is a cache, this should use a cache vfs, not a
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	193 # store vfs
44635 99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44634 diff changeset	194
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44634 diff changeset	195 tryunlink = revlog.opener.tryunlink
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44634 diff changeset	196
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44634 diff changeset	197 def abortck(tr):
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44634 diff changeset	198 tryunlink(datafile)
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44634 diff changeset	199
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44634 diff changeset	200 callback_id = b"delete-%s" % datafile
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44634 diff changeset	201
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44634 diff changeset	202 # some flavor of the transaction abort does not cleanup new file, it
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44634 diff changeset	203 # simply empty them.
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44634 diff changeset	204 tr.addabort(callback_id, abortck)
44363 f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44340 diff changeset	205 with revlog.opener(datafile, b'w+') as fd:
44333 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	206 fd.write(data)
44363 f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44340 diff changeset	207 if feed_data:
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44340 diff changeset	208 if use_mmap:
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44340 diff changeset	209 new_data = data
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44340 diff changeset	210 else:
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44340 diff changeset	211 fd.flush()
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44340 diff changeset	212 new_data = util.buffer(util.mmapread(fd, len(data)))
44336 8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44335 diff changeset	213 target_docket.data_length = len(data)
44335 e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44333 diff changeset	214 target_docket.tip_rev = revlog.tiprev()
44515 6c906eaedd0d nodemap: track the tip_node for validation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44497 diff changeset	215 target_docket.tip_node = revlog.node(target_docket.tip_rev)
44333 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	216 # EXP-TODO: if this is a cache, this should use a cache vfs, not a
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	217 # store vfs
44516 64e2f603de9d nodemap: make sure hooks have access to an up-to-date version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44515 diff changeset	218 file_path = revlog.nodemap_file
64e2f603de9d nodemap: make sure hooks have access to an up-to-date version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44515 diff changeset	219 if pending:
64e2f603de9d nodemap: make sure hooks have access to an up-to-date version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44515 diff changeset	220 file_path += b'.a'
44634 01b0805534bb nodemap: make sure on disk change get rolled back with the transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44517 diff changeset	221 tr.registertmp(file_path)
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44517 diff changeset	222 else:
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44517 diff changeset	223 tr.addbackup(file_path)
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44517 diff changeset	224
44516 64e2f603de9d nodemap: make sure hooks have access to an up-to-date version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44515 diff changeset	225 with revlog.opener(file_path, b'w', atomictemp=True) as fp:
44333 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	226 fp.write(target_docket.serialize())
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	227 revlog._nodemap_docket = target_docket
44363 f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44340 diff changeset	228 if feed_data:
44340 6ecc34b31137 nodemap: update the index with the newly written data (when appropriate) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44339 diff changeset	229 revlog.index.update_nodemap_data(target_docket, new_data)
6ecc34b31137 nodemap: update the index with the newly written data (when appropriate) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44339 diff changeset	230
44333 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	231 # search for old index file in all cases, some older process might have
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	232 # left one behind.
44322 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44321 diff changeset	233 olds = _other_rawdata_filepath(revlog, target_docket)
44312 563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44311 diff changeset	234 if olds:
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44311 diff changeset	235 realvfs = getattr(revlog, '_realopener', revlog.opener)
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44311 diff changeset	236
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44311 diff changeset	237 def cleanup(tr):
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44311 diff changeset	238 for oldfile in olds:
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44311 diff changeset	239 realvfs.tryunlink(oldfile)
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44311 diff changeset	240
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44311 diff changeset	241 callback_id = b"revlog-cleanup-nodemap-%s" % revlog.nodemap_file
44497 3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44494 diff changeset	242 tr.addpostclose(callback_id, cleanup)
44311 2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	243
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	244
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	245 ### Nodemap docket file
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	246 #
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	247 # The nodemap data are stored on disk using 2 files:
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	248 #
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	249 # * a raw data files containing a persistent nodemap
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	250 # (see `Nodemap Trie` section)
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	251 #
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	252 # * a small "docket" file containing medatadata
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	253 #
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	254 # While the nodemap data can be multiple tens of megabytes, the "docket" is
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	255 # small, it is easy to update it automatically or to duplicated its content
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	256 # during a transaction.
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	257 #
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	258 # Multiple raw data can exist at the same time (The currently valid one and a
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	259 # new one beind used by an in progress transaction). To accomodate this, the
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	260 # filename hosting the raw data has a variable parts. The exact filename is
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	261 # specified inside the "docket" file.
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	262 #
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	263 # The docket file contains information to find, qualify and validate the raw
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	264 # data. Its content is currently very light, but it will expand as the on disk
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	265 # nodemap gains the necessary features to be used in production.
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	266
44790 261e71752d1f nodemap: move on disk file to version 1 Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44789 diff changeset	267 ONDISK_VERSION = 1
44311 2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	268 S_VERSION = struct.Struct(">B")
44515 6c906eaedd0d nodemap: track the tip_node for validation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44497 diff changeset	269 S_HEADER = struct.Struct(">BQQQQ")
44311 2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	270
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	271 ID_SIZE = 8
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	272
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	273
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	274 def _make_uid():
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	275 """return a new unique identifier.
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	276
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	277 The identifier is random and composed of ascii characters."""
46113 59fa3890d40a node: import symbols explicitly Joerg Sonnenberger <joerg@bec.de> parents: 46089 diff changeset	278 return hex(os.urandom(ID_SIZE))
44311 2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	279
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	280
44322 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44321 diff changeset	281 class NodeMapDocket(object):
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44321 diff changeset	282 """metadata associated with persistent nodemap data
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44321 diff changeset	283
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44321 diff changeset	284 The persistent data may come from disk or be on their way to disk.
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44321 diff changeset	285 """
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44321 diff changeset	286
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44321 diff changeset	287 def __init__(self, uid=None):
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44321 diff changeset	288 if uid is None:
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44321 diff changeset	289 uid = _make_uid()
44494 283fd803afa5 nodemap: document the docket attributes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44462 diff changeset	290 # a unique identifier for the data file:
283fd803afa5 nodemap: document the docket attributes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44462 diff changeset	291 # - When new data are appended, it is preserved.
283fd803afa5 nodemap: document the docket attributes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44462 diff changeset	292 # - When a new data file is created, a new identifier is generated.
44322 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44321 diff changeset	293 self.uid = uid
44494 283fd803afa5 nodemap: document the docket attributes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44462 diff changeset	294 # the tipmost revision stored in the data file. This revision and all
283fd803afa5 nodemap: document the docket attributes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44462 diff changeset	295 # revision before it are expected to be encoded in the data file.
44335 e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44333 diff changeset	296 self.tip_rev = None
44515 6c906eaedd0d nodemap: track the tip_node for validation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44497 diff changeset	297 # the node of that tipmost revision, if it mismatch the current index
6c906eaedd0d nodemap: track the tip_node for validation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44497 diff changeset	298 # data the docket is not valid for the current index and should be
6c906eaedd0d nodemap: track the tip_node for validation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44497 diff changeset	299 # discarded.
6c906eaedd0d nodemap: track the tip_node for validation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44497 diff changeset	300 #
6c906eaedd0d nodemap: track the tip_node for validation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44497 diff changeset	301 # note: this method is not perfect as some destructive operation could
6c906eaedd0d nodemap: track the tip_node for validation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44497 diff changeset	302 # preserve the same tip_rev + tip_node while altering lower revision.
6c906eaedd0d nodemap: track the tip_node for validation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44497 diff changeset	303 # However this multiple other caches have the same vulnerability (eg:
6c906eaedd0d nodemap: track the tip_node for validation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44497 diff changeset	304 # brancmap cache).
6c906eaedd0d nodemap: track the tip_node for validation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44497 diff changeset	305 self.tip_node = None
44494 283fd803afa5 nodemap: document the docket attributes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44462 diff changeset	306 # the size (in bytes) of the persisted data to encode the nodemap valid
283fd803afa5 nodemap: document the docket attributes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44462 diff changeset	307 # for `tip_rev`.
283fd803afa5 nodemap: document the docket attributes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44462 diff changeset	308 # - data file shorter than this are corrupted,
283fd803afa5 nodemap: document the docket attributes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44462 diff changeset	309 # - any extra data should be ignored.
44336 8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44335 diff changeset	310 self.data_length = None
44494 283fd803afa5 nodemap: document the docket attributes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44462 diff changeset	311 # the amount (in bytes) of "dead" data, still in the data file but no
283fd803afa5 nodemap: document the docket attributes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44462 diff changeset	312 # longer used for the nodemap.
44336 8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44335 diff changeset	313 self.data_unused = 0
44322 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44321 diff changeset	314
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44321 diff changeset	315 def copy(self):
44335 e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44333 diff changeset	316 new = NodeMapDocket(uid=self.uid)
e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44333 diff changeset	317 new.tip_rev = self.tip_rev
44515 6c906eaedd0d nodemap: track the tip_node for validation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44497 diff changeset	318 new.tip_node = self.tip_node
44336 8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44335 diff changeset	319 new.data_length = self.data_length
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44335 diff changeset	320 new.data_unused = self.data_unused
44335 e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44333 diff changeset	321 return new
44322 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44321 diff changeset	322
44337 1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44336 diff changeset	323 def __cmp__(self, other):
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44336 diff changeset	324 if self.uid < other.uid:
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44336 diff changeset	325 return -1
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44336 diff changeset	326 if self.uid > other.uid:
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44336 diff changeset	327 return 1
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44336 diff changeset	328 elif self.data_length < other.data_length:
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44336 diff changeset	329 return -1
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44336 diff changeset	330 elif self.data_length > other.data_length:
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44336 diff changeset	331 return 1
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44336 diff changeset	332 return 0
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44336 diff changeset	333
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44336 diff changeset	334 def __eq__(self, other):
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44336 diff changeset	335 return self.uid == other.uid and self.data_length == other.data_length
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44336 diff changeset	336
44322 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44321 diff changeset	337 def serialize(self):
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44321 diff changeset	338 """return serialized bytes for a docket using the passed uid"""
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44321 diff changeset	339 data = []
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44321 diff changeset	340 data.append(S_VERSION.pack(ONDISK_VERSION))
44336 8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44335 diff changeset	341 headers = (
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44335 diff changeset	342 len(self.uid),
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44335 diff changeset	343 self.tip_rev,
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44335 diff changeset	344 self.data_length,
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44335 diff changeset	345 self.data_unused,
44515 6c906eaedd0d nodemap: track the tip_node for validation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44497 diff changeset	346 len(self.tip_node),
44336 8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44335 diff changeset	347 )
44335 e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44333 diff changeset	348 data.append(S_HEADER.pack(*headers))
44322 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44321 diff changeset	349 data.append(self.uid)
44515 6c906eaedd0d nodemap: track the tip_node for validation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44497 diff changeset	350 data.append(self.tip_node)
44322 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44321 diff changeset	351 return b''.join(data)
44311 2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	352
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	353
44322 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44321 diff changeset	354 def _rawdata_filepath(revlog, docket):
44311 2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	355 """The (vfs relative) nodemap's rawdata file for a given uid"""
44516 64e2f603de9d nodemap: make sure hooks have access to an up-to-date version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44515 diff changeset	356 if revlog.nodemap_file.endswith(b'.n.a'):
64e2f603de9d nodemap: make sure hooks have access to an up-to-date version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44515 diff changeset	357 prefix = revlog.nodemap_file[:-4]
64e2f603de9d nodemap: make sure hooks have access to an up-to-date version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44515 diff changeset	358 else:
64e2f603de9d nodemap: make sure hooks have access to an up-to-date version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44515 diff changeset	359 prefix = revlog.nodemap_file[:-2]
44322 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44321 diff changeset	360 return b"%s-%s.nd" % (prefix, docket.uid)
44308 5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44307 diff changeset	361
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44307 diff changeset	362
44322 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44321 diff changeset	363 def _other_rawdata_filepath(revlog, docket):
44312 563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44311 diff changeset	364 prefix = revlog.nodemap_file[:-2]
44462 6aee0647e026 nodemap: fix missing r-prefix on regular expression Augie Fackler <augie@google.com> parents: 44445 diff changeset	365 pattern = re.compile(br"(^\|/)%s-[0-9a-f]+\.nd$" % prefix)
44322 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44321 diff changeset	366 new_file_path = _rawdata_filepath(revlog, docket)
44312 563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44311 diff changeset	367 new_file_name = revlog.opener.basename(new_file_path)
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44311 diff changeset	368 dirpath = revlog.opener.dirname(new_file_path)
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44311 diff changeset	369 others = []
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44311 diff changeset	370 for f in revlog.opener.listdir(dirpath):
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44311 diff changeset	371 if pattern.match(f) and f != new_file_name:
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44311 diff changeset	372 others.append(f)
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44311 diff changeset	373 return others
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44311 diff changeset	374
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44311 diff changeset	375
44307 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	376 ### Nodemap Trie
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	377 #
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	378 # This is a simple reference implementation to compute and persist a nodemap
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	379 # trie. This reference implementation is write only. The python version of this
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	380 # is not expected to be actually used, since it wont provide performance
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	381 # improvement over existing non-persistent C implementation.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	382 #
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	383 # The nodemap is persisted as Trie using 4bits-address/16-entries block. each
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	384 # revision can be adressed using its node shortest prefix.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	385 #
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	386 # The trie is stored as a sequence of block. Each block contains 16 entries
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	387 # (signed 64bit integer, big endian). Each entry can be one of the following:
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	388 #
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	389 # * value >= 0 -> index of sub-block
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	390 # * value == -1 -> no value
46199 8e7ce6555ea7 nodemap: match comment to actual code Joerg Sonnenberger <joerg@bec.de> parents: 46113 diff changeset	391 # * value < -1 -> encoded revision: rev = -(value+2)
8e7ce6555ea7 nodemap: match comment to actual code Joerg Sonnenberger <joerg@bec.de> parents: 46113 diff changeset	392 #
8e7ce6555ea7 nodemap: match comment to actual code Joerg Sonnenberger <joerg@bec.de> parents: 46113 diff changeset	393 # See REV_OFFSET and _transform_rev below.
44307 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	394 #
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	395 # The implementation focus on simplicity, not on performance. A Rust
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	396 # implementation should provide a efficient version of the same binary
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	397 # persistence. This reference python implementation is never meant to be
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	398 # extensively use in production.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	399
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	400
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	401 def persistent_data(index):
45942 89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44792 diff changeset	402 """return the persistent binary form for a nodemap for a given index"""
44307 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	403 trie = _build_trie(index)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	404 return _persist_trie(trie)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	405
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	406
44333 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	407 def update_persistent_data(index, root, max_idx, last_rev):
45942 89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44792 diff changeset	408 """return the incremental update for persistent nodemap from a given index"""
44336 8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44335 diff changeset	409 changed_block, trie = _update_trie(index, root, last_rev)
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44335 diff changeset	410 return (
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44335 diff changeset	411 changed_block * S_BLOCK.size,
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44335 diff changeset	412 _persist_trie(trie, existing_idx=max_idx),
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44335 diff changeset	413 )
44333 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	414
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	415
44307 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	416 S_BLOCK = struct.Struct(">" + ("l" * 16))
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	417
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	418 NO_ENTRY = -1
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	419 # rev 0 need to be -2 because 0 is used by block, -1 is a special value.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	420 REV_OFFSET = 2
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	421
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	422
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	423 def _transform_rev(rev):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	424 """Return the number used to represent the rev in the tree.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	425
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	426 (or retrieve a rev number from such representation)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	427
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	428 Note that this is an involution, a function equal to its inverse (i.e.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	429 which gives the identity when applied to itself).
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	430 """
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	431 return -(rev + REV_OFFSET)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	432
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	433
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	434 def _to_int(hex_digit):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	435 """turn an hexadecimal digit into a proper integer"""
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	436 return int(hex_digit, 16)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	437
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	438
44315 7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44314 diff changeset	439 class Block(dict):
7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44314 diff changeset	440 """represent a block of the Trie
7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44314 diff changeset	441
7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44314 diff changeset	442 contains up to 16 entry indexed from 0 to 15"""
7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44314 diff changeset	443
44321 f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44319 diff changeset	444 def __init__(self):
f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44319 diff changeset	445 super(Block, self).__init__()
f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44319 diff changeset	446 # If this block exist on disk, here is its ID
f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44319 diff changeset	447 self.ondisk_id = None
f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44319 diff changeset	448
44316 55b12f2593c1 nodemap: move the iteratio inside the Block object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44315 diff changeset	449 def __iter__(self):
55b12f2593c1 nodemap: move the iteratio inside the Block object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44315 diff changeset	450 return iter(self.get(i) for i in range(16))
55b12f2593c1 nodemap: move the iteratio inside the Block object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44315 diff changeset	451
44315 7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44314 diff changeset	452
44307 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	453 def _build_trie(index):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	454 """build a nodemap trie
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	455
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	456 The nodemap stores revision number for each unique prefix.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	457
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	458 Each block is a dictionary with keys in `[0, 15]`. Values are either
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	459 another block or a revision number.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	460 """
44315 7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44314 diff changeset	461 root = Block()
44307 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	462 for rev in range(len(index)):
46113 59fa3890d40a node: import symbols explicitly Joerg Sonnenberger <joerg@bec.de> parents: 46089 diff changeset	463 current_hex = hex(index[rev][7])
59fa3890d40a node: import symbols explicitly Joerg Sonnenberger <joerg@bec.de> parents: 46089 diff changeset	464 _insert_into_block(index, 0, root, rev, current_hex)
44307 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	465 return root
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	466
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	467
44333 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	468 def _update_trie(index, root, last_rev):
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	469 """consume"""
44336 8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44335 diff changeset	470 changed = 0
44333 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	471 for rev in range(last_rev + 1, len(index)):
46113 59fa3890d40a node: import symbols explicitly Joerg Sonnenberger <joerg@bec.de> parents: 46089 diff changeset	472 current_hex = hex(index[rev][7])
59fa3890d40a node: import symbols explicitly Joerg Sonnenberger <joerg@bec.de> parents: 46089 diff changeset	473 changed += _insert_into_block(index, 0, root, rev, current_hex)
44336 8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44335 diff changeset	474 return changed, root
44333 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	475
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	476
44307 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	477 def _insert_into_block(index, level, block, current_rev, current_hex):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	478 """insert a new revision in a block
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	479
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	480 index: the index we are adding revision for
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	481 level: the depth of the current block in the trie
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	482 block: the block currently being considered
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	483 current_rev: the revision number we are adding
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	484 current_hex: the hexadecimal representation of the of that revision
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	485 """
44336 8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44335 diff changeset	486 changed = 1
44333 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	487 if block.ondisk_id is not None:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	488 block.ondisk_id = None
44307 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	489 hex_digit = _to_int(current_hex[level : level + 1])
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	490 entry = block.get(hex_digit)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	491 if entry is None:
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	492 # no entry, simply store the revision number
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	493 block[hex_digit] = current_rev
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	494 elif isinstance(entry, dict):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	495 # need to recurse to an underlying block
44336 8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44335 diff changeset	496 changed += _insert_into_block(
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44335 diff changeset	497 index, level + 1, entry, current_rev, current_hex
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44335 diff changeset	498 )
44307 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	499 else:
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	500 # collision with a previously unique prefix, inserting new
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	501 # vertices to fit both entry.
46113 59fa3890d40a node: import symbols explicitly Joerg Sonnenberger <joerg@bec.de> parents: 46089 diff changeset	502 other_hex = hex(index[entry][7])
44307 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	503 other_rev = entry
44315 7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44314 diff changeset	504 new = Block()
44307 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	505 block[hex_digit] = new
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	506 _insert_into_block(index, level + 1, new, other_rev, other_hex)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	507 _insert_into_block(index, level + 1, new, current_rev, current_hex)
44336 8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44335 diff changeset	508 return changed
44307 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	509
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	510
44333 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	511 def _persist_trie(root, existing_idx=None):
44307 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	512 """turn a nodemap trie into persistent binary data
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	513
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	514 See `_build_trie` for nodemap trie structure"""
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	515 block_map = {}
44333 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	516 if existing_idx is not None:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	517 base_idx = existing_idx + 1
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	518 else:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	519 base_idx = 0
44307 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	520 chunks = []
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	521 for tn in _walk_trie(root):
44333 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	522 if tn.ondisk_id is not None:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	523 block_map[id(tn)] = tn.ondisk_id
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	524 else:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	525 block_map[id(tn)] = len(chunks) + base_idx
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	526 chunks.append(_persist_block(tn, block_map))
44307 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	527 return b''.join(chunks)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	528
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	529
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	530 def _walk_trie(block):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	531 """yield all the block in a trie
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	532
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	533 Children blocks are always yield before their parent block.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	534 """
44696 ed475420e0af nodemap: rename a variable to prevent shawoding "_" Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44636 diff changeset	535 for (__, item) in sorted(block.items()):
44307 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	536 if isinstance(item, dict):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	537 for sub_block in _walk_trie(item):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	538 yield sub_block
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	539 yield block
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	540
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	541
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	542 def _persist_block(block_node, block_map):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	543 """produce persistent binary data for a single block
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	544
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	545 Children block are assumed to be already persisted and present in
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	546 block_map.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	547 """
44316 55b12f2593c1 nodemap: move the iteratio inside the Block object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44315 diff changeset	548 data = tuple(_to_value(v, block_map) for v in block_node)
44307 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	549 return S_BLOCK.pack(*data)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	550
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	551
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	552 def _to_value(item, block_map):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	553 """persist any value as an integer"""
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	554 if item is None:
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	555 return NO_ENTRY
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	556 elif isinstance(item, dict):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	557 return block_map[id(item)]
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	558 else:
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	559 return _transform_rev(item)
44317 78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44316 diff changeset	560
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44316 diff changeset	561
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44316 diff changeset	562 def parse_data(data):
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44316 diff changeset	563 """parse parse nodemap data into a nodemap Trie"""
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44316 diff changeset	564 if (len(data) % S_BLOCK.size) != 0:
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44316 diff changeset	565 msg = "nodemap data size is not a multiple of block size (%d): %d"
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44316 diff changeset	566 raise error.Abort(msg % (S_BLOCK.size, len(data)))
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44316 diff changeset	567 if not data:
44333 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	568 return Block(), None
44317 78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44316 diff changeset	569 block_map = {}
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44316 diff changeset	570 new_blocks = []
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44316 diff changeset	571 for i in range(0, len(data), S_BLOCK.size):
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44316 diff changeset	572 block = Block()
44321 f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44319 diff changeset	573 block.ondisk_id = len(block_map)
f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44319 diff changeset	574 block_map[block.ondisk_id] = block
44317 78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44316 diff changeset	575 block_data = data[i : i + S_BLOCK.size]
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44316 diff changeset	576 values = S_BLOCK.unpack(block_data)
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44316 diff changeset	577 new_blocks.append((block, values))
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44316 diff changeset	578 for b, values in new_blocks:
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44316 diff changeset	579 for idx, v in enumerate(values):
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44316 diff changeset	580 if v == NO_ENTRY:
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44316 diff changeset	581 continue
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44316 diff changeset	582 elif v >= 0:
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44316 diff changeset	583 b[idx] = block_map[v]
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44316 diff changeset	584 else:
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44316 diff changeset	585 b[idx] = _transform_rev(v)
44333 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	586 return block, i // S_BLOCK.size
44318 20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44317 diff changeset	587
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44317 diff changeset	588
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44317 diff changeset	589 # debug utility
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44317 diff changeset	590
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44317 diff changeset	591
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44317 diff changeset	592 def check_data(ui, index, data):
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44317 diff changeset	593 """verify that the provided nodemap data are valid for the given idex"""
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44317 diff changeset	594 ret = 0
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44317 diff changeset	595 ui.status((b"revision in index: %d\n") % len(index))
44333 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	596 root, __ = parse_data(data)
44318 20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44317 diff changeset	597 all_revs = set(_all_revisions(root))
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44317 diff changeset	598 ui.status((b"revision in nodemap: %d\n") % len(all_revs))
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44317 diff changeset	599 for r in range(len(index)):
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44317 diff changeset	600 if r not in all_revs:
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44317 diff changeset	601 msg = b" revision missing from nodemap: %d\n" % r
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44317 diff changeset	602 ui.write_err(msg)
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44317 diff changeset	603 ret = 1
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44317 diff changeset	604 else:
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44317 diff changeset	605 all_revs.remove(r)
46113 59fa3890d40a node: import symbols explicitly Joerg Sonnenberger <joerg@bec.de> parents: 46089 diff changeset	606 nm_rev = _find_node(root, hex(index[r][7]))
44319 d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44318 diff changeset	607 if nm_rev is None:
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44318 diff changeset	608 msg = b" revision node does not match any entries: %d\n" % r
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44318 diff changeset	609 ui.write_err(msg)
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44318 diff changeset	610 ret = 1
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44318 diff changeset	611 elif nm_rev != r:
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44318 diff changeset	612 msg = (
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44318 diff changeset	613 b" revision node does not match the expected revision: "
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44318 diff changeset	614 b"%d != %d\n" % (r, nm_rev)
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44318 diff changeset	615 )
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44318 diff changeset	616 ui.write_err(msg)
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44318 diff changeset	617 ret = 1
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44318 diff changeset	618
44318 20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44317 diff changeset	619 if all_revs:
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44317 diff changeset	620 for r in sorted(all_revs):
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44317 diff changeset	621 msg = b" extra revision in nodemap: %d\n" % r
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44317 diff changeset	622 ui.write_err(msg)
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44317 diff changeset	623 ret = 1
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44317 diff changeset	624 return ret
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44317 diff changeset	625
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44317 diff changeset	626
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44317 diff changeset	627 def _all_revisions(root):
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44317 diff changeset	628 """return all revisions stored in a Trie"""
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44317 diff changeset	629 for block in _walk_trie(root):
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44317 diff changeset	630 for v in block:
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44317 diff changeset	631 if v is None or isinstance(v, Block):
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44317 diff changeset	632 continue
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44317 diff changeset	633 yield v
44319 d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44318 diff changeset	634
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44318 diff changeset	635
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44318 diff changeset	636 def _find_node(block, node):
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44318 diff changeset	637 """find the revision associated with a given node"""
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44318 diff changeset	638 entry = block.get(_to_int(node[0:1]))
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44318 diff changeset	639 if isinstance(entry, dict):
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44318 diff changeset	640 return _find_node(entry, node[1:])
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44318 diff changeset	641 return entry
46469 835aafb2543f revlog: refactor logic to compute nodemap file in separate function Pulkit Goyal <7895pulkit@gmail.com> parents: 46468 diff changeset	642
835aafb2543f revlog: refactor logic to compute nodemap file in separate function Pulkit Goyal <7895pulkit@gmail.com> parents: 46468 diff changeset	643
835aafb2543f revlog: refactor logic to compute nodemap file in separate function Pulkit Goyal <7895pulkit@gmail.com> parents: 46468 diff changeset	644 def get_nodemap_file(opener, indexfile):
835aafb2543f revlog: refactor logic to compute nodemap file in separate function Pulkit Goyal <7895pulkit@gmail.com> parents: 46468 diff changeset	645 if indexfile.endswith(b'.a'):
835aafb2543f revlog: refactor logic to compute nodemap file in separate function Pulkit Goyal <7895pulkit@gmail.com> parents: 46468 diff changeset	646 pending_path = indexfile[:-4] + b".n.a"
835aafb2543f revlog: refactor logic to compute nodemap file in separate function Pulkit Goyal <7895pulkit@gmail.com> parents: 46468 diff changeset	647 if opener.exists(pending_path):
835aafb2543f revlog: refactor logic to compute nodemap file in separate function Pulkit Goyal <7895pulkit@gmail.com> parents: 46468 diff changeset	648 return pending_path
835aafb2543f revlog: refactor logic to compute nodemap file in separate function Pulkit Goyal <7895pulkit@gmail.com> parents: 46468 diff changeset	649 else:
835aafb2543f revlog: refactor logic to compute nodemap file in separate function Pulkit Goyal <7895pulkit@gmail.com> parents: 46468 diff changeset	650 return indexfile[:-4] + b".n"
835aafb2543f revlog: refactor logic to compute nodemap file in separate function Pulkit Goyal <7895pulkit@gmail.com> parents: 46468 diff changeset	651 else:
835aafb2543f revlog: refactor logic to compute nodemap file in separate function Pulkit Goyal <7895pulkit@gmail.com> parents: 46468 diff changeset	652 return indexfile[:-2] + b".n"

Mercurial > public > mercurial-scm > hg

annotate mercurial/revlogutils/nodemap.py @ 46472:98e39f04d60e