mercurial-scm/hg-stable: mercurial/revlogutils/nodemap.py annotate

annotate mercurial/revlogutils/nodemap.py @ 46971:a3720569a43f

nodemap: deal with data mmap error If the file is too small, the mmapread call would raise a ValueError. We catch that and ignore nodemap content (as we do without mmap). This make the repository slightly slower (until the next write) but usable. Unlike the current crash. Differential Revision: https://phab.mercurial-scm.org/D10458

author	Pierre-Yves David <pierre-yves.david@octobus.net>
date	Fri, 16 Apr 2021 15:39:00 +0200
parents	e2f7b2695ba1
children	f38bf44e077f

rev	line source
44035 ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	1 # nodemap.py - nodemap related code and utilities
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	2 #
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	3 # Copyright 2019 Pierre-Yves David <pierre-yves.david@octobus.net>
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	4 # Copyright 2019 George Racinet <georges.racinet@octobus.net>
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	5 #
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	6 # This software may be used and distributed according to the terms of the
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	7 # GNU General Public License version 2 or any later version.
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	8
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	9 from __future__ import absolute_import
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	10
44397 f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44374 diff changeset	11 import errno
44354 2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	12 import os
44355 563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	13 import re
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	14 import struct
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	15
46114 59fa3890d40a node: import symbols explicitly Joerg Sonnenberger <joerg@bec.de> parents: 46090 diff changeset	16 from ..node import hex
44865 6493f0a567c2 nodemap: add a new mode option, with an optional "warn" value Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44740 diff changeset	17
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	18 from .. import (
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	19 error,
44357 7f4f7ef3133e nodemap: add a optional `nodemap_add_full` method on indexes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44355 diff changeset	20 util,
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	21 )
44035 ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	22
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	23
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	24 class NodeMap(dict):
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	25 def __missing__(self, x):
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	26 raise error.RevlogError(b'unknown node: %s' % x)
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	27
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	28
44352 6c07480d6659 nodemap: add a function to read the data from disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44351 diff changeset	29 def persisted_data(revlog):
6c07480d6659 nodemap: add a function to read the data from disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44351 diff changeset	30 """read the nodemap for a revlog from disk"""
6c07480d6659 nodemap: add a function to read the data from disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44351 diff changeset	31 if revlog.nodemap_file is None:
6c07480d6659 nodemap: add a function to read the data from disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44351 diff changeset	32 return None
44354 2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	33 pdata = revlog.opener.tryread(revlog.nodemap_file)
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	34 if not pdata:
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	35 return None
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	36 offset = 0
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	37 (version,) = S_VERSION.unpack(pdata[offset : offset + S_VERSION.size])
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	38 if version != ONDISK_VERSION:
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	39 return None
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	40 offset += S_VERSION.size
44369 e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44367 diff changeset	41 headers = S_HEADER.unpack(pdata[offset : offset + S_HEADER.size])
44525 6c906eaedd0d nodemap: track the tip_node for validation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44509 diff changeset	42 uid_size, tip_rev, data_length, data_unused, tip_node_size = headers
44354 2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	43 offset += S_HEADER.size
44365 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	44 docket = NodeMapDocket(pdata[offset : offset + uid_size])
44525 6c906eaedd0d nodemap: track the tip_node for validation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44509 diff changeset	45 offset += uid_size
44369 e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44367 diff changeset	46 docket.tip_rev = tip_rev
44525 6c906eaedd0d nodemap: track the tip_node for validation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44509 diff changeset	47 docket.tip_node = pdata[offset : offset + tip_node_size]
44370 8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	48 docket.data_length = data_length
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	49 docket.data_unused = data_unused
44354 2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	50
44365 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	51 filename = _rawdata_filepath(revlog, docket)
44869 5e3c718692bb nodemap: drop the 'exp-' prefix for internal opener option Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44867 diff changeset	52 use_mmap = revlog.opener.options.get(b"persistent-nodemap.mmap")
44397 f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44374 diff changeset	53 try:
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44374 diff changeset	54 with revlog.opener(filename) as fd:
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44374 diff changeset	55 if use_mmap:
46971 a3720569a43f nodemap: deal with data mmap error Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46796 diff changeset	56 try:
a3720569a43f nodemap: deal with data mmap error Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46796 diff changeset	57 data = util.buffer(util.mmapread(fd, data_length))
a3720569a43f nodemap: deal with data mmap error Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46796 diff changeset	58 except ValueError:
a3720569a43f nodemap: deal with data mmap error Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46796 diff changeset	59 # raised when the read file is too small
a3720569a43f nodemap: deal with data mmap error Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46796 diff changeset	60 data = b''
44397 f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44374 diff changeset	61 else:
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44374 diff changeset	62 data = fd.read(data_length)
46321 d32e7ed81f4f persistent-nodemap: catch the right exception on python Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46312 diff changeset	63 except (IOError, OSError) as e:
46090 8ff2d8359d0f persistent-nodemap: properly ignore non-existent `.nd` data file Simon Sapin <simon-commits@exyr.org> parents: 45957 diff changeset	64 if e.errno == errno.ENOENT:
8ff2d8359d0f persistent-nodemap: properly ignore non-existent `.nd` data file Simon Sapin <simon-commits@exyr.org> parents: 45957 diff changeset	65 return None
8ff2d8359d0f persistent-nodemap: properly ignore non-existent `.nd` data file Simon Sapin <simon-commits@exyr.org> parents: 45957 diff changeset	66 else:
44397 f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44374 diff changeset	67 raise
44373 c7eebdb15139 nodemap: never read more than the expected data amount Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44372 diff changeset	68 if len(data) < data_length:
c7eebdb15139 nodemap: never read more than the expected data amount Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44372 diff changeset	69 return None
c7eebdb15139 nodemap: never read more than the expected data amount Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44372 diff changeset	70 return docket, data
44352 6c07480d6659 nodemap: add a function to read the data from disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44351 diff changeset	71
6c07480d6659 nodemap: add a function to read the data from disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44351 diff changeset	72
44351 5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	73 def setup_persistent_nodemap(tr, revlog):
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	74 """Install whatever is needed transaction side to persist a nodemap on disk
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	75
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	76 (only actually persist the nodemap if this is relevant for this revlog)
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	77 """
44353 daad3aace942 nodemap: only use persistent nodemap for non-inlined revlog Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44352 diff changeset	78 if revlog._inline:
daad3aace942 nodemap: only use persistent nodemap for non-inlined revlog Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44352 diff changeset	79 return # inlined revlog are too small for this to be relevant
44351 5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	80 if revlog.nodemap_file is None:
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	81 return # we do not use persistent_nodemap on this revlog
44527 448d700e0d27 nodemap: make sure the nodemap docket is updated after the changelog Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44526 diff changeset	82
448d700e0d27 nodemap: make sure the nodemap docket is updated after the changelog Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44526 diff changeset	83 # we need to happen after the changelog finalization, in that use "cl-"
448d700e0d27 nodemap: make sure the nodemap docket is updated after the changelog Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44526 diff changeset	84 callback_id = b"nm-revlog-persistent-nodemap-%s" % revlog.nodemap_file
44351 5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	85 if tr.hasfinalize(callback_id):
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	86 return # no need to register again
44526 64e2f603de9d nodemap: make sure hooks have access to an up-to-date version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44525 diff changeset	87 tr.addpending(
46529 83f037acf71a nodemap: make `_persist_nodemap` a public function Pulkit Goyal <7895pulkit@gmail.com> parents: 46321 diff changeset	88 callback_id, lambda tr: persist_nodemap(tr, revlog, pending=True)
44526 64e2f603de9d nodemap: make sure hooks have access to an up-to-date version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44525 diff changeset	89 )
46529 83f037acf71a nodemap: make `_persist_nodemap` a public function Pulkit Goyal <7895pulkit@gmail.com> parents: 46321 diff changeset	90 tr.addfinalize(callback_id, lambda tr: persist_nodemap(tr, revlog))
44509 3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44506 diff changeset	91
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44506 diff changeset	92
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44506 diff changeset	93 class _NoTransaction(object):
45957 89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44869 diff changeset	94 """transaction like object to update the nodemap outside a transaction"""
44509 3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44506 diff changeset	95
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44506 diff changeset	96 def __init__(self):
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44506 diff changeset	97 self._postclose = {}
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44506 diff changeset	98
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44506 diff changeset	99 def addpostclose(self, callback_id, callback_func):
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44506 diff changeset	100 self._postclose[callback_id] = callback_func
44351 5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	101
44638 01b0805534bb nodemap: make sure on disk change get rolled back with the transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44527 diff changeset	102 def registertmp(self, args, *kwargs):
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44527 diff changeset	103 pass
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44527 diff changeset	104
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44527 diff changeset	105 def addbackup(self, args, *kwargs):
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44527 diff changeset	106 pass
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44527 diff changeset	107
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44527 diff changeset	108 def add(self, args, *kwargs):
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44527 diff changeset	109 pass
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44527 diff changeset	110
44639 99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44638 diff changeset	111 def addabort(self, args, *kwargs):
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44638 diff changeset	112 pass
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44638 diff changeset	113
44865 6493f0a567c2 nodemap: add a new mode option, with an optional "warn" value Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44740 diff changeset	114 def _report(self, *args):
6493f0a567c2 nodemap: add a new mode option, with an optional "warn" value Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44740 diff changeset	115 pass
6493f0a567c2 nodemap: add a new mode option, with an optional "warn" value Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44740 diff changeset	116
44351 5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	117
44465 336ec75ed1ac nodemap: warm the persistent nodemap on disk with debugupdatecache Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44397 diff changeset	118 def update_persistent_nodemap(revlog):
336ec75ed1ac nodemap: warm the persistent nodemap on disk with debugupdatecache Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44397 diff changeset	119 """update the persistent nodemap right now
336ec75ed1ac nodemap: warm the persistent nodemap on disk with debugupdatecache Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44397 diff changeset	120
336ec75ed1ac nodemap: warm the persistent nodemap on disk with debugupdatecache Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44397 diff changeset	121 To be used for updating the nodemap on disk outside of a normal transaction
336ec75ed1ac nodemap: warm the persistent nodemap on disk with debugupdatecache Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44397 diff changeset	122 setup (eg, `debugupdatecache`).
336ec75ed1ac nodemap: warm the persistent nodemap on disk with debugupdatecache Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44397 diff changeset	123 """
44740 b4537125eb3c nodemap: skip persistent nodemap warming for revlog not using it Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44697 diff changeset	124 if revlog._inline:
b4537125eb3c nodemap: skip persistent nodemap warming for revlog not using it Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44697 diff changeset	125 return # inlined revlog are too small for this to be relevant
b4537125eb3c nodemap: skip persistent nodemap warming for revlog not using it Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44697 diff changeset	126 if revlog.nodemap_file is None:
b4537125eb3c nodemap: skip persistent nodemap warming for revlog not using it Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44697 diff changeset	127 return # we do not use persistent_nodemap on this revlog
b4537125eb3c nodemap: skip persistent nodemap warming for revlog not using it Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44697 diff changeset	128
44509 3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44506 diff changeset	129 notr = _NoTransaction()
46529 83f037acf71a nodemap: make `_persist_nodemap` a public function Pulkit Goyal <7895pulkit@gmail.com> parents: 46321 diff changeset	130 persist_nodemap(notr, revlog)
44509 3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44506 diff changeset	131 for k in sorted(notr._postclose):
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44506 diff changeset	132 notr._postclose[k](None)
44465 336ec75ed1ac nodemap: warm the persistent nodemap on disk with debugupdatecache Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44397 diff changeset	133
336ec75ed1ac nodemap: warm the persistent nodemap on disk with debugupdatecache Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44397 diff changeset	134
46576 67b5fafd3a46 upgrade: speed up when we have only nodemap to downgrade Pulkit Goyal <7895pulkit@gmail.com> parents: 46533 diff changeset	135 def delete_nodemap(tr, repo, revlog):
67b5fafd3a46 upgrade: speed up when we have only nodemap to downgrade Pulkit Goyal <7895pulkit@gmail.com> parents: 46533 diff changeset	136 """ Delete nodemap data on disk for a given revlog"""
67b5fafd3a46 upgrade: speed up when we have only nodemap to downgrade Pulkit Goyal <7895pulkit@gmail.com> parents: 46533 diff changeset	137 if revlog.nodemap_file is None:
67b5fafd3a46 upgrade: speed up when we have only nodemap to downgrade Pulkit Goyal <7895pulkit@gmail.com> parents: 46533 diff changeset	138 msg = "calling persist nodemap on a revlog without the feature enabled"
67b5fafd3a46 upgrade: speed up when we have only nodemap to downgrade Pulkit Goyal <7895pulkit@gmail.com> parents: 46533 diff changeset	139 raise error.ProgrammingError(msg)
67b5fafd3a46 upgrade: speed up when we have only nodemap to downgrade Pulkit Goyal <7895pulkit@gmail.com> parents: 46533 diff changeset	140 repo.svfs.unlink(revlog.nodemap_file)
67b5fafd3a46 upgrade: speed up when we have only nodemap to downgrade Pulkit Goyal <7895pulkit@gmail.com> parents: 46533 diff changeset	141
67b5fafd3a46 upgrade: speed up when we have only nodemap to downgrade Pulkit Goyal <7895pulkit@gmail.com> parents: 46533 diff changeset	142
46533 98e39f04d60e upgrade: implement partial upgrade for upgrading persistent-nodemap Pulkit Goyal <7895pulkit@gmail.com> parents: 46531 diff changeset	143 def persist_nodemap(tr, revlog, pending=False, force=False):
45957 89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44869 diff changeset	144 """Write nodemap data on disk for a given revlog"""
44351 5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	145 if getattr(revlog, 'filteredrevs', ()):
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	146 raise error.ProgrammingError(
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	147 "cannot persist nodemap of a filtered changelog"
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	148 )
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	149 if revlog.nodemap_file is None:
46533 98e39f04d60e upgrade: implement partial upgrade for upgrading persistent-nodemap Pulkit Goyal <7895pulkit@gmail.com> parents: 46531 diff changeset	150 if force:
98e39f04d60e upgrade: implement partial upgrade for upgrading persistent-nodemap Pulkit Goyal <7895pulkit@gmail.com> parents: 46531 diff changeset	151 revlog.nodemap_file = get_nodemap_file(
98e39f04d60e upgrade: implement partial upgrade for upgrading persistent-nodemap Pulkit Goyal <7895pulkit@gmail.com> parents: 46531 diff changeset	152 revlog.opener, revlog.indexfile
98e39f04d60e upgrade: implement partial upgrade for upgrading persistent-nodemap Pulkit Goyal <7895pulkit@gmail.com> parents: 46531 diff changeset	153 )
98e39f04d60e upgrade: implement partial upgrade for upgrading persistent-nodemap Pulkit Goyal <7895pulkit@gmail.com> parents: 46531 diff changeset	154 else:
98e39f04d60e upgrade: implement partial upgrade for upgrading persistent-nodemap Pulkit Goyal <7895pulkit@gmail.com> parents: 46531 diff changeset	155 msg = "calling persist nodemap on a revlog without the feature enabled"
98e39f04d60e upgrade: implement partial upgrade for upgrading persistent-nodemap Pulkit Goyal <7895pulkit@gmail.com> parents: 46531 diff changeset	156 raise error.ProgrammingError(msg)
44367 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	157
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	158 can_incremental = util.safehasattr(revlog.index, "nodemap_data_incremental")
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	159 ondisk_docket = revlog._nodemap_docket
44397 f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44374 diff changeset	160 feed_data = util.safehasattr(revlog.index, "update_nodemap_data")
44869 5e3c718692bb nodemap: drop the 'exp-' prefix for internal opener option Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44867 diff changeset	161 use_mmap = revlog.opener.options.get(b"persistent-nodemap.mmap")
44367 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	162
44371 1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	163 data = None
44367 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	164 # first attemp an incremental update of the data
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	165 if can_incremental and ondisk_docket is not None:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	166 target_docket = revlog._nodemap_docket.copy()
44371 1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	167 (
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	168 src_docket,
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	169 data_changed_count,
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	170 data,
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	171 ) = revlog.index.nodemap_data_incremental()
44640 c70bcaf7927b nodemap: automatically "vacuum" the persistent nodemap when too sparse Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44639 diff changeset	172 new_length = target_docket.data_length + len(data)
c70bcaf7927b nodemap: automatically "vacuum" the persistent nodemap when too sparse Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44639 diff changeset	173 new_unused = target_docket.data_unused + data_changed_count
44371 1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	174 if src_docket != target_docket:
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	175 data = None
44640 c70bcaf7927b nodemap: automatically "vacuum" the persistent nodemap when too sparse Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44639 diff changeset	176 elif new_length <= (new_unused * 10): # under 10% of unused data
c70bcaf7927b nodemap: automatically "vacuum" the persistent nodemap when too sparse Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44639 diff changeset	177 data = None
44371 1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	178 else:
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	179 datafile = _rawdata_filepath(revlog, target_docket)
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	180 # EXP-TODO: if this is a cache, this should use a cache vfs, not a
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	181 # store vfs
44638 01b0805534bb nodemap: make sure on disk change get rolled back with the transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44527 diff changeset	182 tr.add(datafile, target_docket.data_length)
44372 2ea6a67ff502 nodemap: write new data from the expected current data length Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44371 diff changeset	183 with revlog.opener(datafile, b'r+') as fd:
2ea6a67ff502 nodemap: write new data from the expected current data length Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44371 diff changeset	184 fd.seek(target_docket.data_length)
44371 1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	185 fd.write(data)
44397 f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44374 diff changeset	186 if feed_data:
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44374 diff changeset	187 if use_mmap:
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44374 diff changeset	188 fd.seek(0)
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44374 diff changeset	189 new_data = fd.read(new_length)
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44374 diff changeset	190 else:
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44374 diff changeset	191 fd.flush()
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44374 diff changeset	192 new_data = util.buffer(util.mmapread(fd, new_length))
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44374 diff changeset	193 target_docket.data_length = new_length
44640 c70bcaf7927b nodemap: automatically "vacuum" the persistent nodemap when too sparse Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44639 diff changeset	194 target_docket.data_unused = new_unused
44371 1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	195
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	196 if data is None:
44367 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	197 # otherwise fallback to a full new export
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	198 target_docket = NodeMapDocket()
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	199 datafile = _rawdata_filepath(revlog, target_docket)
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	200 if util.safehasattr(revlog.index, "nodemap_data_all"):
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	201 data = revlog.index.nodemap_data_all()
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	202 else:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	203 data = persistent_data(revlog.index)
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	204 # EXP-TODO: if this is a cache, this should use a cache vfs, not a
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	205 # store vfs
44639 99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44638 diff changeset	206
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44638 diff changeset	207 tryunlink = revlog.opener.tryunlink
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44638 diff changeset	208
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44638 diff changeset	209 def abortck(tr):
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44638 diff changeset	210 tryunlink(datafile)
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44638 diff changeset	211
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44638 diff changeset	212 callback_id = b"delete-%s" % datafile
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44638 diff changeset	213
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44638 diff changeset	214 # some flavor of the transaction abort does not cleanup new file, it
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44638 diff changeset	215 # simply empty them.
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44638 diff changeset	216 tr.addabort(callback_id, abortck)
44397 f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44374 diff changeset	217 with revlog.opener(datafile, b'w+') as fd:
44367 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	218 fd.write(data)
44397 f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44374 diff changeset	219 if feed_data:
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44374 diff changeset	220 if use_mmap:
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44374 diff changeset	221 new_data = data
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44374 diff changeset	222 else:
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44374 diff changeset	223 fd.flush()
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44374 diff changeset	224 new_data = util.buffer(util.mmapread(fd, len(data)))
44370 8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	225 target_docket.data_length = len(data)
44369 e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44367 diff changeset	226 target_docket.tip_rev = revlog.tiprev()
44525 6c906eaedd0d nodemap: track the tip_node for validation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44509 diff changeset	227 target_docket.tip_node = revlog.node(target_docket.tip_rev)
44367 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	228 # EXP-TODO: if this is a cache, this should use a cache vfs, not a
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	229 # store vfs
44526 64e2f603de9d nodemap: make sure hooks have access to an up-to-date version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44525 diff changeset	230 file_path = revlog.nodemap_file
64e2f603de9d nodemap: make sure hooks have access to an up-to-date version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44525 diff changeset	231 if pending:
64e2f603de9d nodemap: make sure hooks have access to an up-to-date version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44525 diff changeset	232 file_path += b'.a'
44638 01b0805534bb nodemap: make sure on disk change get rolled back with the transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44527 diff changeset	233 tr.registertmp(file_path)
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44527 diff changeset	234 else:
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44527 diff changeset	235 tr.addbackup(file_path)
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44527 diff changeset	236
44526 64e2f603de9d nodemap: make sure hooks have access to an up-to-date version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44525 diff changeset	237 with revlog.opener(file_path, b'w', atomictemp=True) as fp:
44367 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	238 fp.write(target_docket.serialize())
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	239 revlog._nodemap_docket = target_docket
44397 f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44374 diff changeset	240 if feed_data:
44374 6ecc34b31137 nodemap: update the index with the newly written data (when appropriate) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44373 diff changeset	241 revlog.index.update_nodemap_data(target_docket, new_data)
6ecc34b31137 nodemap: update the index with the newly written data (when appropriate) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44373 diff changeset	242
44367 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	243 # search for old index file in all cases, some older process might have
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	244 # left one behind.
44365 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	245 olds = _other_rawdata_filepath(revlog, target_docket)
44355 563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	246 if olds:
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	247 realvfs = getattr(revlog, '_realopener', revlog.opener)
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	248
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	249 def cleanup(tr):
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	250 for oldfile in olds:
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	251 realvfs.tryunlink(oldfile)
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	252
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	253 callback_id = b"revlog-cleanup-nodemap-%s" % revlog.nodemap_file
44509 3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44506 diff changeset	254 tr.addpostclose(callback_id, cleanup)
44354 2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	255
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	256
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	257 ### Nodemap docket file
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	258 #
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	259 # The nodemap data are stored on disk using 2 files:
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	260 #
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	261 # * a raw data files containing a persistent nodemap
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	262 # (see `Nodemap Trie` section)
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	263 #
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	264 # * a small "docket" file containing medatadata
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	265 #
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	266 # While the nodemap data can be multiple tens of megabytes, the "docket" is
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	267 # small, it is easy to update it automatically or to duplicated its content
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	268 # during a transaction.
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	269 #
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	270 # Multiple raw data can exist at the same time (The currently valid one and a
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	271 # new one beind used by an in progress transaction). To accomodate this, the
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	272 # filename hosting the raw data has a variable parts. The exact filename is
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	273 # specified inside the "docket" file.
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	274 #
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	275 # The docket file contains information to find, qualify and validate the raw
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	276 # data. Its content is currently very light, but it will expand as the on disk
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	277 # nodemap gains the necessary features to be used in production.
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	278
44867 261e71752d1f nodemap: move on disk file to version 1 Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44866 diff changeset	279 ONDISK_VERSION = 1
44354 2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	280 S_VERSION = struct.Struct(">B")
44525 6c906eaedd0d nodemap: track the tip_node for validation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44509 diff changeset	281 S_HEADER = struct.Struct(">BQQQQ")
44354 2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	282
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	283 ID_SIZE = 8
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	284
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	285
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	286 def _make_uid():
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	287 """return a new unique identifier.
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	288
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	289 The identifier is random and composed of ascii characters."""
46114 59fa3890d40a node: import symbols explicitly Joerg Sonnenberger <joerg@bec.de> parents: 46090 diff changeset	290 return hex(os.urandom(ID_SIZE))
44354 2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	291
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	292
44365 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	293 class NodeMapDocket(object):
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	294 """metadata associated with persistent nodemap data
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	295
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	296 The persistent data may come from disk or be on their way to disk.
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	297 """
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	298
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	299 def __init__(self, uid=None):
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	300 if uid is None:
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	301 uid = _make_uid()
44506 283fd803afa5 nodemap: document the docket attributes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44480 diff changeset	302 # a unique identifier for the data file:
283fd803afa5 nodemap: document the docket attributes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44480 diff changeset	303 # - When new data are appended, it is preserved.
283fd803afa5 nodemap: document the docket attributes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44480 diff changeset	304 # - When a new data file is created, a new identifier is generated.
44365 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	305 self.uid = uid
44506 283fd803afa5 nodemap: document the docket attributes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44480 diff changeset	306 # the tipmost revision stored in the data file. This revision and all
283fd803afa5 nodemap: document the docket attributes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44480 diff changeset	307 # revision before it are expected to be encoded in the data file.
44369 e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44367 diff changeset	308 self.tip_rev = None
44525 6c906eaedd0d nodemap: track the tip_node for validation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44509 diff changeset	309 # the node of that tipmost revision, if it mismatch the current index
6c906eaedd0d nodemap: track the tip_node for validation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44509 diff changeset	310 # data the docket is not valid for the current index and should be
6c906eaedd0d nodemap: track the tip_node for validation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44509 diff changeset	311 # discarded.
6c906eaedd0d nodemap: track the tip_node for validation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44509 diff changeset	312 #
6c906eaedd0d nodemap: track the tip_node for validation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44509 diff changeset	313 # note: this method is not perfect as some destructive operation could
6c906eaedd0d nodemap: track the tip_node for validation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44509 diff changeset	314 # preserve the same tip_rev + tip_node while altering lower revision.
6c906eaedd0d nodemap: track the tip_node for validation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44509 diff changeset	315 # However this multiple other caches have the same vulnerability (eg:
6c906eaedd0d nodemap: track the tip_node for validation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44509 diff changeset	316 # brancmap cache).
6c906eaedd0d nodemap: track the tip_node for validation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44509 diff changeset	317 self.tip_node = None
44506 283fd803afa5 nodemap: document the docket attributes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44480 diff changeset	318 # the size (in bytes) of the persisted data to encode the nodemap valid
283fd803afa5 nodemap: document the docket attributes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44480 diff changeset	319 # for `tip_rev`.
283fd803afa5 nodemap: document the docket attributes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44480 diff changeset	320 # - data file shorter than this are corrupted,
283fd803afa5 nodemap: document the docket attributes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44480 diff changeset	321 # - any extra data should be ignored.
44370 8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	322 self.data_length = None
44506 283fd803afa5 nodemap: document the docket attributes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44480 diff changeset	323 # the amount (in bytes) of "dead" data, still in the data file but no
283fd803afa5 nodemap: document the docket attributes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44480 diff changeset	324 # longer used for the nodemap.
44370 8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	325 self.data_unused = 0
44365 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	326
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	327 def copy(self):
44369 e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44367 diff changeset	328 new = NodeMapDocket(uid=self.uid)
e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44367 diff changeset	329 new.tip_rev = self.tip_rev
44525 6c906eaedd0d nodemap: track the tip_node for validation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44509 diff changeset	330 new.tip_node = self.tip_node
44370 8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	331 new.data_length = self.data_length
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	332 new.data_unused = self.data_unused
44369 e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44367 diff changeset	333 return new
44365 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	334
44371 1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	335 def __cmp__(self, other):
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	336 if self.uid < other.uid:
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	337 return -1
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	338 if self.uid > other.uid:
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	339 return 1
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	340 elif self.data_length < other.data_length:
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	341 return -1
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	342 elif self.data_length > other.data_length:
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	343 return 1
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	344 return 0
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	345
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	346 def __eq__(self, other):
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	347 return self.uid == other.uid and self.data_length == other.data_length
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	348
44365 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	349 def serialize(self):
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	350 """return serialized bytes for a docket using the passed uid"""
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	351 data = []
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	352 data.append(S_VERSION.pack(ONDISK_VERSION))
44370 8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	353 headers = (
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	354 len(self.uid),
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	355 self.tip_rev,
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	356 self.data_length,
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	357 self.data_unused,
44525 6c906eaedd0d nodemap: track the tip_node for validation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44509 diff changeset	358 len(self.tip_node),
44370 8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	359 )
44369 e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44367 diff changeset	360 data.append(S_HEADER.pack(*headers))
44365 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	361 data.append(self.uid)
44525 6c906eaedd0d nodemap: track the tip_node for validation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44509 diff changeset	362 data.append(self.tip_node)
44365 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	363 return b''.join(data)
44354 2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	364
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	365
44365 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	366 def _rawdata_filepath(revlog, docket):
44354 2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	367 """The (vfs relative) nodemap's rawdata file for a given uid"""
44526 64e2f603de9d nodemap: make sure hooks have access to an up-to-date version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44525 diff changeset	368 if revlog.nodemap_file.endswith(b'.n.a'):
64e2f603de9d nodemap: make sure hooks have access to an up-to-date version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44525 diff changeset	369 prefix = revlog.nodemap_file[:-4]
64e2f603de9d nodemap: make sure hooks have access to an up-to-date version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44525 diff changeset	370 else:
64e2f603de9d nodemap: make sure hooks have access to an up-to-date version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44525 diff changeset	371 prefix = revlog.nodemap_file[:-2]
44365 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	372 return b"%s-%s.nd" % (prefix, docket.uid)
44351 5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	373
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	374
44365 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	375 def _other_rawdata_filepath(revlog, docket):
44355 563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	376 prefix = revlog.nodemap_file[:-2]
44480 6aee0647e026 nodemap: fix missing r-prefix on regular expression Augie Fackler <augie@google.com> parents: 44465 diff changeset	377 pattern = re.compile(br"(^\|/)%s-[0-9a-f]+\.nd$" % prefix)
44365 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	378 new_file_path = _rawdata_filepath(revlog, docket)
44355 563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	379 new_file_name = revlog.opener.basename(new_file_path)
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	380 dirpath = revlog.opener.dirname(new_file_path)
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	381 others = []
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	382 for f in revlog.opener.listdir(dirpath):
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	383 if pattern.match(f) and f != new_file_name:
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	384 others.append(f)
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	385 return others
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	386
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	387
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	388 ### Nodemap Trie
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	389 #
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	390 # This is a simple reference implementation to compute and persist a nodemap
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	391 # trie. This reference implementation is write only. The python version of this
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	392 # is not expected to be actually used, since it wont provide performance
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	393 # improvement over existing non-persistent C implementation.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	394 #
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	395 # The nodemap is persisted as Trie using 4bits-address/16-entries block. each
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	396 # revision can be adressed using its node shortest prefix.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	397 #
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	398 # The trie is stored as a sequence of block. Each block contains 16 entries
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	399 # (signed 64bit integer, big endian). Each entry can be one of the following:
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	400 #
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	401 # * value >= 0 -> index of sub-block
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	402 # * value == -1 -> no value
46199 8e7ce6555ea7 nodemap: match comment to actual code Joerg Sonnenberger <joerg@bec.de> parents: 46114 diff changeset	403 # * value < -1 -> encoded revision: rev = -(value+2)
8e7ce6555ea7 nodemap: match comment to actual code Joerg Sonnenberger <joerg@bec.de> parents: 46114 diff changeset	404 #
8e7ce6555ea7 nodemap: match comment to actual code Joerg Sonnenberger <joerg@bec.de> parents: 46114 diff changeset	405 # See REV_OFFSET and _transform_rev below.
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	406 #
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	407 # The implementation focus on simplicity, not on performance. A Rust
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	408 # implementation should provide a efficient version of the same binary
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	409 # persistence. This reference python implementation is never meant to be
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	410 # extensively use in production.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	411
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	412
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	413 def persistent_data(index):
45957 89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44869 diff changeset	414 """return the persistent binary form for a nodemap for a given index"""
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	415 trie = _build_trie(index)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	416 return _persist_trie(trie)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	417
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	418
44367 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	419 def update_persistent_data(index, root, max_idx, last_rev):
45957 89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 44869 diff changeset	420 """return the incremental update for persistent nodemap from a given index"""
44370 8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	421 changed_block, trie = _update_trie(index, root, last_rev)
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	422 return (
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	423 changed_block * S_BLOCK.size,
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	424 _persist_trie(trie, existing_idx=max_idx),
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	425 )
44367 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	426
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	427
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	428 S_BLOCK = struct.Struct(">" + ("l" * 16))
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	429
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	430 NO_ENTRY = -1
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	431 # rev 0 need to be -2 because 0 is used by block, -1 is a special value.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	432 REV_OFFSET = 2
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	433
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	434
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	435 def _transform_rev(rev):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	436 """Return the number used to represent the rev in the tree.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	437
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	438 (or retrieve a rev number from such representation)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	439
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	440 Note that this is an involution, a function equal to its inverse (i.e.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	441 which gives the identity when applied to itself).
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	442 """
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	443 return -(rev + REV_OFFSET)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	444
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	445
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	446 def _to_int(hex_digit):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	447 """turn an hexadecimal digit into a proper integer"""
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	448 return int(hex_digit, 16)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	449
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	450
44358 7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44357 diff changeset	451 class Block(dict):
7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44357 diff changeset	452 """represent a block of the Trie
7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44357 diff changeset	453
7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44357 diff changeset	454 contains up to 16 entry indexed from 0 to 15"""
7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44357 diff changeset	455
44364 f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44362 diff changeset	456 def __init__(self):
f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44362 diff changeset	457 super(Block, self).__init__()
f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44362 diff changeset	458 # If this block exist on disk, here is its ID
f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44362 diff changeset	459 self.ondisk_id = None
f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44362 diff changeset	460
44359 55b12f2593c1 nodemap: move the iteratio inside the Block object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44358 diff changeset	461 def __iter__(self):
55b12f2593c1 nodemap: move the iteratio inside the Block object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44358 diff changeset	462 return iter(self.get(i) for i in range(16))
55b12f2593c1 nodemap: move the iteratio inside the Block object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44358 diff changeset	463
44358 7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44357 diff changeset	464
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	465 def _build_trie(index):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	466 """build a nodemap trie
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	467
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	468 The nodemap stores revision number for each unique prefix.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	469
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	470 Each block is a dictionary with keys in `[0, 15]`. Values are either
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	471 another block or a revision number.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	472 """
44358 7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44357 diff changeset	473 root = Block()
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	474 for rev in range(len(index)):
46114 59fa3890d40a node: import symbols explicitly Joerg Sonnenberger <joerg@bec.de> parents: 46090 diff changeset	475 current_hex = hex(index[rev][7])
59fa3890d40a node: import symbols explicitly Joerg Sonnenberger <joerg@bec.de> parents: 46090 diff changeset	476 _insert_into_block(index, 0, root, rev, current_hex)
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	477 return root
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	478
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	479
44367 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	480 def _update_trie(index, root, last_rev):
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	481 """consume"""
44370 8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	482 changed = 0
44367 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	483 for rev in range(last_rev + 1, len(index)):
46114 59fa3890d40a node: import symbols explicitly Joerg Sonnenberger <joerg@bec.de> parents: 46090 diff changeset	484 current_hex = hex(index[rev][7])
59fa3890d40a node: import symbols explicitly Joerg Sonnenberger <joerg@bec.de> parents: 46090 diff changeset	485 changed += _insert_into_block(index, 0, root, rev, current_hex)
44370 8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	486 return changed, root
44367 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	487
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	488
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	489 def _insert_into_block(index, level, block, current_rev, current_hex):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	490 """insert a new revision in a block
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	491
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	492 index: the index we are adding revision for
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	493 level: the depth of the current block in the trie
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	494 block: the block currently being considered
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	495 current_rev: the revision number we are adding
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	496 current_hex: the hexadecimal representation of the of that revision
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	497 """
44370 8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	498 changed = 1
44367 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	499 if block.ondisk_id is not None:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	500 block.ondisk_id = None
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	501 hex_digit = _to_int(current_hex[level : level + 1])
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	502 entry = block.get(hex_digit)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	503 if entry is None:
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	504 # no entry, simply store the revision number
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	505 block[hex_digit] = current_rev
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	506 elif isinstance(entry, dict):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	507 # need to recurse to an underlying block
44370 8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	508 changed += _insert_into_block(
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	509 index, level + 1, entry, current_rev, current_hex
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	510 )
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	511 else:
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	512 # collision with a previously unique prefix, inserting new
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	513 # vertices to fit both entry.
46114 59fa3890d40a node: import symbols explicitly Joerg Sonnenberger <joerg@bec.de> parents: 46090 diff changeset	514 other_hex = hex(index[entry][7])
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	515 other_rev = entry
44358 7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44357 diff changeset	516 new = Block()
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	517 block[hex_digit] = new
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	518 _insert_into_block(index, level + 1, new, other_rev, other_hex)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	519 _insert_into_block(index, level + 1, new, current_rev, current_hex)
44370 8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	520 return changed
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	521
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	522
44367 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	523 def _persist_trie(root, existing_idx=None):
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	524 """turn a nodemap trie into persistent binary data
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	525
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	526 See `_build_trie` for nodemap trie structure"""
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	527 block_map = {}
44367 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	528 if existing_idx is not None:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	529 base_idx = existing_idx + 1
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	530 else:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	531 base_idx = 0
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	532 chunks = []
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	533 for tn in _walk_trie(root):
44367 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	534 if tn.ondisk_id is not None:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	535 block_map[id(tn)] = tn.ondisk_id
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	536 else:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	537 block_map[id(tn)] = len(chunks) + base_idx
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	538 chunks.append(_persist_block(tn, block_map))
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	539 return b''.join(chunks)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	540
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	541
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	542 def _walk_trie(block):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	543 """yield all the block in a trie
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	544
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	545 Children blocks are always yield before their parent block.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	546 """
44696 ed475420e0af nodemap: rename a variable to prevent shawoding "_" Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44640 diff changeset	547 for (__, item) in sorted(block.items()):
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	548 if isinstance(item, dict):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	549 for sub_block in _walk_trie(item):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	550 yield sub_block
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	551 yield block
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	552
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	553
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	554 def _persist_block(block_node, block_map):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	555 """produce persistent binary data for a single block
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	556
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	557 Children block are assumed to be already persisted and present in
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	558 block_map.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	559 """
44359 55b12f2593c1 nodemap: move the iteratio inside the Block object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44358 diff changeset	560 data = tuple(_to_value(v, block_map) for v in block_node)
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	561 return S_BLOCK.pack(*data)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	562
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	563
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	564 def _to_value(item, block_map):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	565 """persist any value as an integer"""
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	566 if item is None:
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	567 return NO_ENTRY
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	568 elif isinstance(item, dict):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	569 return block_map[id(item)]
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	570 else:
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	571 return _transform_rev(item)
44360 78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	572
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	573
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	574 def parse_data(data):
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	575 """parse parse nodemap data into a nodemap Trie"""
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	576 if (len(data) % S_BLOCK.size) != 0:
46391 98c816e89fac nodemap: convert error message to bytes Matt Harbison <matt_harbison@yahoo.com> parents: 46321 diff changeset	577 msg = b"nodemap data size is not a multiple of block size (%d): %d"
44360 78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	578 raise error.Abort(msg % (S_BLOCK.size, len(data)))
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	579 if not data:
44367 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	580 return Block(), None
44360 78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	581 block_map = {}
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	582 new_blocks = []
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	583 for i in range(0, len(data), S_BLOCK.size):
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	584 block = Block()
44364 f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44362 diff changeset	585 block.ondisk_id = len(block_map)
f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44362 diff changeset	586 block_map[block.ondisk_id] = block
44360 78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	587 block_data = data[i : i + S_BLOCK.size]
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	588 values = S_BLOCK.unpack(block_data)
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	589 new_blocks.append((block, values))
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	590 for b, values in new_blocks:
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	591 for idx, v in enumerate(values):
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	592 if v == NO_ENTRY:
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	593 continue
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	594 elif v >= 0:
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	595 b[idx] = block_map[v]
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	596 else:
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	597 b[idx] = _transform_rev(v)
44367 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	598 return block, i // S_BLOCK.size
44361 20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	599
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	600
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	601 # debug utility
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	602
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	603
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	604 def check_data(ui, index, data):
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	605 """verify that the provided nodemap data are valid for the given idex"""
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	606 ret = 0
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	607 ui.status((b"revision in index: %d\n") % len(index))
44367 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	608 root, __ = parse_data(data)
44361 20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	609 all_revs = set(_all_revisions(root))
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	610 ui.status((b"revision in nodemap: %d\n") % len(all_revs))
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	611 for r in range(len(index)):
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	612 if r not in all_revs:
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	613 msg = b" revision missing from nodemap: %d\n" % r
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	614 ui.write_err(msg)
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	615 ret = 1
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	616 else:
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	617 all_revs.remove(r)
46114 59fa3890d40a node: import symbols explicitly Joerg Sonnenberger <joerg@bec.de> parents: 46090 diff changeset	618 nm_rev = _find_node(root, hex(index[r][7]))
44362 d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	619 if nm_rev is None:
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	620 msg = b" revision node does not match any entries: %d\n" % r
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	621 ui.write_err(msg)
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	622 ret = 1
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	623 elif nm_rev != r:
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	624 msg = (
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	625 b" revision node does not match the expected revision: "
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	626 b"%d != %d\n" % (r, nm_rev)
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	627 )
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	628 ui.write_err(msg)
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	629 ret = 1
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	630
44361 20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	631 if all_revs:
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	632 for r in sorted(all_revs):
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	633 msg = b" extra revision in nodemap: %d\n" % r
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	634 ui.write_err(msg)
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	635 ret = 1
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	636 return ret
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	637
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	638
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	639 def _all_revisions(root):
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	640 """return all revisions stored in a Trie"""
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	641 for block in _walk_trie(root):
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	642 for v in block:
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	643 if v is None or isinstance(v, Block):
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	644 continue
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	645 yield v
44362 d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	646
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	647
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	648 def _find_node(block, node):
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	649 """find the revision associated with a given node"""
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	650 entry = block.get(_to_int(node[0:1]))
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	651 if isinstance(entry, dict):
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	652 return _find_node(entry, node[1:])
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	653 return entry
46530 835aafb2543f revlog: refactor logic to compute nodemap file in separate function Pulkit Goyal <7895pulkit@gmail.com> parents: 46529 diff changeset	654
835aafb2543f revlog: refactor logic to compute nodemap file in separate function Pulkit Goyal <7895pulkit@gmail.com> parents: 46529 diff changeset	655
835aafb2543f revlog: refactor logic to compute nodemap file in separate function Pulkit Goyal <7895pulkit@gmail.com> parents: 46529 diff changeset	656 def get_nodemap_file(opener, indexfile):
835aafb2543f revlog: refactor logic to compute nodemap file in separate function Pulkit Goyal <7895pulkit@gmail.com> parents: 46529 diff changeset	657 if indexfile.endswith(b'.a'):
835aafb2543f revlog: refactor logic to compute nodemap file in separate function Pulkit Goyal <7895pulkit@gmail.com> parents: 46529 diff changeset	658 pending_path = indexfile[:-4] + b".n.a"
835aafb2543f revlog: refactor logic to compute nodemap file in separate function Pulkit Goyal <7895pulkit@gmail.com> parents: 46529 diff changeset	659 if opener.exists(pending_path):
835aafb2543f revlog: refactor logic to compute nodemap file in separate function Pulkit Goyal <7895pulkit@gmail.com> parents: 46529 diff changeset	660 return pending_path
835aafb2543f revlog: refactor logic to compute nodemap file in separate function Pulkit Goyal <7895pulkit@gmail.com> parents: 46529 diff changeset	661 else:
835aafb2543f revlog: refactor logic to compute nodemap file in separate function Pulkit Goyal <7895pulkit@gmail.com> parents: 46529 diff changeset	662 return indexfile[:-4] + b".n"
835aafb2543f revlog: refactor logic to compute nodemap file in separate function Pulkit Goyal <7895pulkit@gmail.com> parents: 46529 diff changeset	663 else:
835aafb2543f revlog: refactor logic to compute nodemap file in separate function Pulkit Goyal <7895pulkit@gmail.com> parents: 46529 diff changeset	664 return indexfile[:-2] + b".n"

Mercurial > public > mercurial-scm > hg-stable

annotate mercurial/revlogutils/nodemap.py @ 46971:a3720569a43f