mercurial-scm/hg-stable: mercurial/revlogutils/nodemap.py annotate

annotate mercurial/revlogutils/nodemap.py @ 44369:e41a164db7a9

nodemap: track the maximum revision tracked in the nodemap We need a simple way to detect when the on disk data contains less revision than the index we read from disk. The docket file is meant for this, we just had to start tracking that data. We should also try to detect strip operation, but we will deal with this in later changesets. Right now we are focusing on defining the API for index supporting persistent nodemap. Differential Revision: https://phab.mercurial-scm.org/D7888

author	Pierre-Yves David <pierre-yves.david@octobus.net>
date	Wed, 15 Jan 2020 15:50:14 +0100
parents	50ad851efd9b
children	8374b69aef75

rev	line source
44035 ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	1 # nodemap.py - nodemap related code and utilities
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	2 #
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	3 # Copyright 2019 Pierre-Yves David <pierre-yves.david@octobus.net>
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	4 # Copyright 2019 George Racinet <georges.racinet@octobus.net>
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	5 #
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	6 # This software may be used and distributed according to the terms of the
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	7 # GNU General Public License version 2 or any later version.
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	8
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	9 from __future__ import absolute_import
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	10
44354 2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	11 import os
44355 563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	12 import re
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	13 import struct
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	14
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	15 from .. import (
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	16 error,
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	17 node as nodemod,
44357 7f4f7ef3133e nodemap: add a optional `nodemap_add_full` method on indexes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44355 diff changeset	18 util,
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	19 )
44035 ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	20
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	21
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	22 class NodeMap(dict):
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	23 def __missing__(self, x):
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	24 raise error.RevlogError(b'unknown node: %s' % x)
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	25
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	26
44352 6c07480d6659 nodemap: add a function to read the data from disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44351 diff changeset	27 def persisted_data(revlog):
6c07480d6659 nodemap: add a function to read the data from disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44351 diff changeset	28 """read the nodemap for a revlog from disk"""
6c07480d6659 nodemap: add a function to read the data from disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44351 diff changeset	29 if revlog.nodemap_file is None:
6c07480d6659 nodemap: add a function to read the data from disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44351 diff changeset	30 return None
44354 2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	31 pdata = revlog.opener.tryread(revlog.nodemap_file)
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	32 if not pdata:
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	33 return None
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	34 offset = 0
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	35 (version,) = S_VERSION.unpack(pdata[offset : offset + S_VERSION.size])
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	36 if version != ONDISK_VERSION:
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	37 return None
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	38 offset += S_VERSION.size
44369 e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44367 diff changeset	39 headers = S_HEADER.unpack(pdata[offset : offset + S_HEADER.size])
e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44367 diff changeset	40 uid_size, tip_rev = headers
44354 2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	41 offset += S_HEADER.size
44365 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	42 docket = NodeMapDocket(pdata[offset : offset + uid_size])
44369 e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44367 diff changeset	43 docket.tip_rev = tip_rev
44354 2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	44
44365 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	45 filename = _rawdata_filepath(revlog, docket)
44366 76a96e3a2bbb nodemap: keep track of the docket for loaded data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44365 diff changeset	46 return docket, revlog.opener.tryread(filename)
44352 6c07480d6659 nodemap: add a function to read the data from disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44351 diff changeset	47
6c07480d6659 nodemap: add a function to read the data from disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44351 diff changeset	48
44351 5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	49 def setup_persistent_nodemap(tr, revlog):
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	50 """Install whatever is needed transaction side to persist a nodemap on disk
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	51
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	52 (only actually persist the nodemap if this is relevant for this revlog)
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	53 """
44353 daad3aace942 nodemap: only use persistent nodemap for non-inlined revlog Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44352 diff changeset	54 if revlog._inline:
daad3aace942 nodemap: only use persistent nodemap for non-inlined revlog Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44352 diff changeset	55 return # inlined revlog are too small for this to be relevant
44351 5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	56 if revlog.nodemap_file is None:
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	57 return # we do not use persistent_nodemap on this revlog
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	58 callback_id = b"revlog-persistent-nodemap-%s" % revlog.nodemap_file
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	59 if tr.hasfinalize(callback_id):
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	60 return # no need to register again
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	61 tr.addfinalize(callback_id, lambda tr: _persist_nodemap(tr, revlog))
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	62
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	63
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	64 def _persist_nodemap(tr, revlog):
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	65 """Write nodemap data on disk for a given revlog
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	66 """
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	67 if getattr(revlog, 'filteredrevs', ()):
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	68 raise error.ProgrammingError(
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	69 "cannot persist nodemap of a filtered changelog"
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	70 )
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	71 if revlog.nodemap_file is None:
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	72 msg = "calling persist nodemap on a revlog without the feature enableb"
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	73 raise error.ProgrammingError(msg)
44367 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	74
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	75 can_incremental = util.safehasattr(revlog.index, "nodemap_data_incremental")
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	76 ondisk_docket = revlog._nodemap_docket
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	77
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	78 # first attemp an incremental update of the data
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	79 if can_incremental and ondisk_docket is not None:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	80 target_docket = revlog._nodemap_docket.copy()
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	81 data = revlog.index.nodemap_data_incremental()
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	82 datafile = _rawdata_filepath(revlog, target_docket)
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	83 # EXP-TODO: if this is a cache, this should use a cache vfs, not a
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	84 # store vfs
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	85 with revlog.opener(datafile, b'a') as fd:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	86 fd.write(data)
44357 7f4f7ef3133e nodemap: add a optional `nodemap_add_full` method on indexes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44355 diff changeset	87 else:
44367 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	88 # otherwise fallback to a full new export
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	89 target_docket = NodeMapDocket()
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	90 datafile = _rawdata_filepath(revlog, target_docket)
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	91 if util.safehasattr(revlog.index, "nodemap_data_all"):
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	92 data = revlog.index.nodemap_data_all()
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	93 else:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	94 data = persistent_data(revlog.index)
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	95 # EXP-TODO: if this is a cache, this should use a cache vfs, not a
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	96 # store vfs
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	97 with revlog.opener(datafile, b'w') as fd:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	98 fd.write(data)
44369 e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44367 diff changeset	99 target_docket.tip_rev = revlog.tiprev()
44367 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	100 # EXP-TODO: if this is a cache, this should use a cache vfs, not a
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	101 # store vfs
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	102 with revlog.opener(revlog.nodemap_file, b'w', atomictemp=True) as fp:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	103 fp.write(target_docket.serialize())
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	104 revlog._nodemap_docket = target_docket
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	105 # EXP-TODO: if the transaction abort, we should remove the new data and
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	106 # reinstall the old one.
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	107
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	108 # search for old index file in all cases, some older process might have
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	109 # left one behind.
44365 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	110 olds = _other_rawdata_filepath(revlog, target_docket)
44355 563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	111 if olds:
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	112 realvfs = getattr(revlog, '_realopener', revlog.opener)
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	113
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	114 def cleanup(tr):
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	115 for oldfile in olds:
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	116 realvfs.tryunlink(oldfile)
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	117
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	118 callback_id = b"revlog-cleanup-nodemap-%s" % revlog.nodemap_file
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	119 tr.addpostclose(callback_id, cleanup)
44354 2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	120
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	121
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	122 ### Nodemap docket file
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	123 #
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	124 # The nodemap data are stored on disk using 2 files:
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	125 #
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	126 # * a raw data files containing a persistent nodemap
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	127 # (see `Nodemap Trie` section)
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	128 #
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	129 # * a small "docket" file containing medatadata
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	130 #
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	131 # While the nodemap data can be multiple tens of megabytes, the "docket" is
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	132 # small, it is easy to update it automatically or to duplicated its content
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	133 # during a transaction.
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	134 #
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	135 # Multiple raw data can exist at the same time (The currently valid one and a
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	136 # new one beind used by an in progress transaction). To accomodate this, the
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	137 # filename hosting the raw data has a variable parts. The exact filename is
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	138 # specified inside the "docket" file.
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	139 #
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	140 # The docket file contains information to find, qualify and validate the raw
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	141 # data. Its content is currently very light, but it will expand as the on disk
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	142 # nodemap gains the necessary features to be used in production.
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	143
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	144 # version 0 is experimental, no BC garantee, do no use outside of tests.
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	145 ONDISK_VERSION = 0
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	146
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	147 S_VERSION = struct.Struct(">B")
44369 e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44367 diff changeset	148 S_HEADER = struct.Struct(">BQ")
44354 2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	149
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	150 ID_SIZE = 8
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	151
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	152
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	153 def _make_uid():
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	154 """return a new unique identifier.
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	155
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	156 The identifier is random and composed of ascii characters."""
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	157 return nodemod.hex(os.urandom(ID_SIZE))
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	158
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	159
44365 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	160 class NodeMapDocket(object):
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	161 """metadata associated with persistent nodemap data
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	162
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	163 The persistent data may come from disk or be on their way to disk.
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	164 """
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	165
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	166 def __init__(self, uid=None):
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	167 if uid is None:
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	168 uid = _make_uid()
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	169 self.uid = uid
44369 e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44367 diff changeset	170 self.tip_rev = None
44365 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	171
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	172 def copy(self):
44369 e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44367 diff changeset	173 new = NodeMapDocket(uid=self.uid)
e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44367 diff changeset	174 new.tip_rev = self.tip_rev
e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44367 diff changeset	175 return new
44365 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	176
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	177 def serialize(self):
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	178 """return serialized bytes for a docket using the passed uid"""
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	179 data = []
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	180 data.append(S_VERSION.pack(ONDISK_VERSION))
44369 e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44367 diff changeset	181 headers = (len(self.uid), self.tip_rev)
e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44367 diff changeset	182 data.append(S_HEADER.pack(*headers))
44365 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	183 data.append(self.uid)
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	184 return b''.join(data)
44354 2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	185
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	186
44365 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	187 def _rawdata_filepath(revlog, docket):
44354 2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	188 """The (vfs relative) nodemap's rawdata file for a given uid"""
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	189 prefix = revlog.nodemap_file[:-2]
44365 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	190 return b"%s-%s.nd" % (prefix, docket.uid)
44351 5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	191
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	192
44365 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	193 def _other_rawdata_filepath(revlog, docket):
44355 563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	194 prefix = revlog.nodemap_file[:-2]
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	195 pattern = re.compile(b"(^\|/)%s-[0-9a-f]+\.nd$" % prefix)
44365 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	196 new_file_path = _rawdata_filepath(revlog, docket)
44355 563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	197 new_file_name = revlog.opener.basename(new_file_path)
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	198 dirpath = revlog.opener.dirname(new_file_path)
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	199 others = []
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	200 for f in revlog.opener.listdir(dirpath):
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	201 if pattern.match(f) and f != new_file_name:
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	202 others.append(f)
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	203 return others
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	204
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	205
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	206 ### Nodemap Trie
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	207 #
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	208 # This is a simple reference implementation to compute and persist a nodemap
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	209 # trie. This reference implementation is write only. The python version of this
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	210 # is not expected to be actually used, since it wont provide performance
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	211 # improvement over existing non-persistent C implementation.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	212 #
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	213 # The nodemap is persisted as Trie using 4bits-address/16-entries block. each
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	214 # revision can be adressed using its node shortest prefix.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	215 #
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	216 # The trie is stored as a sequence of block. Each block contains 16 entries
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	217 # (signed 64bit integer, big endian). Each entry can be one of the following:
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	218 #
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	219 # * value >= 0 -> index of sub-block
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	220 # * value == -1 -> no value
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	221 # * value < -1 -> a revision value: rev = -(value+10)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	222 #
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	223 # The implementation focus on simplicity, not on performance. A Rust
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	224 # implementation should provide a efficient version of the same binary
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	225 # persistence. This reference python implementation is never meant to be
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	226 # extensively use in production.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	227
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	228
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	229 def persistent_data(index):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	230 """return the persistent binary form for a nodemap for a given index
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	231 """
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	232 trie = _build_trie(index)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	233 return _persist_trie(trie)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	234
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	235
44367 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	236 def update_persistent_data(index, root, max_idx, last_rev):
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	237 """return the incremental update for persistent nodemap from a given index
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	238 """
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	239 trie = _update_trie(index, root, last_rev)
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	240 return _persist_trie(trie, existing_idx=max_idx)
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	241
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	242
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	243 S_BLOCK = struct.Struct(">" + ("l" * 16))
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	244
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	245 NO_ENTRY = -1
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	246 # rev 0 need to be -2 because 0 is used by block, -1 is a special value.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	247 REV_OFFSET = 2
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	248
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	249
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	250 def _transform_rev(rev):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	251 """Return the number used to represent the rev in the tree.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	252
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	253 (or retrieve a rev number from such representation)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	254
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	255 Note that this is an involution, a function equal to its inverse (i.e.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	256 which gives the identity when applied to itself).
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	257 """
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	258 return -(rev + REV_OFFSET)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	259
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	260
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	261 def _to_int(hex_digit):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	262 """turn an hexadecimal digit into a proper integer"""
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	263 return int(hex_digit, 16)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	264
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	265
44358 7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44357 diff changeset	266 class Block(dict):
7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44357 diff changeset	267 """represent a block of the Trie
7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44357 diff changeset	268
7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44357 diff changeset	269 contains up to 16 entry indexed from 0 to 15"""
7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44357 diff changeset	270
44364 f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44362 diff changeset	271 def __init__(self):
f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44362 diff changeset	272 super(Block, self).__init__()
f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44362 diff changeset	273 # If this block exist on disk, here is its ID
f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44362 diff changeset	274 self.ondisk_id = None
f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44362 diff changeset	275
44359 55b12f2593c1 nodemap: move the iteratio inside the Block object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44358 diff changeset	276 def __iter__(self):
55b12f2593c1 nodemap: move the iteratio inside the Block object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44358 diff changeset	277 return iter(self.get(i) for i in range(16))
55b12f2593c1 nodemap: move the iteratio inside the Block object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44358 diff changeset	278
44358 7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44357 diff changeset	279
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	280 def _build_trie(index):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	281 """build a nodemap trie
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	282
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	283 The nodemap stores revision number for each unique prefix.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	284
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	285 Each block is a dictionary with keys in `[0, 15]`. Values are either
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	286 another block or a revision number.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	287 """
44358 7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44357 diff changeset	288 root = Block()
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	289 for rev in range(len(index)):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	290 hex = nodemod.hex(index[rev][7])
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	291 _insert_into_block(index, 0, root, rev, hex)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	292 return root
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	293
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	294
44367 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	295 def _update_trie(index, root, last_rev):
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	296 """consume"""
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	297 for rev in range(last_rev + 1, len(index)):
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	298 hex = nodemod.hex(index[rev][7])
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	299 _insert_into_block(index, 0, root, rev, hex)
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	300 return root
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	301
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	302
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	303 def _insert_into_block(index, level, block, current_rev, current_hex):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	304 """insert a new revision in a block
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	305
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	306 index: the index we are adding revision for
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	307 level: the depth of the current block in the trie
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	308 block: the block currently being considered
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	309 current_rev: the revision number we are adding
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	310 current_hex: the hexadecimal representation of the of that revision
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	311 """
44367 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	312 if block.ondisk_id is not None:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	313 block.ondisk_id = None
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	314 hex_digit = _to_int(current_hex[level : level + 1])
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	315 entry = block.get(hex_digit)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	316 if entry is None:
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	317 # no entry, simply store the revision number
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	318 block[hex_digit] = current_rev
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	319 elif isinstance(entry, dict):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	320 # need to recurse to an underlying block
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	321 _insert_into_block(index, level + 1, entry, current_rev, current_hex)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	322 else:
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	323 # collision with a previously unique prefix, inserting new
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	324 # vertices to fit both entry.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	325 other_hex = nodemod.hex(index[entry][7])
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	326 other_rev = entry
44358 7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44357 diff changeset	327 new = Block()
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	328 block[hex_digit] = new
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	329 _insert_into_block(index, level + 1, new, other_rev, other_hex)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	330 _insert_into_block(index, level + 1, new, current_rev, current_hex)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	331
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	332
44367 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	333 def _persist_trie(root, existing_idx=None):
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	334 """turn a nodemap trie into persistent binary data
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	335
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	336 See `_build_trie` for nodemap trie structure"""
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	337 block_map = {}
44367 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	338 if existing_idx is not None:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	339 base_idx = existing_idx + 1
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	340 else:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	341 base_idx = 0
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	342 chunks = []
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	343 for tn in _walk_trie(root):
44367 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	344 if tn.ondisk_id is not None:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	345 block_map[id(tn)] = tn.ondisk_id
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	346 else:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	347 block_map[id(tn)] = len(chunks) + base_idx
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	348 chunks.append(_persist_block(tn, block_map))
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	349 return b''.join(chunks)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	350
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	351
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	352 def _walk_trie(block):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	353 """yield all the block in a trie
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	354
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	355 Children blocks are always yield before their parent block.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	356 """
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	357 for (_, item) in sorted(block.items()):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	358 if isinstance(item, dict):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	359 for sub_block in _walk_trie(item):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	360 yield sub_block
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	361 yield block
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	362
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	363
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	364 def _persist_block(block_node, block_map):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	365 """produce persistent binary data for a single block
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	366
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	367 Children block are assumed to be already persisted and present in
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	368 block_map.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	369 """
44359 55b12f2593c1 nodemap: move the iteratio inside the Block object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44358 diff changeset	370 data = tuple(_to_value(v, block_map) for v in block_node)
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	371 return S_BLOCK.pack(*data)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	372
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	373
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	374 def _to_value(item, block_map):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	375 """persist any value as an integer"""
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	376 if item is None:
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	377 return NO_ENTRY
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	378 elif isinstance(item, dict):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	379 return block_map[id(item)]
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	380 else:
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	381 return _transform_rev(item)
44360 78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	382
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	383
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	384 def parse_data(data):
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	385 """parse parse nodemap data into a nodemap Trie"""
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	386 if (len(data) % S_BLOCK.size) != 0:
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	387 msg = "nodemap data size is not a multiple of block size (%d): %d"
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	388 raise error.Abort(msg % (S_BLOCK.size, len(data)))
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	389 if not data:
44367 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	390 return Block(), None
44360 78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	391 block_map = {}
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	392 new_blocks = []
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	393 for i in range(0, len(data), S_BLOCK.size):
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	394 block = Block()
44364 f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44362 diff changeset	395 block.ondisk_id = len(block_map)
f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44362 diff changeset	396 block_map[block.ondisk_id] = block
44360 78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	397 block_data = data[i : i + S_BLOCK.size]
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	398 values = S_BLOCK.unpack(block_data)
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	399 new_blocks.append((block, values))
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	400 for b, values in new_blocks:
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	401 for idx, v in enumerate(values):
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	402 if v == NO_ENTRY:
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	403 continue
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	404 elif v >= 0:
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	405 b[idx] = block_map[v]
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	406 else:
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	407 b[idx] = _transform_rev(v)
44367 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	408 return block, i // S_BLOCK.size
44361 20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	409
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	410
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	411 # debug utility
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	412
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	413
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	414 def check_data(ui, index, data):
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	415 """verify that the provided nodemap data are valid for the given idex"""
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	416 ret = 0
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	417 ui.status((b"revision in index: %d\n") % len(index))
44367 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	418 root, __ = parse_data(data)
44361 20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	419 all_revs = set(_all_revisions(root))
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	420 ui.status((b"revision in nodemap: %d\n") % len(all_revs))
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	421 for r in range(len(index)):
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	422 if r not in all_revs:
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	423 msg = b" revision missing from nodemap: %d\n" % r
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	424 ui.write_err(msg)
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	425 ret = 1
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	426 else:
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	427 all_revs.remove(r)
44362 d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	428 nm_rev = _find_node(root, nodemod.hex(index[r][7]))
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	429 if nm_rev is None:
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	430 msg = b" revision node does not match any entries: %d\n" % r
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	431 ui.write_err(msg)
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	432 ret = 1
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	433 elif nm_rev != r:
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	434 msg = (
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	435 b" revision node does not match the expected revision: "
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	436 b"%d != %d\n" % (r, nm_rev)
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	437 )
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	438 ui.write_err(msg)
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	439 ret = 1
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	440
44361 20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	441 if all_revs:
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	442 for r in sorted(all_revs):
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	443 msg = b" extra revision in nodemap: %d\n" % r
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	444 ui.write_err(msg)
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	445 ret = 1
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	446 return ret
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	447
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	448
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	449 def _all_revisions(root):
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	450 """return all revisions stored in a Trie"""
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	451 for block in _walk_trie(root):
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	452 for v in block:
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	453 if v is None or isinstance(v, Block):
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	454 continue
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	455 yield v
44362 d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	456
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	457
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	458 def _find_node(block, node):
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	459 """find the revision associated with a given node"""
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	460 entry = block.get(_to_int(node[0:1]))
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	461 if isinstance(entry, dict):
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	462 return _find_node(entry, node[1:])
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	463 return entry

Mercurial > public > mercurial-scm > hg-stable

annotate mercurial/revlogutils/nodemap.py @ 44369:e41a164db7a9