mercurial-scm/hg: mercurial/revlogutils/nodemap.py annotate

annotate mercurial/revlogutils/nodemap.py @ 44339:c7eebdb15139

nodemap: never read more than the expected data amount Since we are tracking this number we can use it to detect corrupted rawdata file and to only read the correct amount of data when possible. Differential Revision: https://phab.mercurial-scm.org/D7892

author	Pierre-Yves David <pierre-yves.david@octobus.net>
date	Wed, 15 Jan 2020 15:50:52 +0100
parents	2ea6a67ff502
children	6ecc34b31137

rev	line source
44034 ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	1 # nodemap.py - nodemap related code and utilities
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	2 #
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	3 # Copyright 2019 Pierre-Yves David <pierre-yves.david@octobus.net>
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	4 # Copyright 2019 George Racinet <georges.racinet@octobus.net>
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	5 #
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	6 # This software may be used and distributed according to the terms of the
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	7 # GNU General Public License version 2 or any later version.
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	8
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	9 from __future__ import absolute_import
44307 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	10
44311 2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	11 import os
44312 563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44311 diff changeset	12 import re
44307 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	13 import struct
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	14
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	15 from .. import (
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	16 error,
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	17 node as nodemod,
44314 7f4f7ef3133e nodemap: add a optional `nodemap_add_full` method on indexes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44312 diff changeset	18 util,
44307 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	19 )
44034 ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	20
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	21
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	22 class NodeMap(dict):
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	23 def __missing__(self, x):
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	24 raise error.RevlogError(b'unknown node: %s' % x)
44307 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	25
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	26
44309 6c07480d6659 nodemap: add a function to read the data from disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44308 diff changeset	27 def persisted_data(revlog):
6c07480d6659 nodemap: add a function to read the data from disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44308 diff changeset	28 """read the nodemap for a revlog from disk"""
6c07480d6659 nodemap: add a function to read the data from disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44308 diff changeset	29 if revlog.nodemap_file is None:
6c07480d6659 nodemap: add a function to read the data from disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44308 diff changeset	30 return None
44311 2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	31 pdata = revlog.opener.tryread(revlog.nodemap_file)
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	32 if not pdata:
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	33 return None
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	34 offset = 0
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	35 (version,) = S_VERSION.unpack(pdata[offset : offset + S_VERSION.size])
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	36 if version != ONDISK_VERSION:
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	37 return None
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	38 offset += S_VERSION.size
44335 e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44333 diff changeset	39 headers = S_HEADER.unpack(pdata[offset : offset + S_HEADER.size])
44336 8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44335 diff changeset	40 uid_size, tip_rev, data_length, data_unused = headers
44311 2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	41 offset += S_HEADER.size
44322 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44321 diff changeset	42 docket = NodeMapDocket(pdata[offset : offset + uid_size])
44335 e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44333 diff changeset	43 docket.tip_rev = tip_rev
44336 8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44335 diff changeset	44 docket.data_length = data_length
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44335 diff changeset	45 docket.data_unused = data_unused
44311 2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	46
44322 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44321 diff changeset	47 filename = _rawdata_filepath(revlog, docket)
44339 c7eebdb15139 nodemap: never read more than the expected data amount Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44338 diff changeset	48 data = revlog.opener.tryread(filename)
c7eebdb15139 nodemap: never read more than the expected data amount Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44338 diff changeset	49 if len(data) < data_length:
c7eebdb15139 nodemap: never read more than the expected data amount Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44338 diff changeset	50 return None
c7eebdb15139 nodemap: never read more than the expected data amount Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44338 diff changeset	51 elif len(data) > data_length:
c7eebdb15139 nodemap: never read more than the expected data amount Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44338 diff changeset	52 data = data[:data_length]
c7eebdb15139 nodemap: never read more than the expected data amount Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44338 diff changeset	53 return docket, data
44309 6c07480d6659 nodemap: add a function to read the data from disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44308 diff changeset	54
6c07480d6659 nodemap: add a function to read the data from disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44308 diff changeset	55
44308 5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44307 diff changeset	56 def setup_persistent_nodemap(tr, revlog):
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44307 diff changeset	57 """Install whatever is needed transaction side to persist a nodemap on disk
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44307 diff changeset	58
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44307 diff changeset	59 (only actually persist the nodemap if this is relevant for this revlog)
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44307 diff changeset	60 """
44310 daad3aace942 nodemap: only use persistent nodemap for non-inlined revlog Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44309 diff changeset	61 if revlog._inline:
daad3aace942 nodemap: only use persistent nodemap for non-inlined revlog Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44309 diff changeset	62 return # inlined revlog are too small for this to be relevant
44308 5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44307 diff changeset	63 if revlog.nodemap_file is None:
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44307 diff changeset	64 return # we do not use persistent_nodemap on this revlog
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44307 diff changeset	65 callback_id = b"revlog-persistent-nodemap-%s" % revlog.nodemap_file
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44307 diff changeset	66 if tr.hasfinalize(callback_id):
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44307 diff changeset	67 return # no need to register again
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44307 diff changeset	68 tr.addfinalize(callback_id, lambda tr: _persist_nodemap(tr, revlog))
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44307 diff changeset	69
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44307 diff changeset	70
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44307 diff changeset	71 def _persist_nodemap(tr, revlog):
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44307 diff changeset	72 """Write nodemap data on disk for a given revlog
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44307 diff changeset	73 """
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44307 diff changeset	74 if getattr(revlog, 'filteredrevs', ()):
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44307 diff changeset	75 raise error.ProgrammingError(
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44307 diff changeset	76 "cannot persist nodemap of a filtered changelog"
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44307 diff changeset	77 )
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44307 diff changeset	78 if revlog.nodemap_file is None:
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44307 diff changeset	79 msg = "calling persist nodemap on a revlog without the feature enableb"
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44307 diff changeset	80 raise error.ProgrammingError(msg)
44333 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	81
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	82 can_incremental = util.safehasattr(revlog.index, "nodemap_data_incremental")
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	83 ondisk_docket = revlog._nodemap_docket
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	84
44337 1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44336 diff changeset	85 data = None
44333 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	86 # first attemp an incremental update of the data
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	87 if can_incremental and ondisk_docket is not None:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	88 target_docket = revlog._nodemap_docket.copy()
44337 1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44336 diff changeset	89 (
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44336 diff changeset	90 src_docket,
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44336 diff changeset	91 data_changed_count,
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44336 diff changeset	92 data,
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44336 diff changeset	93 ) = revlog.index.nodemap_data_incremental()
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44336 diff changeset	94 if src_docket != target_docket:
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44336 diff changeset	95 data = None
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44336 diff changeset	96 else:
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44336 diff changeset	97 datafile = _rawdata_filepath(revlog, target_docket)
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44336 diff changeset	98 # EXP-TODO: if this is a cache, this should use a cache vfs, not a
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44336 diff changeset	99 # store vfs
44338 2ea6a67ff502 nodemap: write new data from the expected current data length Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44337 diff changeset	100 with revlog.opener(datafile, b'r+') as fd:
2ea6a67ff502 nodemap: write new data from the expected current data length Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44337 diff changeset	101 fd.seek(target_docket.data_length)
44337 1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44336 diff changeset	102 fd.write(data)
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44336 diff changeset	103 target_docket.data_length += len(data)
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44336 diff changeset	104 target_docket.data_unused += data_changed_count
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44336 diff changeset	105
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44336 diff changeset	106 if data is None:
44333 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	107 # otherwise fallback to a full new export
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	108 target_docket = NodeMapDocket()
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	109 datafile = _rawdata_filepath(revlog, target_docket)
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	110 if util.safehasattr(revlog.index, "nodemap_data_all"):
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	111 data = revlog.index.nodemap_data_all()
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	112 else:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	113 data = persistent_data(revlog.index)
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	114 # EXP-TODO: if this is a cache, this should use a cache vfs, not a
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	115 # store vfs
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	116 with revlog.opener(datafile, b'w') as fd:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	117 fd.write(data)
44336 8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44335 diff changeset	118 target_docket.data_length = len(data)
44335 e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44333 diff changeset	119 target_docket.tip_rev = revlog.tiprev()
44333 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	120 # EXP-TODO: if this is a cache, this should use a cache vfs, not a
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	121 # store vfs
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	122 with revlog.opener(revlog.nodemap_file, b'w', atomictemp=True) as fp:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	123 fp.write(target_docket.serialize())
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	124 revlog._nodemap_docket = target_docket
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	125 # EXP-TODO: if the transaction abort, we should remove the new data and
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	126 # reinstall the old one.
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	127
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	128 # search for old index file in all cases, some older process might have
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	129 # left one behind.
44322 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44321 diff changeset	130 olds = _other_rawdata_filepath(revlog, target_docket)
44312 563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44311 diff changeset	131 if olds:
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44311 diff changeset	132 realvfs = getattr(revlog, '_realopener', revlog.opener)
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44311 diff changeset	133
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44311 diff changeset	134 def cleanup(tr):
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44311 diff changeset	135 for oldfile in olds:
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44311 diff changeset	136 realvfs.tryunlink(oldfile)
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44311 diff changeset	137
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44311 diff changeset	138 callback_id = b"revlog-cleanup-nodemap-%s" % revlog.nodemap_file
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44311 diff changeset	139 tr.addpostclose(callback_id, cleanup)
44311 2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	140
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	141
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	142 ### Nodemap docket file
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	143 #
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	144 # The nodemap data are stored on disk using 2 files:
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	145 #
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	146 # * a raw data files containing a persistent nodemap
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	147 # (see `Nodemap Trie` section)
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	148 #
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	149 # * a small "docket" file containing medatadata
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	150 #
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	151 # While the nodemap data can be multiple tens of megabytes, the "docket" is
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	152 # small, it is easy to update it automatically or to duplicated its content
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	153 # during a transaction.
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	154 #
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	155 # Multiple raw data can exist at the same time (The currently valid one and a
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	156 # new one beind used by an in progress transaction). To accomodate this, the
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	157 # filename hosting the raw data has a variable parts. The exact filename is
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	158 # specified inside the "docket" file.
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	159 #
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	160 # The docket file contains information to find, qualify and validate the raw
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	161 # data. Its content is currently very light, but it will expand as the on disk
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	162 # nodemap gains the necessary features to be used in production.
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	163
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	164 # version 0 is experimental, no BC garantee, do no use outside of tests.
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	165 ONDISK_VERSION = 0
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	166 S_VERSION = struct.Struct(">B")
44336 8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44335 diff changeset	167 S_HEADER = struct.Struct(">BQQQ")
44311 2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	168
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	169 ID_SIZE = 8
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	170
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	171
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	172 def _make_uid():
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	173 """return a new unique identifier.
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	174
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	175 The identifier is random and composed of ascii characters."""
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	176 return nodemod.hex(os.urandom(ID_SIZE))
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	177
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	178
44322 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44321 diff changeset	179 class NodeMapDocket(object):
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44321 diff changeset	180 """metadata associated with persistent nodemap data
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44321 diff changeset	181
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44321 diff changeset	182 The persistent data may come from disk or be on their way to disk.
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44321 diff changeset	183 """
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44321 diff changeset	184
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44321 diff changeset	185 def __init__(self, uid=None):
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44321 diff changeset	186 if uid is None:
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44321 diff changeset	187 uid = _make_uid()
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44321 diff changeset	188 self.uid = uid
44335 e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44333 diff changeset	189 self.tip_rev = None
44336 8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44335 diff changeset	190 self.data_length = None
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44335 diff changeset	191 self.data_unused = 0
44322 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44321 diff changeset	192
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44321 diff changeset	193 def copy(self):
44335 e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44333 diff changeset	194 new = NodeMapDocket(uid=self.uid)
e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44333 diff changeset	195 new.tip_rev = self.tip_rev
44336 8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44335 diff changeset	196 new.data_length = self.data_length
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44335 diff changeset	197 new.data_unused = self.data_unused
44335 e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44333 diff changeset	198 return new
44322 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44321 diff changeset	199
44337 1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44336 diff changeset	200 def __cmp__(self, other):
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44336 diff changeset	201 if self.uid < other.uid:
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44336 diff changeset	202 return -1
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44336 diff changeset	203 if self.uid > other.uid:
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44336 diff changeset	204 return 1
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44336 diff changeset	205 elif self.data_length < other.data_length:
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44336 diff changeset	206 return -1
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44336 diff changeset	207 elif self.data_length > other.data_length:
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44336 diff changeset	208 return 1
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44336 diff changeset	209 return 0
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44336 diff changeset	210
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44336 diff changeset	211 def __eq__(self, other):
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44336 diff changeset	212 return self.uid == other.uid and self.data_length == other.data_length
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44336 diff changeset	213
44322 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44321 diff changeset	214 def serialize(self):
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44321 diff changeset	215 """return serialized bytes for a docket using the passed uid"""
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44321 diff changeset	216 data = []
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44321 diff changeset	217 data.append(S_VERSION.pack(ONDISK_VERSION))
44336 8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44335 diff changeset	218 headers = (
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44335 diff changeset	219 len(self.uid),
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44335 diff changeset	220 self.tip_rev,
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44335 diff changeset	221 self.data_length,
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44335 diff changeset	222 self.data_unused,
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44335 diff changeset	223 )
44335 e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44333 diff changeset	224 data.append(S_HEADER.pack(*headers))
44322 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44321 diff changeset	225 data.append(self.uid)
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44321 diff changeset	226 return b''.join(data)
44311 2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	227
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	228
44322 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44321 diff changeset	229 def _rawdata_filepath(revlog, docket):
44311 2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	230 """The (vfs relative) nodemap's rawdata file for a given uid"""
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44310 diff changeset	231 prefix = revlog.nodemap_file[:-2]
44322 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44321 diff changeset	232 return b"%s-%s.nd" % (prefix, docket.uid)
44308 5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44307 diff changeset	233
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44307 diff changeset	234
44322 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44321 diff changeset	235 def _other_rawdata_filepath(revlog, docket):
44312 563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44311 diff changeset	236 prefix = revlog.nodemap_file[:-2]
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44311 diff changeset	237 pattern = re.compile(b"(^\|/)%s-[0-9a-f]+\.nd$" % prefix)
44322 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44321 diff changeset	238 new_file_path = _rawdata_filepath(revlog, docket)
44312 563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44311 diff changeset	239 new_file_name = revlog.opener.basename(new_file_path)
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44311 diff changeset	240 dirpath = revlog.opener.dirname(new_file_path)
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44311 diff changeset	241 others = []
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44311 diff changeset	242 for f in revlog.opener.listdir(dirpath):
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44311 diff changeset	243 if pattern.match(f) and f != new_file_name:
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44311 diff changeset	244 others.append(f)
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44311 diff changeset	245 return others
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44311 diff changeset	246
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44311 diff changeset	247
44307 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	248 ### Nodemap Trie
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	249 #
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	250 # This is a simple reference implementation to compute and persist a nodemap
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	251 # trie. This reference implementation is write only. The python version of this
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	252 # is not expected to be actually used, since it wont provide performance
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	253 # improvement over existing non-persistent C implementation.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	254 #
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	255 # The nodemap is persisted as Trie using 4bits-address/16-entries block. each
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	256 # revision can be adressed using its node shortest prefix.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	257 #
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	258 # The trie is stored as a sequence of block. Each block contains 16 entries
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	259 # (signed 64bit integer, big endian). Each entry can be one of the following:
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	260 #
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	261 # * value >= 0 -> index of sub-block
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	262 # * value == -1 -> no value
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	263 # * value < -1 -> a revision value: rev = -(value+10)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	264 #
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	265 # The implementation focus on simplicity, not on performance. A Rust
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	266 # implementation should provide a efficient version of the same binary
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	267 # persistence. This reference python implementation is never meant to be
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	268 # extensively use in production.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	269
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	270
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	271 def persistent_data(index):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	272 """return the persistent binary form for a nodemap for a given index
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	273 """
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	274 trie = _build_trie(index)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	275 return _persist_trie(trie)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	276
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	277
44333 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	278 def update_persistent_data(index, root, max_idx, last_rev):
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	279 """return the incremental update for persistent nodemap from a given index
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	280 """
44336 8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44335 diff changeset	281 changed_block, trie = _update_trie(index, root, last_rev)
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44335 diff changeset	282 return (
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44335 diff changeset	283 changed_block * S_BLOCK.size,
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44335 diff changeset	284 _persist_trie(trie, existing_idx=max_idx),
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44335 diff changeset	285 )
44333 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	286
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	287
44307 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	288 S_BLOCK = struct.Struct(">" + ("l" * 16))
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	289
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	290 NO_ENTRY = -1
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	291 # rev 0 need to be -2 because 0 is used by block, -1 is a special value.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	292 REV_OFFSET = 2
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	293
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	294
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	295 def _transform_rev(rev):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	296 """Return the number used to represent the rev in the tree.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	297
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	298 (or retrieve a rev number from such representation)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	299
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	300 Note that this is an involution, a function equal to its inverse (i.e.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	301 which gives the identity when applied to itself).
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	302 """
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	303 return -(rev + REV_OFFSET)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	304
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	305
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	306 def _to_int(hex_digit):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	307 """turn an hexadecimal digit into a proper integer"""
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	308 return int(hex_digit, 16)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	309
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	310
44315 7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44314 diff changeset	311 class Block(dict):
7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44314 diff changeset	312 """represent a block of the Trie
7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44314 diff changeset	313
7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44314 diff changeset	314 contains up to 16 entry indexed from 0 to 15"""
7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44314 diff changeset	315
44321 f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44319 diff changeset	316 def __init__(self):
f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44319 diff changeset	317 super(Block, self).__init__()
f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44319 diff changeset	318 # If this block exist on disk, here is its ID
f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44319 diff changeset	319 self.ondisk_id = None
f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44319 diff changeset	320
44316 55b12f2593c1 nodemap: move the iteratio inside the Block object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44315 diff changeset	321 def __iter__(self):
55b12f2593c1 nodemap: move the iteratio inside the Block object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44315 diff changeset	322 return iter(self.get(i) for i in range(16))
55b12f2593c1 nodemap: move the iteratio inside the Block object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44315 diff changeset	323
44315 7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44314 diff changeset	324
44307 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	325 def _build_trie(index):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	326 """build a nodemap trie
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	327
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	328 The nodemap stores revision number for each unique prefix.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	329
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	330 Each block is a dictionary with keys in `[0, 15]`. Values are either
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	331 another block or a revision number.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	332 """
44315 7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44314 diff changeset	333 root = Block()
44307 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	334 for rev in range(len(index)):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	335 hex = nodemod.hex(index[rev][7])
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	336 _insert_into_block(index, 0, root, rev, hex)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	337 return root
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	338
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	339
44333 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	340 def _update_trie(index, root, last_rev):
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	341 """consume"""
44336 8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44335 diff changeset	342 changed = 0
44333 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	343 for rev in range(last_rev + 1, len(index)):
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	344 hex = nodemod.hex(index[rev][7])
44336 8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44335 diff changeset	345 changed += _insert_into_block(index, 0, root, rev, hex)
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44335 diff changeset	346 return changed, root
44333 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	347
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	348
44307 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	349 def _insert_into_block(index, level, block, current_rev, current_hex):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	350 """insert a new revision in a block
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	351
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	352 index: the index we are adding revision for
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	353 level: the depth of the current block in the trie
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	354 block: the block currently being considered
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	355 current_rev: the revision number we are adding
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	356 current_hex: the hexadecimal representation of the of that revision
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	357 """
44336 8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44335 diff changeset	358 changed = 1
44333 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	359 if block.ondisk_id is not None:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	360 block.ondisk_id = None
44307 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	361 hex_digit = _to_int(current_hex[level : level + 1])
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	362 entry = block.get(hex_digit)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	363 if entry is None:
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	364 # no entry, simply store the revision number
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	365 block[hex_digit] = current_rev
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	366 elif isinstance(entry, dict):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	367 # need to recurse to an underlying block
44336 8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44335 diff changeset	368 changed += _insert_into_block(
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44335 diff changeset	369 index, level + 1, entry, current_rev, current_hex
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44335 diff changeset	370 )
44307 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	371 else:
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	372 # collision with a previously unique prefix, inserting new
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	373 # vertices to fit both entry.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	374 other_hex = nodemod.hex(index[entry][7])
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	375 other_rev = entry
44315 7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44314 diff changeset	376 new = Block()
44307 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	377 block[hex_digit] = new
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	378 _insert_into_block(index, level + 1, new, other_rev, other_hex)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	379 _insert_into_block(index, level + 1, new, current_rev, current_hex)
44336 8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44335 diff changeset	380 return changed
44307 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	381
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	382
44333 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	383 def _persist_trie(root, existing_idx=None):
44307 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	384 """turn a nodemap trie into persistent binary data
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	385
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	386 See `_build_trie` for nodemap trie structure"""
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	387 block_map = {}
44333 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	388 if existing_idx is not None:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	389 base_idx = existing_idx + 1
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	390 else:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	391 base_idx = 0
44307 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	392 chunks = []
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	393 for tn in _walk_trie(root):
44333 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	394 if tn.ondisk_id is not None:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	395 block_map[id(tn)] = tn.ondisk_id
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	396 else:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	397 block_map[id(tn)] = len(chunks) + base_idx
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	398 chunks.append(_persist_block(tn, block_map))
44307 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	399 return b''.join(chunks)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	400
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	401
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	402 def _walk_trie(block):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	403 """yield all the block in a trie
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	404
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	405 Children blocks are always yield before their parent block.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	406 """
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	407 for (_, item) in sorted(block.items()):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	408 if isinstance(item, dict):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	409 for sub_block in _walk_trie(item):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	410 yield sub_block
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	411 yield block
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	412
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	413
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	414 def _persist_block(block_node, block_map):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	415 """produce persistent binary data for a single block
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	416
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	417 Children block are assumed to be already persisted and present in
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	418 block_map.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	419 """
44316 55b12f2593c1 nodemap: move the iteratio inside the Block object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44315 diff changeset	420 data = tuple(_to_value(v, block_map) for v in block_node)
44307 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	421 return S_BLOCK.pack(*data)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	422
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	423
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	424 def _to_value(item, block_map):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	425 """persist any value as an integer"""
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	426 if item is None:
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	427 return NO_ENTRY
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	428 elif isinstance(item, dict):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	429 return block_map[id(item)]
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	430 else:
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44034 diff changeset	431 return _transform_rev(item)
44317 78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44316 diff changeset	432
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44316 diff changeset	433
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44316 diff changeset	434 def parse_data(data):
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44316 diff changeset	435 """parse parse nodemap data into a nodemap Trie"""
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44316 diff changeset	436 if (len(data) % S_BLOCK.size) != 0:
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44316 diff changeset	437 msg = "nodemap data size is not a multiple of block size (%d): %d"
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44316 diff changeset	438 raise error.Abort(msg % (S_BLOCK.size, len(data)))
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44316 diff changeset	439 if not data:
44333 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	440 return Block(), None
44317 78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44316 diff changeset	441 block_map = {}
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44316 diff changeset	442 new_blocks = []
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44316 diff changeset	443 for i in range(0, len(data), S_BLOCK.size):
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44316 diff changeset	444 block = Block()
44321 f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44319 diff changeset	445 block.ondisk_id = len(block_map)
f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44319 diff changeset	446 block_map[block.ondisk_id] = block
44317 78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44316 diff changeset	447 block_data = data[i : i + S_BLOCK.size]
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44316 diff changeset	448 values = S_BLOCK.unpack(block_data)
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44316 diff changeset	449 new_blocks.append((block, values))
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44316 diff changeset	450 for b, values in new_blocks:
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44316 diff changeset	451 for idx, v in enumerate(values):
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44316 diff changeset	452 if v == NO_ENTRY:
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44316 diff changeset	453 continue
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44316 diff changeset	454 elif v >= 0:
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44316 diff changeset	455 b[idx] = block_map[v]
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44316 diff changeset	456 else:
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44316 diff changeset	457 b[idx] = _transform_rev(v)
44333 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	458 return block, i // S_BLOCK.size
44318 20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44317 diff changeset	459
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44317 diff changeset	460
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44317 diff changeset	461 # debug utility
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44317 diff changeset	462
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44317 diff changeset	463
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44317 diff changeset	464 def check_data(ui, index, data):
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44317 diff changeset	465 """verify that the provided nodemap data are valid for the given idex"""
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44317 diff changeset	466 ret = 0
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44317 diff changeset	467 ui.status((b"revision in index: %d\n") % len(index))
44333 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44323 diff changeset	468 root, __ = parse_data(data)
44318 20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44317 diff changeset	469 all_revs = set(_all_revisions(root))
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44317 diff changeset	470 ui.status((b"revision in nodemap: %d\n") % len(all_revs))
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44317 diff changeset	471 for r in range(len(index)):
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44317 diff changeset	472 if r not in all_revs:
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44317 diff changeset	473 msg = b" revision missing from nodemap: %d\n" % r
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44317 diff changeset	474 ui.write_err(msg)
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44317 diff changeset	475 ret = 1
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44317 diff changeset	476 else:
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44317 diff changeset	477 all_revs.remove(r)
44319 d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44318 diff changeset	478 nm_rev = _find_node(root, nodemod.hex(index[r][7]))
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44318 diff changeset	479 if nm_rev is None:
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44318 diff changeset	480 msg = b" revision node does not match any entries: %d\n" % r
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44318 diff changeset	481 ui.write_err(msg)
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44318 diff changeset	482 ret = 1
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44318 diff changeset	483 elif nm_rev != r:
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44318 diff changeset	484 msg = (
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44318 diff changeset	485 b" revision node does not match the expected revision: "
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44318 diff changeset	486 b"%d != %d\n" % (r, nm_rev)
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44318 diff changeset	487 )
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44318 diff changeset	488 ui.write_err(msg)
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44318 diff changeset	489 ret = 1
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44318 diff changeset	490
44318 20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44317 diff changeset	491 if all_revs:
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44317 diff changeset	492 for r in sorted(all_revs):
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44317 diff changeset	493 msg = b" extra revision in nodemap: %d\n" % r
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44317 diff changeset	494 ui.write_err(msg)
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44317 diff changeset	495 ret = 1
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44317 diff changeset	496 return ret
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44317 diff changeset	497
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44317 diff changeset	498
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44317 diff changeset	499 def _all_revisions(root):
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44317 diff changeset	500 """return all revisions stored in a Trie"""
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44317 diff changeset	501 for block in _walk_trie(root):
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44317 diff changeset	502 for v in block:
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44317 diff changeset	503 if v is None or isinstance(v, Block):
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44317 diff changeset	504 continue
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44317 diff changeset	505 yield v
44319 d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44318 diff changeset	506
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44318 diff changeset	507
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44318 diff changeset	508 def _find_node(block, node):
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44318 diff changeset	509 """find the revision associated with a given node"""
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44318 diff changeset	510 entry = block.get(_to_int(node[0:1]))
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44318 diff changeset	511 if isinstance(entry, dict):
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44318 diff changeset	512 return _find_node(entry, node[1:])
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44318 diff changeset	513 return entry

Mercurial > public > mercurial-scm > hg

annotate mercurial/revlogutils/nodemap.py @ 44339:c7eebdb15139