mercurial-scm/hg-stable: mercurial/revlogutils/nodemap.py annotate

annotate mercurial/revlogutils/nodemap.py @ 44374:6ecc34b31137

nodemap: update the index with the newly written data (when appropriate) If we are to use mmap to read the nodemap data, and if the python code is responsible for the IO, we need to refresh the mmap after each write and provide it back to the index. We start this dance without the mmap first. Differential Revision: https://phab.mercurial-scm.org/D7893

author	Pierre-Yves David <pierre-yves.david@octobus.net>
date	Wed, 15 Jan 2020 15:51:01 +0100
parents	c7eebdb15139
children	f7459da77f23

rev	line source
44035 ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	1 # nodemap.py - nodemap related code and utilities
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	2 #
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	3 # Copyright 2019 Pierre-Yves David <pierre-yves.david@octobus.net>
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	4 # Copyright 2019 George Racinet <georges.racinet@octobus.net>
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	5 #
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	6 # This software may be used and distributed according to the terms of the
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	7 # GNU General Public License version 2 or any later version.
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	8
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	9 from __future__ import absolute_import
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	10
44354 2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	11 import os
44355 563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	12 import re
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	13 import struct
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	14
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	15 from .. import (
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	16 error,
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	17 node as nodemod,
44357 7f4f7ef3133e nodemap: add a optional `nodemap_add_full` method on indexes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44355 diff changeset	18 util,
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	19 )
44035 ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	20
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	21
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	22 class NodeMap(dict):
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	23 def __missing__(self, x):
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module Pierre-Yves David <pierre-yves.david@octobus.net> parents: diff changeset	24 raise error.RevlogError(b'unknown node: %s' % x)
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	25
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	26
44352 6c07480d6659 nodemap: add a function to read the data from disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44351 diff changeset	27 def persisted_data(revlog):
6c07480d6659 nodemap: add a function to read the data from disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44351 diff changeset	28 """read the nodemap for a revlog from disk"""
6c07480d6659 nodemap: add a function to read the data from disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44351 diff changeset	29 if revlog.nodemap_file is None:
6c07480d6659 nodemap: add a function to read the data from disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44351 diff changeset	30 return None
44354 2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	31 pdata = revlog.opener.tryread(revlog.nodemap_file)
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	32 if not pdata:
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	33 return None
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	34 offset = 0
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	35 (version,) = S_VERSION.unpack(pdata[offset : offset + S_VERSION.size])
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	36 if version != ONDISK_VERSION:
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	37 return None
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	38 offset += S_VERSION.size
44369 e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44367 diff changeset	39 headers = S_HEADER.unpack(pdata[offset : offset + S_HEADER.size])
44370 8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	40 uid_size, tip_rev, data_length, data_unused = headers
44354 2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	41 offset += S_HEADER.size
44365 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	42 docket = NodeMapDocket(pdata[offset : offset + uid_size])
44369 e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44367 diff changeset	43 docket.tip_rev = tip_rev
44370 8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	44 docket.data_length = data_length
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	45 docket.data_unused = data_unused
44354 2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	46
44365 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	47 filename = _rawdata_filepath(revlog, docket)
44373 c7eebdb15139 nodemap: never read more than the expected data amount Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44372 diff changeset	48 data = revlog.opener.tryread(filename)
c7eebdb15139 nodemap: never read more than the expected data amount Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44372 diff changeset	49 if len(data) < data_length:
c7eebdb15139 nodemap: never read more than the expected data amount Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44372 diff changeset	50 return None
c7eebdb15139 nodemap: never read more than the expected data amount Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44372 diff changeset	51 elif len(data) > data_length:
c7eebdb15139 nodemap: never read more than the expected data amount Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44372 diff changeset	52 data = data[:data_length]
c7eebdb15139 nodemap: never read more than the expected data amount Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44372 diff changeset	53 return docket, data
44352 6c07480d6659 nodemap: add a function to read the data from disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44351 diff changeset	54
6c07480d6659 nodemap: add a function to read the data from disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44351 diff changeset	55
44351 5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	56 def setup_persistent_nodemap(tr, revlog):
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	57 """Install whatever is needed transaction side to persist a nodemap on disk
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	58
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	59 (only actually persist the nodemap if this is relevant for this revlog)
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	60 """
44353 daad3aace942 nodemap: only use persistent nodemap for non-inlined revlog Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44352 diff changeset	61 if revlog._inline:
daad3aace942 nodemap: only use persistent nodemap for non-inlined revlog Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44352 diff changeset	62 return # inlined revlog are too small for this to be relevant
44351 5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	63 if revlog.nodemap_file is None:
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	64 return # we do not use persistent_nodemap on this revlog
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	65 callback_id = b"revlog-persistent-nodemap-%s" % revlog.nodemap_file
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	66 if tr.hasfinalize(callback_id):
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	67 return # no need to register again
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	68 tr.addfinalize(callback_id, lambda tr: _persist_nodemap(tr, revlog))
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	69
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	70
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	71 def _persist_nodemap(tr, revlog):
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	72 """Write nodemap data on disk for a given revlog
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	73 """
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	74 if getattr(revlog, 'filteredrevs', ()):
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	75 raise error.ProgrammingError(
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	76 "cannot persist nodemap of a filtered changelog"
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	77 )
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	78 if revlog.nodemap_file is None:
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	79 msg = "calling persist nodemap on a revlog without the feature enableb"
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	80 raise error.ProgrammingError(msg)
44367 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	81
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	82 can_incremental = util.safehasattr(revlog.index, "nodemap_data_incremental")
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	83 ondisk_docket = revlog._nodemap_docket
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	84
44371 1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	85 data = None
44367 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	86 # first attemp an incremental update of the data
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	87 if can_incremental and ondisk_docket is not None:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	88 target_docket = revlog._nodemap_docket.copy()
44371 1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	89 (
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	90 src_docket,
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	91 data_changed_count,
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	92 data,
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	93 ) = revlog.index.nodemap_data_incremental()
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	94 if src_docket != target_docket:
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	95 data = None
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	96 else:
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	97 datafile = _rawdata_filepath(revlog, target_docket)
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	98 # EXP-TODO: if this is a cache, this should use a cache vfs, not a
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	99 # store vfs
44372 2ea6a67ff502 nodemap: write new data from the expected current data length Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44371 diff changeset	100 with revlog.opener(datafile, b'r+') as fd:
2ea6a67ff502 nodemap: write new data from the expected current data length Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44371 diff changeset	101 fd.seek(target_docket.data_length)
44371 1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	102 fd.write(data)
44374 6ecc34b31137 nodemap: update the index with the newly written data (when appropriate) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44373 diff changeset	103 fd.seek(0)
6ecc34b31137 nodemap: update the index with the newly written data (when appropriate) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44373 diff changeset	104 new_data = fd.read(target_docket.data_length + len(data))
44371 1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	105 target_docket.data_length += len(data)
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	106 target_docket.data_unused += data_changed_count
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	107
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	108 if data is None:
44367 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	109 # otherwise fallback to a full new export
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	110 target_docket = NodeMapDocket()
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	111 datafile = _rawdata_filepath(revlog, target_docket)
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	112 if util.safehasattr(revlog.index, "nodemap_data_all"):
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	113 data = revlog.index.nodemap_data_all()
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	114 else:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	115 data = persistent_data(revlog.index)
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	116 # EXP-TODO: if this is a cache, this should use a cache vfs, not a
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	117 # store vfs
44374 6ecc34b31137 nodemap: update the index with the newly written data (when appropriate) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44373 diff changeset	118 new_data = data
44367 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	119 with revlog.opener(datafile, b'w') as fd:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	120 fd.write(data)
44370 8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	121 target_docket.data_length = len(data)
44369 e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44367 diff changeset	122 target_docket.tip_rev = revlog.tiprev()
44367 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	123 # EXP-TODO: if this is a cache, this should use a cache vfs, not a
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	124 # store vfs
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	125 with revlog.opener(revlog.nodemap_file, b'w', atomictemp=True) as fp:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	126 fp.write(target_docket.serialize())
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	127 revlog._nodemap_docket = target_docket
44374 6ecc34b31137 nodemap: update the index with the newly written data (when appropriate) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44373 diff changeset	128 if util.safehasattr(revlog.index, "update_nodemap_data"):
6ecc34b31137 nodemap: update the index with the newly written data (when appropriate) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44373 diff changeset	129 revlog.index.update_nodemap_data(target_docket, new_data)
6ecc34b31137 nodemap: update the index with the newly written data (when appropriate) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44373 diff changeset	130
44367 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	131 # EXP-TODO: if the transaction abort, we should remove the new data and
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	132 # reinstall the old one.
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	133
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	134 # search for old index file in all cases, some older process might have
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	135 # left one behind.
44365 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	136 olds = _other_rawdata_filepath(revlog, target_docket)
44355 563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	137 if olds:
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	138 realvfs = getattr(revlog, '_realopener', revlog.opener)
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	139
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	140 def cleanup(tr):
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	141 for oldfile in olds:
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	142 realvfs.tryunlink(oldfile)
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	143
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	144 callback_id = b"revlog-cleanup-nodemap-%s" % revlog.nodemap_file
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	145 tr.addpostclose(callback_id, cleanup)
44354 2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	146
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	147
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	148 ### Nodemap docket file
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	149 #
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	150 # The nodemap data are stored on disk using 2 files:
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	151 #
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	152 # * a raw data files containing a persistent nodemap
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	153 # (see `Nodemap Trie` section)
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	154 #
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	155 # * a small "docket" file containing medatadata
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	156 #
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	157 # While the nodemap data can be multiple tens of megabytes, the "docket" is
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	158 # small, it is easy to update it automatically or to duplicated its content
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	159 # during a transaction.
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	160 #
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	161 # Multiple raw data can exist at the same time (The currently valid one and a
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	162 # new one beind used by an in progress transaction). To accomodate this, the
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	163 # filename hosting the raw data has a variable parts. The exact filename is
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	164 # specified inside the "docket" file.
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	165 #
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	166 # The docket file contains information to find, qualify and validate the raw
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	167 # data. Its content is currently very light, but it will expand as the on disk
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	168 # nodemap gains the necessary features to be used in production.
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	169
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	170 # version 0 is experimental, no BC garantee, do no use outside of tests.
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	171 ONDISK_VERSION = 0
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	172 S_VERSION = struct.Struct(">B")
44370 8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	173 S_HEADER = struct.Struct(">BQQQ")
44354 2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	174
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	175 ID_SIZE = 8
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	176
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	177
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	178 def _make_uid():
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	179 """return a new unique identifier.
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	180
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	181 The identifier is random and composed of ascii characters."""
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	182 return nodemod.hex(os.urandom(ID_SIZE))
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	183
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	184
44365 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	185 class NodeMapDocket(object):
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	186 """metadata associated with persistent nodemap data
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	187
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	188 The persistent data may come from disk or be on their way to disk.
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	189 """
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	190
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	191 def __init__(self, uid=None):
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	192 if uid is None:
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	193 uid = _make_uid()
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	194 self.uid = uid
44369 e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44367 diff changeset	195 self.tip_rev = None
44370 8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	196 self.data_length = None
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	197 self.data_unused = 0
44365 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	198
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	199 def copy(self):
44369 e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44367 diff changeset	200 new = NodeMapDocket(uid=self.uid)
e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44367 diff changeset	201 new.tip_rev = self.tip_rev
44370 8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	202 new.data_length = self.data_length
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	203 new.data_unused = self.data_unused
44369 e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44367 diff changeset	204 return new
44365 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	205
44371 1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	206 def __cmp__(self, other):
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	207 if self.uid < other.uid:
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	208 return -1
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	209 if self.uid > other.uid:
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	210 return 1
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	211 elif self.data_length < other.data_length:
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	212 return -1
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	213 elif self.data_length > other.data_length:
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	214 return 1
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	215 return 0
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	216
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	217 def __eq__(self, other):
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	218 return self.uid == other.uid and self.data_length == other.data_length
1d2b37def017 nodemap: double check the source docket when doing incremental update Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44370 diff changeset	219
44365 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	220 def serialize(self):
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	221 """return serialized bytes for a docket using the passed uid"""
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	222 data = []
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	223 data.append(S_VERSION.pack(ONDISK_VERSION))
44370 8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	224 headers = (
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	225 len(self.uid),
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	226 self.tip_rev,
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	227 self.data_length,
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	228 self.data_unused,
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	229 )
44369 e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44367 diff changeset	230 data.append(S_HEADER.pack(*headers))
44365 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	231 data.append(self.uid)
72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	232 return b''.join(data)
44354 2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	233
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	234
44365 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	235 def _rawdata_filepath(revlog, docket):
44354 2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	236 """The (vfs relative) nodemap's rawdata file for a given uid"""
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44353 diff changeset	237 prefix = revlog.nodemap_file[:-2]
44365 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	238 return b"%s-%s.nd" % (prefix, docket.uid)
44351 5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	239
5962fd0d1045 nodemap: write nodemap data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44350 diff changeset	240
44365 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	241 def _other_rawdata_filepath(revlog, docket):
44355 563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	242 prefix = revlog.nodemap_file[:-2]
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	243 pattern = re.compile(b"(^\|/)%s-[0-9a-f]+\.nd$" % prefix)
44365 72c15641c8b4 nodemap: introduce an explicit class/object for the docket Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44364 diff changeset	244 new_file_path = _rawdata_filepath(revlog, docket)
44355 563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	245 new_file_name = revlog.opener.basename(new_file_path)
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	246 dirpath = revlog.opener.dirname(new_file_path)
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	247 others = []
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	248 for f in revlog.opener.listdir(dirpath):
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	249 if pattern.match(f) and f != new_file_name:
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	250 others.append(f)
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	251 return others
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	252
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44354 diff changeset	253
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	254 ### Nodemap Trie
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	255 #
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	256 # This is a simple reference implementation to compute and persist a nodemap
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	257 # trie. This reference implementation is write only. The python version of this
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	258 # is not expected to be actually used, since it wont provide performance
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	259 # improvement over existing non-persistent C implementation.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	260 #
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	261 # The nodemap is persisted as Trie using 4bits-address/16-entries block. each
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	262 # revision can be adressed using its node shortest prefix.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	263 #
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	264 # The trie is stored as a sequence of block. Each block contains 16 entries
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	265 # (signed 64bit integer, big endian). Each entry can be one of the following:
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	266 #
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	267 # * value >= 0 -> index of sub-block
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	268 # * value == -1 -> no value
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	269 # * value < -1 -> a revision value: rev = -(value+10)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	270 #
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	271 # The implementation focus on simplicity, not on performance. A Rust
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	272 # implementation should provide a efficient version of the same binary
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	273 # persistence. This reference python implementation is never meant to be
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	274 # extensively use in production.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	275
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	276
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	277 def persistent_data(index):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	278 """return the persistent binary form for a nodemap for a given index
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	279 """
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	280 trie = _build_trie(index)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	281 return _persist_trie(trie)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	282
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	283
44367 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	284 def update_persistent_data(index, root, max_idx, last_rev):
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	285 """return the incremental update for persistent nodemap from a given index
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	286 """
44370 8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	287 changed_block, trie = _update_trie(index, root, last_rev)
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	288 return (
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	289 changed_block * S_BLOCK.size,
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	290 _persist_trie(trie, existing_idx=max_idx),
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	291 )
44367 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	292
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	293
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	294 S_BLOCK = struct.Struct(">" + ("l" * 16))
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	295
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	296 NO_ENTRY = -1
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	297 # rev 0 need to be -2 because 0 is used by block, -1 is a special value.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	298 REV_OFFSET = 2
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	299
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	300
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	301 def _transform_rev(rev):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	302 """Return the number used to represent the rev in the tree.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	303
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	304 (or retrieve a rev number from such representation)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	305
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	306 Note that this is an involution, a function equal to its inverse (i.e.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	307 which gives the identity when applied to itself).
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	308 """
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	309 return -(rev + REV_OFFSET)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	310
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	311
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	312 def _to_int(hex_digit):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	313 """turn an hexadecimal digit into a proper integer"""
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	314 return int(hex_digit, 16)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	315
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	316
44358 7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44357 diff changeset	317 class Block(dict):
7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44357 diff changeset	318 """represent a block of the Trie
7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44357 diff changeset	319
7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44357 diff changeset	320 contains up to 16 entry indexed from 0 to 15"""
7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44357 diff changeset	321
44364 f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44362 diff changeset	322 def __init__(self):
f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44362 diff changeset	323 super(Block, self).__init__()
f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44362 diff changeset	324 # If this block exist on disk, here is its ID
f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44362 diff changeset	325 self.ondisk_id = None
f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44362 diff changeset	326
44359 55b12f2593c1 nodemap: move the iteratio inside the Block object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44358 diff changeset	327 def __iter__(self):
55b12f2593c1 nodemap: move the iteratio inside the Block object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44358 diff changeset	328 return iter(self.get(i) for i in range(16))
55b12f2593c1 nodemap: move the iteratio inside the Block object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44358 diff changeset	329
44358 7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44357 diff changeset	330
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	331 def _build_trie(index):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	332 """build a nodemap trie
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	333
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	334 The nodemap stores revision number for each unique prefix.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	335
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	336 Each block is a dictionary with keys in `[0, 15]`. Values are either
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	337 another block or a revision number.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	338 """
44358 7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44357 diff changeset	339 root = Block()
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	340 for rev in range(len(index)):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	341 hex = nodemod.hex(index[rev][7])
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	342 _insert_into_block(index, 0, root, rev, hex)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	343 return root
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	344
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	345
44367 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	346 def _update_trie(index, root, last_rev):
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	347 """consume"""
44370 8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	348 changed = 0
44367 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	349 for rev in range(last_rev + 1, len(index)):
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	350 hex = nodemod.hex(index[rev][7])
44370 8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	351 changed += _insert_into_block(index, 0, root, rev, hex)
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	352 return changed, root
44367 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	353
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	354
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	355 def _insert_into_block(index, level, block, current_rev, current_hex):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	356 """insert a new revision in a block
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	357
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	358 index: the index we are adding revision for
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	359 level: the depth of the current block in the trie
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	360 block: the block currently being considered
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	361 current_rev: the revision number we are adding
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	362 current_hex: the hexadecimal representation of the of that revision
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	363 """
44370 8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	364 changed = 1
44367 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	365 if block.ondisk_id is not None:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	366 block.ondisk_id = None
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	367 hex_digit = _to_int(current_hex[level : level + 1])
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	368 entry = block.get(hex_digit)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	369 if entry is None:
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	370 # no entry, simply store the revision number
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	371 block[hex_digit] = current_rev
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	372 elif isinstance(entry, dict):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	373 # need to recurse to an underlying block
44370 8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	374 changed += _insert_into_block(
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	375 index, level + 1, entry, current_rev, current_hex
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	376 )
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	377 else:
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	378 # collision with a previously unique prefix, inserting new
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	379 # vertices to fit both entry.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	380 other_hex = nodemod.hex(index[entry][7])
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	381 other_rev = entry
44358 7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44357 diff changeset	382 new = Block()
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	383 block[hex_digit] = new
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	384 _insert_into_block(index, level + 1, new, other_rev, other_hex)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	385 _insert_into_block(index, level + 1, new, current_rev, current_hex)
44370 8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44369 diff changeset	386 return changed
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	387
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	388
44367 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	389 def _persist_trie(root, existing_idx=None):
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	390 """turn a nodemap trie into persistent binary data
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	391
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	392 See `_build_trie` for nodemap trie structure"""
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	393 block_map = {}
44367 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	394 if existing_idx is not None:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	395 base_idx = existing_idx + 1
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	396 else:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	397 base_idx = 0
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	398 chunks = []
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	399 for tn in _walk_trie(root):
44367 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	400 if tn.ondisk_id is not None:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	401 block_map[id(tn)] = tn.ondisk_id
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	402 else:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	403 block_map[id(tn)] = len(chunks) + base_idx
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	404 chunks.append(_persist_block(tn, block_map))
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	405 return b''.join(chunks)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	406
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	407
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	408 def _walk_trie(block):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	409 """yield all the block in a trie
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	410
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	411 Children blocks are always yield before their parent block.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	412 """
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	413 for (_, item) in sorted(block.items()):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	414 if isinstance(item, dict):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	415 for sub_block in _walk_trie(item):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	416 yield sub_block
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	417 yield block
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	418
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	419
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	420 def _persist_block(block_node, block_map):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	421 """produce persistent binary data for a single block
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	422
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	423 Children block are assumed to be already persisted and present in
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	424 block_map.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	425 """
44359 55b12f2593c1 nodemap: move the iteratio inside the Block object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44358 diff changeset	426 data = tuple(_to_value(v, block_map) for v in block_node)
44350 c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	427 return S_BLOCK.pack(*data)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	428
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	429
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	430 def _to_value(item, block_map):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	431 """persist any value as an integer"""
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	432 if item is None:
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	433 return NO_ENTRY
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	434 elif isinstance(item, dict):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	435 return block_map[id(item)]
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	436 else:
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44035 diff changeset	437 return _transform_rev(item)
44360 78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	438
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	439
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	440 def parse_data(data):
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	441 """parse parse nodemap data into a nodemap Trie"""
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	442 if (len(data) % S_BLOCK.size) != 0:
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	443 msg = "nodemap data size is not a multiple of block size (%d): %d"
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	444 raise error.Abort(msg % (S_BLOCK.size, len(data)))
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	445 if not data:
44367 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	446 return Block(), None
44360 78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	447 block_map = {}
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	448 new_blocks = []
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	449 for i in range(0, len(data), S_BLOCK.size):
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	450 block = Block()
44364 f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44362 diff changeset	451 block.ondisk_id = len(block_map)
f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44362 diff changeset	452 block_map[block.ondisk_id] = block
44360 78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	453 block_data = data[i : i + S_BLOCK.size]
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	454 values = S_BLOCK.unpack(block_data)
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	455 new_blocks.append((block, values))
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	456 for b, values in new_blocks:
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	457 for idx, v in enumerate(values):
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	458 if v == NO_ENTRY:
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	459 continue
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	460 elif v >= 0:
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	461 b[idx] = block_map[v]
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	462 else:
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44359 diff changeset	463 b[idx] = _transform_rev(v)
44367 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	464 return block, i // S_BLOCK.size
44361 20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	465
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	466
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	467 # debug utility
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	468
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	469
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	470 def check_data(ui, index, data):
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	471 """verify that the provided nodemap data are valid for the given idex"""
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	472 ret = 0
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	473 ui.status((b"revision in index: %d\n") % len(index))
44367 50ad851efd9b nodemap: introduce append-only incremental update of the persistent data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44366 diff changeset	474 root, __ = parse_data(data)
44361 20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	475 all_revs = set(_all_revisions(root))
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	476 ui.status((b"revision in nodemap: %d\n") % len(all_revs))
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	477 for r in range(len(index)):
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	478 if r not in all_revs:
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	479 msg = b" revision missing from nodemap: %d\n" % r
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	480 ui.write_err(msg)
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	481 ret = 1
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	482 else:
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	483 all_revs.remove(r)
44362 d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	484 nm_rev = _find_node(root, nodemod.hex(index[r][7]))
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	485 if nm_rev is None:
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	486 msg = b" revision node does not match any entries: %d\n" % r
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	487 ui.write_err(msg)
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	488 ret = 1
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	489 elif nm_rev != r:
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	490 msg = (
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	491 b" revision node does not match the expected revision: "
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	492 b"%d != %d\n" % (r, nm_rev)
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	493 )
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	494 ui.write_err(msg)
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	495 ret = 1
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	496
44361 20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	497 if all_revs:
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	498 for r in sorted(all_revs):
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	499 msg = b" extra revision in nodemap: %d\n" % r
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	500 ui.write_err(msg)
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	501 ret = 1
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	502 return ret
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	503
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	504
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	505 def _all_revisions(root):
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	506 """return all revisions stored in a Trie"""
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	507 for block in _walk_trie(root):
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	508 for v in block:
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	509 if v is None or isinstance(v, Block):
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	510 continue
20e125cdd719 nodemap: add basic checking of the on disk nodemap content Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44360 diff changeset	511 yield v
44362 d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	512
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	513
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	514 def _find_node(block, node):
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	515 """find the revision associated with a given node"""
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	516 entry = block.get(_to_int(node[0:1]))
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	517 if isinstance(entry, dict):
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	518 return _find_node(entry, node[1:])
d58206b70199 nodemap: all check that revision and nodes match in the nodemap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 44361 diff changeset	519 return entry

Mercurial > public > mercurial-scm > hg-stable

annotate mercurial/revlogutils/nodemap.py @ 44374:6ecc34b31137