Mercurial > public > mercurial-scm > hg-stable

annotate mercurial/revlogutils/sidedata.py @ 43036:e8bc4c3d9a0b

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
sidedata: add a function for _processflagsraw usage Entry with side data cannot directly validate their rawtext Differential Revision: https://phab.mercurial-scm.org/D6892
author Pierre-Yves David <pierre-yves.david@octobus.net>
date Wed, 04 Sep 2019 03:19:58 +0200
parents ea83abf95630
children 142deb539ccf
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
43033
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
1 # sidedata.py - Logic around store extra data alongside revlog revisions
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
2 #
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
3 # Copyright 2019 Pierre-Yves David <pierre-yves.david@octobus.net)
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
4 #
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
5 # This software may be used and distributed according to the terms of the
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
6 # GNU General Public License version 2 or any later version.
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
7 """core code for "sidedata" support
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
8
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
9 The "sidedata" are stored alongside the revision without actually being part of
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
10 its content and not affecting its hash. It's main use cases is to cache
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
11 important information related to a changesets.
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
12
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
13 The current implementation is experimental and subject to changes. Do not rely
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
14 on it in production.
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
15
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
16 Sidedata are stored in the revlog itself, withing the revision rawtext. They
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
17 are inserted, removed from it using the flagprocessors mechanism. The following
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
18 format is currently used::
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
19
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
20 initial header:
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
21 <number of sidedata; 2 bytes>
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
22 sidedata (repeated N times):
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
23 <sidedata-key; 2 bytes>
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
24 <sidedata-entry-length: 4 bytes>
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
25 <sidedata-content-sha1-digest: 20 bytes>
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
26 <sidedata-content; X bytes>
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
27 normal raw text:
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
28 <all bytes remaining in the rawtext>
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
29
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
30 This is a simple and effective format. It should be enought to experiment with
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
31 the concept.
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
32 """
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
33
21025a4107d4 sidedata: add a new module with basic documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
34 from __future__ import absolute_import
43034
294afb982a88 sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43033
diff changeset
35
294afb982a88 sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43033
diff changeset
36 import hashlib
294afb982a88 sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43033
diff changeset
37 import struct
294afb982a88 sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43033
diff changeset
38
294afb982a88 sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43033
diff changeset
39 from .. import error
294afb982a88 sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43033
diff changeset
40
294afb982a88 sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43033
diff changeset
41 SIDEDATA_HEADER = struct.Struct('>H')
294afb982a88 sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43033
diff changeset
42 SIDEDATA_ENTRY = struct.Struct('>HL20s')
294afb982a88 sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43033
diff changeset
43
43035
ea83abf95630 sidedata: add a function to write sidedata into a raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43034
diff changeset
44 def sidedatawriteprocessor(rl, text, sidedata):
ea83abf95630 sidedata: add a function to write sidedata into a raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43034
diff changeset
45 sidedata = list(sidedata.items())
ea83abf95630 sidedata: add a function to write sidedata into a raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43034
diff changeset
46 sidedata.sort()
ea83abf95630 sidedata: add a function to write sidedata into a raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43034
diff changeset
47 rawtext = [SIDEDATA_HEADER.pack(len(sidedata))]
ea83abf95630 sidedata: add a function to write sidedata into a raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43034
diff changeset
48 for key, value in sidedata:
ea83abf95630 sidedata: add a function to write sidedata into a raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43034
diff changeset
49 digest = hashlib.sha1(value).digest()
ea83abf95630 sidedata: add a function to write sidedata into a raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43034
diff changeset
50 rawtext.append(SIDEDATA_ENTRY.pack(key, len(value), digest))
ea83abf95630 sidedata: add a function to write sidedata into a raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43034
diff changeset
51 for key, value in sidedata:
ea83abf95630 sidedata: add a function to write sidedata into a raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43034
diff changeset
52 rawtext.append(value)
ea83abf95630 sidedata: add a function to write sidedata into a raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43034
diff changeset
53 rawtext.append(bytes(text))
ea83abf95630 sidedata: add a function to write sidedata into a raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43034
diff changeset
54 return ''.join(rawtext), False
ea83abf95630 sidedata: add a function to write sidedata into a raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43034
diff changeset
55
43034
294afb982a88 sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43033
diff changeset
56 def sidedatareadprocessor(rl, text):
294afb982a88 sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43033
diff changeset
57 sidedata = {}
294afb982a88 sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43033
diff changeset
58 offset = 0
294afb982a88 sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43033
diff changeset
59 nbentry, = SIDEDATA_HEADER.unpack(text[:SIDEDATA_HEADER.size])
294afb982a88 sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43033
diff changeset
60 offset += SIDEDATA_HEADER.size
294afb982a88 sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43033
diff changeset
61 dataoffset = SIDEDATA_HEADER.size + (SIDEDATA_ENTRY.size * nbentry)
294afb982a88 sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43033
diff changeset
62 for i in range(nbentry):
294afb982a88 sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43033
diff changeset
63 nextoffset = offset + SIDEDATA_ENTRY.size
294afb982a88 sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43033
diff changeset
64 key, size, storeddigest = SIDEDATA_ENTRY.unpack(text[offset:nextoffset])
294afb982a88 sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43033
diff changeset
65 offset = nextoffset
294afb982a88 sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43033
diff changeset
66 # read the data associated with that entry
294afb982a88 sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43033
diff changeset
67 nextdataoffset = dataoffset + size
294afb982a88 sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43033
diff changeset
68 entrytext = text[dataoffset:nextdataoffset]
294afb982a88 sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43033
diff changeset
69 readdigest = hashlib.sha1(entrytext).digest()
294afb982a88 sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43033
diff changeset
70 if storeddigest != readdigest:
294afb982a88 sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43033
diff changeset
71 raise error.SidedataHashError(key, storeddigest, readdigest)
294afb982a88 sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43033
diff changeset
72 sidedata[key] = entrytext
294afb982a88 sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43033
diff changeset
73 dataoffset = nextdataoffset
294afb982a88 sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43033
diff changeset
74 text = text[dataoffset:]
294afb982a88 sidedata: add a function to read sidedata from revlog raw text
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43033
diff changeset
75 return text, True, sidedata
43036
e8bc4c3d9a0b sidedata: add a function for _processflagsraw usage
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43035
diff changeset
76
e8bc4c3d9a0b sidedata: add a function for _processflagsraw usage
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43035
diff changeset
77 def sidedatarawprocessor(rl, text):
e8bc4c3d9a0b sidedata: add a function for _processflagsraw usage
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43035
diff changeset
78 # side data modifies rawtext and prevent rawtext hash validation
e8bc4c3d9a0b sidedata: add a function for _processflagsraw usage
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43035
diff changeset
79 return False