Mercurial > public > mercurial-scm > hg

annotate mercurial/revlogutils/randomaccessfile.py @ 51103:d83d788590a8

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
changelog-delay: move the delay/divert logic inside the (inner) revlog Instead of hacking throught the vfs/opener, we implement the delay/divert logic inside the `_InnerRevlog` and `randomaccessfile` object. This will allow to an alternative implementation of the `_InnerRevlog` that does not need to use Python details. As a result, the new implementation can use the transaction less agressively and avoid some extra output since no data had been written yet. That seems like a good side effect.
author Pierre-Yves David <pierre-yves.david@octobus.net>
date Tue, 24 Oct 2023 11:08:49 +0200
parents 222b89224397
children a93e52f0b6ff
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
47425
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
1 # Copyright Mercurial Contributors
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
2 #
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
3 # This software may be used and distributed according to the terms of the
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
4 # GNU General Public License version 2 or any later version.
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
5
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
6 import contextlib
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
7
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
8 from ..i18n import _
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
9 from .. import (
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
10 error,
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
11 util,
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
12 )
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
13
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
14
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
15 _MAX_CACHED_CHUNK_SIZE = 1048576 # 1 MiB
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
16
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
17 PARTIAL_READ_MSG = _(
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
18 b'partial read of revlog %s; expected %d bytes from offset %d, got %d'
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
19 )
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
20
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
21
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
22 def _is_power_of_two(n):
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
23 return (n & (n - 1) == 0) and n != 0
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
24
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
25
51100
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
26 class appender:
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
27 """the changelog index must be updated last on disk, so we use this class
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
28 to delay writes to it"""
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
29
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
30 def __init__(self, vfs, name, mode, buf):
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
31 self.data = buf
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
32 fp = vfs(name, mode)
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
33 self.fp = fp
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
34 self.offset = fp.tell()
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
35 self.size = vfs.fstat(fp).st_size
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
36 self._end = self.size
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
37
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
38 def end(self):
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
39 return self._end
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
40
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
41 def tell(self):
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
42 return self.offset
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
43
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
44 def flush(self):
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
45 pass
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
46
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
47 @property
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
48 def closed(self):
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
49 return self.fp.closed
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
50
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
51 def close(self):
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
52 self.fp.close()
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
53
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
54 def seek(self, offset, whence=0):
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
55 '''virtual file offset spans real file and data'''
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
56 if whence == 0:
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
57 self.offset = offset
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
58 elif whence == 1:
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
59 self.offset += offset
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
60 elif whence == 2:
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
61 self.offset = self.end() + offset
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
62 if self.offset < self.size:
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
63 self.fp.seek(self.offset)
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
64
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
65 def read(self, count=-1):
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
66 '''only trick here is reads that span real file and data'''
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
67 ret = b""
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
68 if self.offset < self.size:
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
69 s = self.fp.read(count)
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
70 ret = s
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
71 self.offset += len(s)
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
72 if count > 0:
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
73 count -= len(s)
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
74 if count != 0:
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
75 doff = self.offset - self.size
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
76 self.data.insert(0, b"".join(self.data))
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
77 del self.data[1:]
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
78 s = self.data[0][doff : doff + count]
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
79 self.offset += len(s)
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
80 ret += s
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
81 return ret
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
82
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
83 def write(self, s):
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
84 self.data.append(bytes(s))
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
85 self.offset += len(s)
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
86 self._end += len(s)
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
87
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
88 def __enter__(self):
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
89 self.fp.__enter__()
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
90 return self
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
91
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
92 def __exit__(self, *args):
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
93 return self.fp.__exit__(*args)
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
94
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
95
48946
642e31cb55f0 py3: use class X: instead of class X(object):
Gregory Szorc <gregory.szorc@gmail.com>
parents: 47463
diff changeset
96 class randomaccessfile:
47425
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
97 """Accessing arbitrary chuncks of data within a file, with some caching"""
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
98
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
99 def __init__(
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
100 self,
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
101 opener,
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
102 filename,
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
103 default_cached_chunk_size,
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
104 initial_cache=None,
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
105 ):
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
106 # Required by bitwise manipulation below
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
107 assert _is_power_of_two(default_cached_chunk_size)
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
108
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
109 self.opener = opener
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
110 self.filename = filename
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
111 self.default_cached_chunk_size = default_cached_chunk_size
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
112 self.writing_handle = None # This is set from revlog.py
47463
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
113 self.reading_handle = None
47425
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
114 self._cached_chunk = b''
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
115 self._cached_chunk_position = 0 # Offset from the start of the file
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
116 if initial_cache:
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
117 self._cached_chunk_position, self._cached_chunk = initial_cache
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
118
51103
d83d788590a8 changelog-delay: move the delay/divert logic inside the (inner) revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51100
diff changeset
119 self._delay_buffer = None
d83d788590a8 changelog-delay: move the delay/divert logic inside the (inner) revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51100
diff changeset
120
47425
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
121 def clear_cache(self):
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
122 self._cached_chunk = b''
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
123 self._cached_chunk_position = 0
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
124
51099
594f912818ab changelog-delay: adds some check around delaying and diverting write
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51024
diff changeset
125 @property
594f912818ab changelog-delay: adds some check around delaying and diverting write
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51024
diff changeset
126 def is_open(self):
594f912818ab changelog-delay: adds some check around delaying and diverting write
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51024
diff changeset
127 """True if any file handle is being held
594f912818ab changelog-delay: adds some check around delaying and diverting write
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51024
diff changeset
128
594f912818ab changelog-delay: adds some check around delaying and diverting write
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51024
diff changeset
129 Used for assert and debug in the python code"""
594f912818ab changelog-delay: adds some check around delaying and diverting write
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51024
diff changeset
130 return (
594f912818ab changelog-delay: adds some check around delaying and diverting write
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51024
diff changeset
131 self.reading_handle is not None or self.writing_handle is not None
594f912818ab changelog-delay: adds some check around delaying and diverting write
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51024
diff changeset
132 )
594f912818ab changelog-delay: adds some check around delaying and diverting write
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51024
diff changeset
133
47425
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
134 def _open(self, mode=b'r'):
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
135 """Return a file object"""
51103
d83d788590a8 changelog-delay: move the delay/divert logic inside the (inner) revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51100
diff changeset
136 if self._delay_buffer is None:
d83d788590a8 changelog-delay: move the delay/divert logic inside the (inner) revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51100
diff changeset
137 return self.opener(self.filename, mode=mode)
d83d788590a8 changelog-delay: move the delay/divert logic inside the (inner) revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51100
diff changeset
138 else:
d83d788590a8 changelog-delay: move the delay/divert logic inside the (inner) revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51100
diff changeset
139 return appender(
d83d788590a8 changelog-delay: move the delay/divert logic inside the (inner) revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51100
diff changeset
140 self.opener, self.filename, mode, self._delay_buffer
d83d788590a8 changelog-delay: move the delay/divert logic inside the (inner) revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51100
diff changeset
141 )
47425
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
142
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
143 @contextlib.contextmanager
51024
3314c41c3759 randomaccessfile: drop explicit passing of file description
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48946
diff changeset
144 def _read_handle(self):
47425
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
145 """File object suitable for reading data"""
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
146 # Use a file handle being actively used for writes, if available.
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
147 # There is some danger to doing this because reads will seek the
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
148 # file. However, revlog._writeentry performs a SEEK_END before all
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
149 # writes, so we should be safe.
51024
3314c41c3759 randomaccessfile: drop explicit passing of file description
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48946
diff changeset
150 if self.writing_handle:
47425
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
151 yield self.writing_handle
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
152
47463
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
153 elif self.reading_handle:
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
154 yield self.reading_handle
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
155
47425
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
156 # Otherwise open a new file handle.
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
157 else:
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
158 with self._open() as fp:
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
159 yield fp
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
160
47463
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
161 @contextlib.contextmanager
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
162 def reading(self):
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
163 """Context manager that keeps the file open for reading"""
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
164 if (
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
165 self.reading_handle is None
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
166 and self.writing_handle is None
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
167 and self.filename is not None
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
168 ):
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
169 with self._open() as fp:
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
170 self.reading_handle = fp
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
171 try:
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
172 yield
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
173 finally:
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
174 self.reading_handle = None
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
175 else:
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
176 yield
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
177
51024
3314c41c3759 randomaccessfile: drop explicit passing of file description
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48946
diff changeset
178 def read_chunk(self, offset, length):
47425
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
179 """Read a chunk of bytes from the file.
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
180
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
181 Accepts an absolute offset, length to read, and an optional existing
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
182 file handle to read from.
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
183
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
184 If an existing file handle is passed, it will be seeked and the
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
185 original seek position will NOT be restored.
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
186
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
187 Returns a str or buffer of raw byte data.
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
188
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
189 Raises if the requested number of bytes could not be read.
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
190 """
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
191 end = offset + length
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
192 cache_start = self._cached_chunk_position
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
193 cache_end = cache_start + len(self._cached_chunk)
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
194 # Is the requested chunk within the cache?
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
195 if cache_start <= offset and end <= cache_end:
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
196 if cache_start == offset and end == cache_end:
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
197 return self._cached_chunk # avoid a copy
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
198 relative_start = offset - cache_start
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
199 return util.buffer(self._cached_chunk, relative_start, length)
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
200
51024
3314c41c3759 randomaccessfile: drop explicit passing of file description
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48946
diff changeset
201 return self._read_and_update_cache(offset, length)
47425
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
202
51024
3314c41c3759 randomaccessfile: drop explicit passing of file description
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48946
diff changeset
203 def _read_and_update_cache(self, offset, length):
47425
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
204 # Cache data both forward and backward around the requested
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
205 # data, in a fixed size window. This helps speed up operations
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
206 # involving reading the revlog backwards.
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
207 real_offset = offset & ~(self.default_cached_chunk_size - 1)
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
208 real_length = (
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
209 (offset + length + self.default_cached_chunk_size)
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
210 & ~(self.default_cached_chunk_size - 1)
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
211 ) - real_offset
51024
3314c41c3759 randomaccessfile: drop explicit passing of file description
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48946
diff changeset
212 with self._read_handle() as file_obj:
47425
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
213 file_obj.seek(real_offset)
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
214 data = file_obj.read(real_length)
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
215
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
216 self._add_cached_chunk(real_offset, data)
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
217
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
218 relative_offset = offset - real_offset
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
219 got = len(data) - relative_offset
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
220 if got < length:
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
221 message = PARTIAL_READ_MSG % (self.filename, length, offset, got)
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
222 raise error.RevlogError(message)
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
223
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
224 if offset != real_offset or real_length != length:
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
225 return util.buffer(data, relative_offset, length)
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
226 return data
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
227
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
228 def _add_cached_chunk(self, offset, data):
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
229 """Add to or replace the cached data chunk.
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
230
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
231 Accepts an absolute offset and the data that is at that location.
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
232 """
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
233 if (
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
234 self._cached_chunk_position + len(self._cached_chunk) == offset
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
235 and len(self._cached_chunk) + len(data) < _MAX_CACHED_CHUNK_SIZE
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
236 ):
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
237 # add to existing cache
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
238 self._cached_chunk += data
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
239 else:
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
240 self._cached_chunk = data
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
241 self._cached_chunk_position = offset