Mercurial > public > mercurial-scm > hg

annotate mercurial/revlogutils/randomaccessfile.py @ 51179:a93e52f0b6ff

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
changelog: disallow delayed write on inline changesets Since this will never happens, we can make the situation invalid and to stop to handling the associated the case. This simplify the random access file reading too.
author Pierre-Yves David <pierre-yves.david@octobus.net>
date Thu, 07 Dec 2023 02:07:16 +0100
parents d83d788590a8
children eb9dea148233
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
47425
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
1 # Copyright Mercurial Contributors
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
2 #
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
3 # This software may be used and distributed according to the terms of the
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
4 # GNU General Public License version 2 or any later version.
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
5
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
6 import contextlib
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
7
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
8 from ..i18n import _
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
9 from .. import (
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
10 error,
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
11 util,
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
12 )
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
13
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
14
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
15 _MAX_CACHED_CHUNK_SIZE = 1048576 # 1 MiB
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
16
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
17 PARTIAL_READ_MSG = _(
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
18 b'partial read of revlog %s; expected %d bytes from offset %d, got %d'
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
19 )
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
20
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
21
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
22 def _is_power_of_two(n):
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
23 return (n & (n - 1) == 0) and n != 0
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
24
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
25
51100
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
26 class appender:
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
27 """the changelog index must be updated last on disk, so we use this class
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
28 to delay writes to it"""
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
29
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
30 def __init__(self, vfs, name, mode, buf):
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
31 self.data = buf
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
32 fp = vfs(name, mode)
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
33 self.fp = fp
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
34 self.offset = fp.tell()
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
35 self.size = vfs.fstat(fp).st_size
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
36 self._end = self.size
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
37
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
38 def end(self):
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
39 return self._end
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
40
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
41 def tell(self):
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
42 return self.offset
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
43
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
44 def flush(self):
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
45 pass
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
46
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
47 @property
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
48 def closed(self):
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
49 return self.fp.closed
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
50
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
51 def close(self):
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
52 self.fp.close()
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
53
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
54 def seek(self, offset, whence=0):
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
55 '''virtual file offset spans real file and data'''
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
56 if whence == 0:
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
57 self.offset = offset
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
58 elif whence == 1:
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
59 self.offset += offset
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
60 elif whence == 2:
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
61 self.offset = self.end() + offset
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
62 if self.offset < self.size:
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
63 self.fp.seek(self.offset)
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
64
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
65 def read(self, count=-1):
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
66 '''only trick here is reads that span real file and data'''
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
67 ret = b""
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
68 if self.offset < self.size:
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
69 s = self.fp.read(count)
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
70 ret = s
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
71 self.offset += len(s)
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
72 if count > 0:
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
73 count -= len(s)
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
74 if count != 0:
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
75 doff = self.offset - self.size
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
76 self.data.insert(0, b"".join(self.data))
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
77 del self.data[1:]
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
78 s = self.data[0][doff : doff + count]
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
79 self.offset += len(s)
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
80 ret += s
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
81 return ret
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
82
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
83 def write(self, s):
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
84 self.data.append(bytes(s))
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
85 self.offset += len(s)
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
86 self._end += len(s)
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
87
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
88 def __enter__(self):
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
89 self.fp.__enter__()
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
90 return self
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
91
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
92 def __exit__(self, *args):
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
93 return self.fp.__exit__(*args)
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
94
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
95
48946
642e31cb55f0 py3: use class X: instead of class X(object):
Gregory Szorc <gregory.szorc@gmail.com>
parents: 47463
diff changeset
96 class randomaccessfile:
47425
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
97 """Accessing arbitrary chuncks of data within a file, with some caching"""
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
98
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
99 def __init__(
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
100 self,
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
101 opener,
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
102 filename,
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
103 default_cached_chunk_size,
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
104 initial_cache=None,
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
105 ):
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
106 # Required by bitwise manipulation below
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
107 assert _is_power_of_two(default_cached_chunk_size)
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
108
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
109 self.opener = opener
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
110 self.filename = filename
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
111 self.default_cached_chunk_size = default_cached_chunk_size
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
112 self.writing_handle = None # This is set from revlog.py
47463
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
113 self.reading_handle = None
47425
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
114 self._cached_chunk = b''
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
115 self._cached_chunk_position = 0 # Offset from the start of the file
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
116 if initial_cache:
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
117 self._cached_chunk_position, self._cached_chunk = initial_cache
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
118
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
119 def clear_cache(self):
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
120 self._cached_chunk = b''
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
121 self._cached_chunk_position = 0
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
122
51099
594f912818ab changelog-delay: adds some check around delaying and diverting write
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51024
diff changeset
123 @property
594f912818ab changelog-delay: adds some check around delaying and diverting write
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51024
diff changeset
124 def is_open(self):
594f912818ab changelog-delay: adds some check around delaying and diverting write
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51024
diff changeset
125 """True if any file handle is being held
594f912818ab changelog-delay: adds some check around delaying and diverting write
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51024
diff changeset
126
594f912818ab changelog-delay: adds some check around delaying and diverting write
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51024
diff changeset
127 Used for assert and debug in the python code"""
594f912818ab changelog-delay: adds some check around delaying and diverting write
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51024
diff changeset
128 return (
594f912818ab changelog-delay: adds some check around delaying and diverting write
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51024
diff changeset
129 self.reading_handle is not None or self.writing_handle is not None
594f912818ab changelog-delay: adds some check around delaying and diverting write
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51024
diff changeset
130 )
594f912818ab changelog-delay: adds some check around delaying and diverting write
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51024
diff changeset
131
47425
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
132 def _open(self, mode=b'r'):
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
133 """Return a file object"""
51179
a93e52f0b6ff changelog: disallow delayed write on inline changesets
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51103
diff changeset
134 return self.opener(self.filename, mode=mode)
47425
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
135
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
136 @contextlib.contextmanager
51024
3314c41c3759 randomaccessfile: drop explicit passing of file description
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48946
diff changeset
137 def _read_handle(self):
47425
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
138 """File object suitable for reading data"""
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
139 # Use a file handle being actively used for writes, if available.
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
140 # There is some danger to doing this because reads will seek the
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
141 # file. However, revlog._writeentry performs a SEEK_END before all
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
142 # writes, so we should be safe.
51024
3314c41c3759 randomaccessfile: drop explicit passing of file description
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48946
diff changeset
143 if self.writing_handle:
47425
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
144 yield self.writing_handle
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
145
47463
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
146 elif self.reading_handle:
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
147 yield self.reading_handle
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
148
47425
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
149 # Otherwise open a new file handle.
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
150 else:
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
151 with self._open() as fp:
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
152 yield fp
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
153
47463
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
154 @contextlib.contextmanager
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
155 def reading(self):
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
156 """Context manager that keeps the file open for reading"""
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
157 if (
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
158 self.reading_handle is None
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
159 and self.writing_handle is None
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
160 and self.filename is not None
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
161 ):
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
162 with self._open() as fp:
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
163 self.reading_handle = fp
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
164 try:
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
165 yield
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
166 finally:
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
167 self.reading_handle = None
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
168 else:
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
169 yield
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
170
51024
3314c41c3759 randomaccessfile: drop explicit passing of file description
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48946
diff changeset
171 def read_chunk(self, offset, length):
47425
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
172 """Read a chunk of bytes from the file.
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
173
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
174 Accepts an absolute offset, length to read, and an optional existing
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
175 file handle to read from.
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
176
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
177 If an existing file handle is passed, it will be seeked and the
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
178 original seek position will NOT be restored.
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
179
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
180 Returns a str or buffer of raw byte data.
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
181
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
182 Raises if the requested number of bytes could not be read.
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
183 """
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
184 end = offset + length
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
185 cache_start = self._cached_chunk_position
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
186 cache_end = cache_start + len(self._cached_chunk)
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
187 # Is the requested chunk within the cache?
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
188 if cache_start <= offset and end <= cache_end:
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
189 if cache_start == offset and end == cache_end:
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
190 return self._cached_chunk # avoid a copy
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
191 relative_start = offset - cache_start
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
192 return util.buffer(self._cached_chunk, relative_start, length)
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
193
51024
3314c41c3759 randomaccessfile: drop explicit passing of file description
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48946
diff changeset
194 return self._read_and_update_cache(offset, length)
47425
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
195
51024
3314c41c3759 randomaccessfile: drop explicit passing of file description
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48946
diff changeset
196 def _read_and_update_cache(self, offset, length):
47425
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
197 # Cache data both forward and backward around the requested
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
198 # data, in a fixed size window. This helps speed up operations
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
199 # involving reading the revlog backwards.
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
200 real_offset = offset & ~(self.default_cached_chunk_size - 1)
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
201 real_length = (
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
202 (offset + length + self.default_cached_chunk_size)
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
203 & ~(self.default_cached_chunk_size - 1)
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
204 ) - real_offset
51024
3314c41c3759 randomaccessfile: drop explicit passing of file description
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48946
diff changeset
205 with self._read_handle() as file_obj:
47425
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
206 file_obj.seek(real_offset)
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
207 data = file_obj.read(real_length)
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
208
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
209 self._add_cached_chunk(real_offset, data)
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
210
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
211 relative_offset = offset - real_offset
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
212 got = len(data) - relative_offset
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
213 if got < length:
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
214 message = PARTIAL_READ_MSG % (self.filename, length, offset, got)
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
215 raise error.RevlogError(message)
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
216
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
217 if offset != real_offset or real_length != length:
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
218 return util.buffer(data, relative_offset, length)
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
219 return data
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
220
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
221 def _add_cached_chunk(self, offset, data):
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
222 """Add to or replace the cached data chunk.
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
223
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
224 Accepts an absolute offset and the data that is at that location.
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
225 """
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
226 if (
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
227 self._cached_chunk_position + len(self._cached_chunk) == offset
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
228 and len(self._cached_chunk) + len(data) < _MAX_CACHED_CHUNK_SIZE
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
229 ):
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
230 # add to existing cache
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
231 self._cached_chunk += data
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
232 else:
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
233 self._cached_chunk = data
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
234 self._cached_chunk_position = offset