Mercurial > public > mercurial-scm > hg-stable

annotate mercurial/revlogutils/randomaccessfile.py @ 53040:cdd7bf612c7b stable tip

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
bundle-spec: properly format boolean parameter (issue6960) This was breaking automatic clone bundle generation. This changeset fixes it and add a test to catch it in the future.
author Pierre-Yves David <pierre-yves.david@octobus.net>
date Tue, 11 Mar 2025 02:29:42 +0100
parents 1c5810ce737e
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
47431
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
1 # Copyright Mercurial Contributors
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
2 #
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
3 # This software may be used and distributed according to the terms of the
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
4 # GNU General Public License version 2 or any later version.
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
5
51902
1c5810ce737e typing: add `from __future__ import annotations` to remaining source files
Matt Harbison <matt_harbison@yahoo.com>
parents: 51893
diff changeset
6 from __future__ import annotations
1c5810ce737e typing: add `from __future__ import annotations` to remaining source files
Matt Harbison <matt_harbison@yahoo.com>
parents: 51893
diff changeset
7
47431
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
8 import contextlib
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
9
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
10 from ..i18n import _
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
11 from .. import (
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
12 error,
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
13 util,
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
14 )
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
15
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
16
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
17 _MAX_CACHED_CHUNK_SIZE = 1048576 # 1 MiB
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
18
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
19 PARTIAL_READ_MSG = _(
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
20 b'partial read of revlog %s; expected %d bytes from offset %d, got %d'
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
21 )
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
22
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
23
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
24 def _is_power_of_two(n):
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
25 return (n & (n - 1) == 0) and n != 0
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
26
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
27
51107
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51106
diff changeset
28 class appender:
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51106
diff changeset
29 """the changelog index must be updated last on disk, so we use this class
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51106
diff changeset
30 to delay writes to it"""
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51106
diff changeset
31
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51106
diff changeset
32 def __init__(self, vfs, name, mode, buf):
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51106
diff changeset
33 self.data = buf
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51106
diff changeset
34 fp = vfs(name, mode)
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51106
diff changeset
35 self.fp = fp
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51106
diff changeset
36 self.offset = fp.tell()
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51106
diff changeset
37 self.size = vfs.fstat(fp).st_size
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51106
diff changeset
38 self._end = self.size
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51106
diff changeset
39
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51106
diff changeset
40 def end(self):
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51106
diff changeset
41 return self._end
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51106
diff changeset
42
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51106
diff changeset
43 def tell(self):
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51106
diff changeset
44 return self.offset
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51106
diff changeset
45
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51106
diff changeset
46 def flush(self):
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51106
diff changeset
47 pass
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51106
diff changeset
48
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51106
diff changeset
49 @property
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51106
diff changeset
50 def closed(self):
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51106
diff changeset
51 return self.fp.closed
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51106
diff changeset
52
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51106
diff changeset
53 def close(self):
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51106
diff changeset
54 self.fp.close()
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51106
diff changeset
55
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51106
diff changeset
56 def seek(self, offset, whence=0):
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51106
diff changeset
57 '''virtual file offset spans real file and data'''
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51106
diff changeset
58 if whence == 0:
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51106
diff changeset
59 self.offset = offset
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51106
diff changeset
60 elif whence == 1:
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51106
diff changeset
61 self.offset += offset
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51106
diff changeset
62 elif whence == 2:
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51106
diff changeset
63 self.offset = self.end() + offset
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51106
diff changeset
64 if self.offset < self.size:
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51106
diff changeset
65 self.fp.seek(self.offset)
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51106
diff changeset
66
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51106
diff changeset
67 def read(self, count=-1):
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51106
diff changeset
68 '''only trick here is reads that span real file and data'''
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51106
diff changeset
69 ret = b""
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51106
diff changeset
70 if self.offset < self.size:
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51106
diff changeset
71 s = self.fp.read(count)
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51106
diff changeset
72 ret = s
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51106
diff changeset
73 self.offset += len(s)
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51106
diff changeset
74 if count > 0:
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51106
diff changeset
75 count -= len(s)
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51106
diff changeset
76 if count != 0:
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51106
diff changeset
77 doff = self.offset - self.size
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51106
diff changeset
78 self.data.insert(0, b"".join(self.data))
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51106
diff changeset
79 del self.data[1:]
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51106
diff changeset
80 s = self.data[0][doff : doff + count]
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51106
diff changeset
81 self.offset += len(s)
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51106
diff changeset
82 ret += s
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51106
diff changeset
83 return ret
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51106
diff changeset
84
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51106
diff changeset
85 def write(self, s):
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51106
diff changeset
86 self.data.append(bytes(s))
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51106
diff changeset
87 self.offset += len(s)
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51106
diff changeset
88 self._end += len(s)
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51106
diff changeset
89
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51106
diff changeset
90 def __enter__(self):
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51106
diff changeset
91 self.fp.__enter__()
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51106
diff changeset
92 return self
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51106
diff changeset
93
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51106
diff changeset
94 def __exit__(self, *args):
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51106
diff changeset
95 return self.fp.__exit__(*args)
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51106
diff changeset
96
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51106
diff changeset
97
49037
642e31cb55f0 py3: use class X: instead of class X(object):
Gregory Szorc <gregory.szorc@gmail.com>
parents: 47469
diff changeset
98 class randomaccessfile:
47431
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
99 """Accessing arbitrary chuncks of data within a file, with some caching"""
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
100
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
101 def __init__(
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
102 self,
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
103 opener,
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
104 filename,
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
105 default_cached_chunk_size,
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
106 initial_cache=None,
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
107 ):
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
108 # Required by bitwise manipulation below
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
109 assert _is_power_of_two(default_cached_chunk_size)
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
110
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
111 self.opener = opener
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
112 self.filename = filename
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
113 self.default_cached_chunk_size = default_cached_chunk_size
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
114 self.writing_handle = None # This is set from revlog.py
47469
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47431
diff changeset
115 self.reading_handle = None
47431
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
116 self._cached_chunk = b''
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
117 self._cached_chunk_position = 0 # Offset from the start of the file
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
118 if initial_cache:
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
119 self._cached_chunk_position, self._cached_chunk = initial_cache
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
120
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
121 def clear_cache(self):
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
122 self._cached_chunk = b''
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
123 self._cached_chunk_position = 0
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
124
51106
594f912818ab changelog-delay: adds some check around delaying and diverting write
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51032
diff changeset
125 @property
594f912818ab changelog-delay: adds some check around delaying and diverting write
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51032
diff changeset
126 def is_open(self):
594f912818ab changelog-delay: adds some check around delaying and diverting write
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51032
diff changeset
127 """True if any file handle is being held
594f912818ab changelog-delay: adds some check around delaying and diverting write
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51032
diff changeset
128
594f912818ab changelog-delay: adds some check around delaying and diverting write
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51032
diff changeset
129 Used for assert and debug in the python code"""
594f912818ab changelog-delay: adds some check around delaying and diverting write
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51032
diff changeset
130 return (
594f912818ab changelog-delay: adds some check around delaying and diverting write
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51032
diff changeset
131 self.reading_handle is not None or self.writing_handle is not None
594f912818ab changelog-delay: adds some check around delaying and diverting write
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51032
diff changeset
132 )
594f912818ab changelog-delay: adds some check around delaying and diverting write
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51032
diff changeset
133
47431
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
134 def _open(self, mode=b'r'):
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
135 """Return a file object"""
51203
a93e52f0b6ff changelog: disallow delayed write on inline changesets
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51110
diff changeset
136 return self.opener(self.filename, mode=mode)
47431
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
137
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
138 @contextlib.contextmanager
51032
3314c41c3759 randomaccessfile: drop explicit passing of file description
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 49037
diff changeset
139 def _read_handle(self):
47431
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
140 """File object suitable for reading data"""
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
141 # Use a file handle being actively used for writes, if available.
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
142 # There is some danger to doing this because reads will seek the
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
143 # file. However, revlog._writeentry performs a SEEK_END before all
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
144 # writes, so we should be safe.
51032
3314c41c3759 randomaccessfile: drop explicit passing of file description
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 49037
diff changeset
145 if self.writing_handle:
47431
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
146 yield self.writing_handle
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
147
47469
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47431
diff changeset
148 elif self.reading_handle:
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47431
diff changeset
149 yield self.reading_handle
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47431
diff changeset
150
47431
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
151 # Otherwise open a new file handle.
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
152 else:
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
153 with self._open() as fp:
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
154 yield fp
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
155
47469
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47431
diff changeset
156 @contextlib.contextmanager
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47431
diff changeset
157 def reading(self):
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47431
diff changeset
158 """Context manager that keeps the file open for reading"""
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47431
diff changeset
159 if (
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47431
diff changeset
160 self.reading_handle is None
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47431
diff changeset
161 and self.writing_handle is None
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47431
diff changeset
162 and self.filename is not None
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47431
diff changeset
163 ):
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47431
diff changeset
164 with self._open() as fp:
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47431
diff changeset
165 self.reading_handle = fp
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47431
diff changeset
166 try:
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47431
diff changeset
167 yield
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47431
diff changeset
168 finally:
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47431
diff changeset
169 self.reading_handle = None
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47431
diff changeset
170 else:
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47431
diff changeset
171 yield
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47431
diff changeset
172
51032
3314c41c3759 randomaccessfile: drop explicit passing of file description
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 49037
diff changeset
173 def read_chunk(self, offset, length):
47431
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
174 """Read a chunk of bytes from the file.
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
175
51893
eb9dea148233 revlog: cleanup some outdated docstrings
Rapha?l Gom?s <rgomes@octobus.net>
parents: 51203
diff changeset
176 Accepts an absolute offset, length to read.
47431
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
177
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
178 Returns a str or buffer of raw byte data.
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
179
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
180 Raises if the requested number of bytes could not be read.
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
181 """
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
182 end = offset + length
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
183 cache_start = self._cached_chunk_position
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
184 cache_end = cache_start + len(self._cached_chunk)
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
185 # Is the requested chunk within the cache?
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
186 if cache_start <= offset and end <= cache_end:
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
187 if cache_start == offset and end == cache_end:
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
188 return self._cached_chunk # avoid a copy
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
189 relative_start = offset - cache_start
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
190 return util.buffer(self._cached_chunk, relative_start, length)
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
191
51032
3314c41c3759 randomaccessfile: drop explicit passing of file description
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 49037
diff changeset
192 return self._read_and_update_cache(offset, length)
47431
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
193
51032
3314c41c3759 randomaccessfile: drop explicit passing of file description
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 49037
diff changeset
194 def _read_and_update_cache(self, offset, length):
47431
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
195 # Cache data both forward and backward around the requested
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
196 # data, in a fixed size window. This helps speed up operations
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
197 # involving reading the revlog backwards.
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
198 real_offset = offset & ~(self.default_cached_chunk_size - 1)
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
199 real_length = (
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
200 (offset + length + self.default_cached_chunk_size)
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
201 & ~(self.default_cached_chunk_size - 1)
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
202 ) - real_offset
51032
3314c41c3759 randomaccessfile: drop explicit passing of file description
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 49037
diff changeset
203 with self._read_handle() as file_obj:
47431
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
204 file_obj.seek(real_offset)
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
205 data = file_obj.read(real_length)
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
206
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
207 self._add_cached_chunk(real_offset, data)
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
208
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
209 relative_offset = offset - real_offset
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
210 got = len(data) - relative_offset
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
211 if got < length:
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
212 message = PARTIAL_READ_MSG % (self.filename, length, offset, got)
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
213 raise error.RevlogError(message)
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
214
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
215 if offset != real_offset or real_length != length:
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
216 return util.buffer(data, relative_offset, length)
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
217 return data
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
218
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
219 def _add_cached_chunk(self, offset, data):
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
220 """Add to or replace the cached data chunk.
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
221
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
222 Accepts an absolute offset and the data that is at that location.
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
223 """
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
224 if (
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
225 self._cached_chunk_position + len(self._cached_chunk) == offset
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
226 and len(self._cached_chunk) + len(data) < _MAX_CACHED_CHUNK_SIZE
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
227 ):
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
228 # add to existing cache
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
229 self._cached_chunk += data
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
230 else:
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
231 self._cached_chunk = data
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
232 self._cached_chunk_position = offset