Mercurial > public > mercurial-scm > hg

annotate mercurial/revlogutils/randomaccessfile.py @ 52049:af54626bf358

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
dirstate-map: add a missing debug wait point when accessing the v2 docket fc8e37c380d3 added synchronization points to the dirstate to allow for race condition testing without actually requiring a time-based race condition to happen. This changes adds the `pre-read-file` wait point before we read the docket, since callers might ask for the parents before anything else is read, leading to the first read being done before the wait point. This removes some differences in test output which were presumed to be speed related, but weren't.
author Rapha?l Gom?s <rgomes@octobus.net>
date Mon, 14 Oct 2024 14:14:21 +0200
parents 1c5810ce737e
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
47425
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
1 # Copyright Mercurial Contributors
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
2 #
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
3 # This software may be used and distributed according to the terms of the
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
4 # GNU General Public License version 2 or any later version.
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
5
51860
1c5810ce737e typing: add `from __future__ import annotations` to remaining source files
Matt Harbison <matt_harbison@yahoo.com>
parents: 51851
diff changeset
6 from __future__ import annotations
1c5810ce737e typing: add `from __future__ import annotations` to remaining source files
Matt Harbison <matt_harbison@yahoo.com>
parents: 51851
diff changeset
7
47425
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
8 import contextlib
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
9
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
10 from ..i18n import _
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
11 from .. import (
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
12 error,
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
13 util,
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
14 )
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
15
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
16
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
17 _MAX_CACHED_CHUNK_SIZE = 1048576 # 1 MiB
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
18
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
19 PARTIAL_READ_MSG = _(
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
20 b'partial read of revlog %s; expected %d bytes from offset %d, got %d'
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
21 )
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
22
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
23
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
24 def _is_power_of_two(n):
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
25 return (n & (n - 1) == 0) and n != 0
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
26
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
27
51100
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
28 class appender:
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
29 """the changelog index must be updated last on disk, so we use this class
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
30 to delay writes to it"""
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
31
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
32 def __init__(self, vfs, name, mode, buf):
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
33 self.data = buf
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
34 fp = vfs(name, mode)
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
35 self.fp = fp
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
36 self.offset = fp.tell()
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
37 self.size = vfs.fstat(fp).st_size
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
38 self._end = self.size
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
39
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
40 def end(self):
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
41 return self._end
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
42
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
43 def tell(self):
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
44 return self.offset
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
45
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
46 def flush(self):
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
47 pass
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
48
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
49 @property
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
50 def closed(self):
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
51 return self.fp.closed
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
52
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
53 def close(self):
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
54 self.fp.close()
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
55
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
56 def seek(self, offset, whence=0):
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
57 '''virtual file offset spans real file and data'''
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
58 if whence == 0:
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
59 self.offset = offset
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
60 elif whence == 1:
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
61 self.offset += offset
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
62 elif whence == 2:
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
63 self.offset = self.end() + offset
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
64 if self.offset < self.size:
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
65 self.fp.seek(self.offset)
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
66
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
67 def read(self, count=-1):
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
68 '''only trick here is reads that span real file and data'''
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
69 ret = b""
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
70 if self.offset < self.size:
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
71 s = self.fp.read(count)
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
72 ret = s
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
73 self.offset += len(s)
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
74 if count > 0:
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
75 count -= len(s)
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
76 if count != 0:
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
77 doff = self.offset - self.size
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
78 self.data.insert(0, b"".join(self.data))
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
79 del self.data[1:]
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
80 s = self.data[0][doff : doff + count]
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
81 self.offset += len(s)
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
82 ret += s
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
83 return ret
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
84
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
85 def write(self, s):
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
86 self.data.append(bytes(s))
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
87 self.offset += len(s)
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
88 self._end += len(s)
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
89
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
90 def __enter__(self):
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
91 self.fp.__enter__()
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
92 return self
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
93
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
94 def __exit__(self, *args):
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
95 return self.fp.__exit__(*args)
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
96
222b89224397 changelog-delay: move the appender class next to randomaccessfile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51099
diff changeset
97
48946
642e31cb55f0 py3: use class X: instead of class X(object):
Gregory Szorc <gregory.szorc@gmail.com>
parents: 47463
diff changeset
98 class randomaccessfile:
47425
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
99 """Accessing arbitrary chuncks of data within a file, with some caching"""
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
100
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
101 def __init__(
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
102 self,
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
103 opener,
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
104 filename,
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
105 default_cached_chunk_size,
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
106 initial_cache=None,
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
107 ):
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
108 # Required by bitwise manipulation below
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
109 assert _is_power_of_two(default_cached_chunk_size)
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
110
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
111 self.opener = opener
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
112 self.filename = filename
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
113 self.default_cached_chunk_size = default_cached_chunk_size
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
114 self.writing_handle = None # This is set from revlog.py
47463
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
115 self.reading_handle = None
47425
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
116 self._cached_chunk = b''
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
117 self._cached_chunk_position = 0 # Offset from the start of the file
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
118 if initial_cache:
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
119 self._cached_chunk_position, self._cached_chunk = initial_cache
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
120
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
121 def clear_cache(self):
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
122 self._cached_chunk = b''
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
123 self._cached_chunk_position = 0
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
124
51099
594f912818ab changelog-delay: adds some check around delaying and diverting write
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51024
diff changeset
125 @property
594f912818ab changelog-delay: adds some check around delaying and diverting write
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51024
diff changeset
126 def is_open(self):
594f912818ab changelog-delay: adds some check around delaying and diverting write
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51024
diff changeset
127 """True if any file handle is being held
594f912818ab changelog-delay: adds some check around delaying and diverting write
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51024
diff changeset
128
594f912818ab changelog-delay: adds some check around delaying and diverting write
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51024
diff changeset
129 Used for assert and debug in the python code"""
594f912818ab changelog-delay: adds some check around delaying and diverting write
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51024
diff changeset
130 return (
594f912818ab changelog-delay: adds some check around delaying and diverting write
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51024
diff changeset
131 self.reading_handle is not None or self.writing_handle is not None
594f912818ab changelog-delay: adds some check around delaying and diverting write
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51024
diff changeset
132 )
594f912818ab changelog-delay: adds some check around delaying and diverting write
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51024
diff changeset
133
47425
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
134 def _open(self, mode=b'r'):
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
135 """Return a file object"""
51179
a93e52f0b6ff changelog: disallow delayed write on inline changesets
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51103
diff changeset
136 return self.opener(self.filename, mode=mode)
47425
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
137
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
138 @contextlib.contextmanager
51024
3314c41c3759 randomaccessfile: drop explicit passing of file description
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48946
diff changeset
139 def _read_handle(self):
47425
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
140 """File object suitable for reading data"""
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
141 # Use a file handle being actively used for writes, if available.
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
142 # There is some danger to doing this because reads will seek the
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
143 # file. However, revlog._writeentry performs a SEEK_END before all
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
144 # writes, so we should be safe.
51024
3314c41c3759 randomaccessfile: drop explicit passing of file description
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48946
diff changeset
145 if self.writing_handle:
47425
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
146 yield self.writing_handle
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
147
47463
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
148 elif self.reading_handle:
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
149 yield self.reading_handle
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
150
47425
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
151 # Otherwise open a new file handle.
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
152 else:
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
153 with self._open() as fp:
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
154 yield fp
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
155
47463
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
156 @contextlib.contextmanager
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
157 def reading(self):
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
158 """Context manager that keeps the file open for reading"""
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
159 if (
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
160 self.reading_handle is None
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
161 and self.writing_handle is None
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
162 and self.filename is not None
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
163 ):
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
164 with self._open() as fp:
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
165 self.reading_handle = fp
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
166 try:
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
167 yield
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
168 finally:
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
169 self.reading_handle = None
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
170 else:
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
171 yield
5fa083a5ff04 copies: Keep changelog sidedata file open during copy tracing
Simon Sapin <simon.sapin@octobus.net>
parents: 47425
diff changeset
172
51024
3314c41c3759 randomaccessfile: drop explicit passing of file description
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48946
diff changeset
173 def read_chunk(self, offset, length):
47425
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
174 """Read a chunk of bytes from the file.
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
175
51851
eb9dea148233 revlog: cleanup some outdated docstrings
Rapha?l Gom?s <rgomes@octobus.net>
parents: 51179
diff changeset
176 Accepts an absolute offset, length to read.
47425
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
177
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
178 Returns a str or buffer of raw byte data.
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
179
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
180 Raises if the requested number of bytes could not be read.
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
181 """
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
182 end = offset + length
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
183 cache_start = self._cached_chunk_position
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
184 cache_end = cache_start + len(self._cached_chunk)
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
185 # Is the requested chunk within the cache?
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
186 if cache_start <= offset and end <= cache_end:
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
187 if cache_start == offset and end == cache_end:
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
188 return self._cached_chunk # avoid a copy
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
189 relative_start = offset - cache_start
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
190 return util.buffer(self._cached_chunk, relative_start, length)
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
191
51024
3314c41c3759 randomaccessfile: drop explicit passing of file description
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48946
diff changeset
192 return self._read_and_update_cache(offset, length)
47425
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
193
51024
3314c41c3759 randomaccessfile: drop explicit passing of file description
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48946
diff changeset
194 def _read_and_update_cache(self, offset, length):
47425
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
195 # Cache data both forward and backward around the requested
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
196 # data, in a fixed size window. This helps speed up operations
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
197 # involving reading the revlog backwards.
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
198 real_offset = offset & ~(self.default_cached_chunk_size - 1)
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
199 real_length = (
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
200 (offset + length + self.default_cached_chunk_size)
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
201 & ~(self.default_cached_chunk_size - 1)
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
202 ) - real_offset
51024
3314c41c3759 randomaccessfile: drop explicit passing of file description
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48946
diff changeset
203 with self._read_handle() as file_obj:
47425
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
204 file_obj.seek(real_offset)
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
205 data = file_obj.read(real_length)
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
206
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
207 self._add_cached_chunk(real_offset, data)
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
208
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
209 relative_offset = offset - real_offset
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
210 got = len(data) - relative_offset
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
211 if got < length:
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
212 message = PARTIAL_READ_MSG % (self.filename, length, offset, got)
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
213 raise error.RevlogError(message)
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
214
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
215 if offset != real_offset or real_length != length:
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
216 return util.buffer(data, relative_offset, length)
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
217 return data
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
218
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
219 def _add_cached_chunk(self, offset, data):
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
220 """Add to or replace the cached data chunk.
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
221
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
222 Accepts an absolute offset and the data that is at that location.
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
223 """
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
224 if (
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
225 self._cached_chunk_position + len(self._cached_chunk) == offset
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
226 and len(self._cached_chunk) + len(data) < _MAX_CACHED_CHUNK_SIZE
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
227 ):
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
228 # add to existing cache
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
229 self._cached_chunk += data
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
230 else:
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
231 self._cached_chunk = data
e0a314bcbc9d revlog: Extract low-level random-access file read caching logic
Simon Sapin <simon.sapin@octobus.net>
parents:
diff changeset
232 self._cached_chunk_position = offset