annotate mercurial/vfs.py @ 51921:f79f98733a5b

vfs: use @abstractmethod instead of homebrewing abstract methods The latter confuses PyCharm after adding more type annotations when, for example, `abstractvfs.rename()` calls `_auditpath()`- the latter unconditionally raised an error, so PyCharm thought the code that came after is unreachable. It also tricked pytype into marking the return type as `Never`, which isn't available until Python 3.11 (outside of `typing_extensions`). This also avoid PyCharm warnings that the call to the superclass constructor was missed (it couldn't be called because it raised an error to prevent instantiation). The statichttprepo module needed to be given an override for one of the abstract methods, so that it can be instantiated. In `abstractvfs`, this method is only called by `rename()`, so I think we can leave this empty. We raise an error in case somebody accidentally calls it in the future- it would have raised this same error prior to this change. I couldn't wrangle `import-checker.py` into accepting importing `ABC` and `abstractmethod`- for each subsequent import, it reports something like: stdlib import "contextlib" follows local import: abc I suspect the problem is that near the `if fullname != '__future__'` check, if the module doesn't fall into the error case, `seenlocal` gets set to the module name. That causes it to be treated like a local module on the next iteration, even though it is in `stdlib_modules`.
author Matt Harbison <matt_harbison@yahoo.com>
date Fri, 20 Sep 2024 00:07:39 -0400
parents 1edac12af730
children adbb183c2f27
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
31227
0f31830fbfc4 vfs: extract 'vfs' class and related code to a new 'vfs' module (API)
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31045
diff changeset
1 # vfs.py - Mercurial 'vfs' classes
13962
8b252e826c68 add: introduce a warning message for non-portable filenames (issue2756) (BC)
Adrian Buehlmann <adrian@cadifra.com>
parents:
diff changeset
2 #
46819
d4ba4d51f85f contributor: change mentions of mpm to olivia
Rapha?l Gom?s <rgomes@octobus.net>
parents: 45957
diff changeset
3 # Copyright Olivia Mackall <olivia@selenic.com>
13962
8b252e826c68 add: introduce a warning message for non-portable filenames (issue2756) (BC)
Adrian Buehlmann <adrian@cadifra.com>
parents:
diff changeset
4 #
8b252e826c68 add: introduce a warning message for non-portable filenames (issue2756) (BC)
Adrian Buehlmann <adrian@cadifra.com>
parents:
diff changeset
5 # This software may be used and distributed according to the terms of the
8b252e826c68 add: introduce a warning message for non-portable filenames (issue2756) (BC)
Adrian Buehlmann <adrian@cadifra.com>
parents:
diff changeset
6 # GNU General Public License version 2 or any later version.
27482
dde3da2246f1 scmutil: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27093
diff changeset
7
51901
f4733654f144 typing: add `from __future__ import annotations` to most files
Matt Harbison <matt_harbison@yahoo.com>
parents: 51739
diff changeset
8 from __future__ import annotations
f4733654f144 typing: add `from __future__ import annotations` to most files
Matt Harbison <matt_harbison@yahoo.com>
parents: 51739
diff changeset
9
51921
f79f98733a5b vfs: use @abstractmethod instead of homebrewing abstract methods
Matt Harbison <matt_harbison@yahoo.com>
parents: 51920
diff changeset
10 import abc
27895
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
11 import contextlib
27482
dde3da2246f1 scmutil: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27093
diff changeset
12 import os
dde3da2246f1 scmutil: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27093
diff changeset
13 import shutil
dde3da2246f1 scmutil: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27093
diff changeset
14 import stat
27895
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
15 import threading
27482
dde3da2246f1 scmutil: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27093
diff changeset
16
49722
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
17 from typing import (
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
18 Optional,
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
19 )
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
20
27482
dde3da2246f1 scmutil: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27093
diff changeset
21 from .i18n import _
dde3da2246f1 scmutil: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27093
diff changeset
22 from . import (
34040
d5b2beca16c0 python3: wrap all uses of <exception>.strerror with strtolocal
Augie Fackler <raf@durin42.com>
parents: 33647
diff changeset
23 encoding,
27482
dde3da2246f1 scmutil: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27093
diff changeset
24 error,
dde3da2246f1 scmutil: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27093
diff changeset
25 pathutil,
30318
af7c60988f6e py3: make scmutil.rcpath() return bytes
Pulkit Goyal <7895pulkit@gmail.com>
parents: 30109
diff changeset
26 pycompat,
27482
dde3da2246f1 scmutil: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27093
diff changeset
27 util,
dde3da2246f1 scmutil: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27093
diff changeset
28 )
18690
4c6f7f0dadab scmutil: split platform-specific bits into their own modules
Kevin Bullock <kbullock@ringworld.org>
parents: 18678
diff changeset
29
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43027
diff changeset
30
49722
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
31 def _avoidambig(path: bytes, oldstat):
33280
646352291f5b vfs: copy if EPERM to avoid file stat ambiguity forcibly at closing
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 33259
diff changeset
32 """Avoid file stat ambiguity forcibly
646352291f5b vfs: copy if EPERM to avoid file stat ambiguity forcibly at closing
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 33259
diff changeset
33
646352291f5b vfs: copy if EPERM to avoid file stat ambiguity forcibly at closing
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 33259
diff changeset
34 This function causes copying ``path`` file, if it is owned by
646352291f5b vfs: copy if EPERM to avoid file stat ambiguity forcibly at closing
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 33259
diff changeset
35 another (see issue5418 and issue5584 for detail).
646352291f5b vfs: copy if EPERM to avoid file stat ambiguity forcibly at closing
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 33259
diff changeset
36 """
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43027
diff changeset
37
33280
646352291f5b vfs: copy if EPERM to avoid file stat ambiguity forcibly at closing
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 33259
diff changeset
38 def checkandavoid():
646352291f5b vfs: copy if EPERM to avoid file stat ambiguity forcibly at closing
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 33259
diff changeset
39 newstat = util.filestat.frompath(path)
646352291f5b vfs: copy if EPERM to avoid file stat ambiguity forcibly at closing
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 33259
diff changeset
40 # return whether file stat ambiguity is (already) avoided
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43027
diff changeset
41 return not newstat.isambig(oldstat) or newstat.avoidambig(path, oldstat)
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43027
diff changeset
42
33280
646352291f5b vfs: copy if EPERM to avoid file stat ambiguity forcibly at closing
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 33259
diff changeset
43 if not checkandavoid():
646352291f5b vfs: copy if EPERM to avoid file stat ambiguity forcibly at closing
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 33259
diff changeset
44 # simply copy to change owner of path to get privilege to
646352291f5b vfs: copy if EPERM to avoid file stat ambiguity forcibly at closing
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 33259
diff changeset
45 # advance mtime (see issue5418)
646352291f5b vfs: copy if EPERM to avoid file stat ambiguity forcibly at closing
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 33259
diff changeset
46 util.rename(util.mktempcopy(path), path)
646352291f5b vfs: copy if EPERM to avoid file stat ambiguity forcibly at closing
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 33259
diff changeset
47 checkandavoid()
646352291f5b vfs: copy if EPERM to avoid file stat ambiguity forcibly at closing
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 33259
diff changeset
48
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43027
diff changeset
49
51921
f79f98733a5b vfs: use @abstractmethod instead of homebrewing abstract methods
Matt Harbison <matt_harbison@yahoo.com>
parents: 51920
diff changeset
50 class abstractvfs(abc.ABC):
14089
d3f7e110c3c0 opener: introduce an abstact superclass of it
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 14068
diff changeset
51 """Abstract base class; cannot be instantiated"""
d3f7e110c3c0 opener: introduce an abstact superclass of it
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 14068
diff changeset
52
47804
98c3fa6a3ac2 vfs: always use / as file separator (issue6546)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47449
diff changeset
53 # default directory separator for vfs
98c3fa6a3ac2 vfs: always use / as file separator (issue6546)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47449
diff changeset
54 #
98c3fa6a3ac2 vfs: always use / as file separator (issue6546)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47449
diff changeset
55 # Other vfs code always use `/` and this works fine because python file API
98c3fa6a3ac2 vfs: always use / as file separator (issue6546)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47449
diff changeset
56 # abstract the use of `/` and make it work transparently. For consistency
98c3fa6a3ac2 vfs: always use / as file separator (issue6546)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47449
diff changeset
57 # vfs will always use `/` when joining. This avoid some confusion in
98c3fa6a3ac2 vfs: always use / as file separator (issue6546)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47449
diff changeset
58 # encoded vfs (see issue6546)
98c3fa6a3ac2 vfs: always use / as file separator (issue6546)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47449
diff changeset
59 _dir_sep = b'/'
98c3fa6a3ac2 vfs: always use / as file separator (issue6546)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47449
diff changeset
60
49722
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
61 # TODO: type return, which is util.posixfile wrapped by a proxy
51921
f79f98733a5b vfs: use @abstractmethod instead of homebrewing abstract methods
Matt Harbison <matt_harbison@yahoo.com>
parents: 51920
diff changeset
62 @abc.abstractmethod
49722
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
63 def __call__(self, path: bytes, mode: bytes = b'rb', **kwargs):
51921
f79f98733a5b vfs: use @abstractmethod instead of homebrewing abstract methods
Matt Harbison <matt_harbison@yahoo.com>
parents: 51920
diff changeset
64 ...
43515
b16912f8c07c vfs: add a NotImplementedError implementation of __call__
Augie Fackler <augie@google.com>
parents: 43514
diff changeset
65
51921
f79f98733a5b vfs: use @abstractmethod instead of homebrewing abstract methods
Matt Harbison <matt_harbison@yahoo.com>
parents: 51920
diff changeset
66 @abc.abstractmethod
49722
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
67 def _auditpath(self, path: bytes, mode: bytes):
51921
f79f98733a5b vfs: use @abstractmethod instead of homebrewing abstract methods
Matt Harbison <matt_harbison@yahoo.com>
parents: 51920
diff changeset
68 ...
41094
d9b6b9ed96d8 vfs: add a `_auditpath` to abstract vfs
Boris Feld <boris.feld@octobus.net>
parents: 41093
diff changeset
69
51921
f79f98733a5b vfs: use @abstractmethod instead of homebrewing abstract methods
Matt Harbison <matt_harbison@yahoo.com>
parents: 51920
diff changeset
70 @abc.abstractmethod
49722
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
71 def join(self, path: Optional[bytes], *insidef: bytes) -> bytes:
51921
f79f98733a5b vfs: use @abstractmethod instead of homebrewing abstract methods
Matt Harbison <matt_harbison@yahoo.com>
parents: 51920
diff changeset
72 ...
43516
8b0fa4de0064 vfs: add NotImplementedError version of join
Augie Fackler <augie@google.com>
parents: 43515
diff changeset
73
49722
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
74 def tryread(self, path: bytes) -> bytes:
16479
fc04698fa778 opener: coding style, use triple quotes for doc string
Thomas Arendsen Hein <thomas@intevation.de>
parents: 16455
diff changeset
75 '''gracefully return an empty string for missing files'''
16455
154219f3a6a4 opener: introduce tryread helper
Matt Mackall <mpm@selenic.com>
parents: 16436
diff changeset
76 try:
154219f3a6a4 opener: introduce tryread helper
Matt Mackall <mpm@selenic.com>
parents: 16436
diff changeset
77 return self.read(path)
49314
2e726c934fcd py3: catch FileNotFoundError instead of checking errno == ENOENT
Manuel Jacob <me@manueljacob.de>
parents: 49037
diff changeset
78 except FileNotFoundError:
2e726c934fcd py3: catch FileNotFoundError instead of checking errno == ENOENT
Manuel Jacob <me@manueljacob.de>
parents: 49037
diff changeset
79 pass
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
80 return b""
16455
154219f3a6a4 opener: introduce tryread helper
Matt Mackall <mpm@selenic.com>
parents: 16436
diff changeset
81
49722
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
82 def tryreadlines(self, path: bytes, mode: bytes = b'rb'):
23368
bf8c3172255c vfs: add "readlines" and "tryreadlines"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 23259
diff changeset
83 '''gracefully return an empty array for missing files'''
bf8c3172255c vfs: add "readlines" and "tryreadlines"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 23259
diff changeset
84 try:
bf8c3172255c vfs: add "readlines" and "tryreadlines"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 23259
diff changeset
85 return self.readlines(path, mode=mode)
49314
2e726c934fcd py3: catch FileNotFoundError instead of checking errno == ENOENT
Manuel Jacob <me@manueljacob.de>
parents: 49037
diff changeset
86 except FileNotFoundError:
2e726c934fcd py3: catch FileNotFoundError instead of checking errno == ENOENT
Manuel Jacob <me@manueljacob.de>
parents: 49037
diff changeset
87 pass
23368
bf8c3172255c vfs: add "readlines" and "tryreadlines"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 23259
diff changeset
88 return []
bf8c3172255c vfs: add "readlines" and "tryreadlines"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 23259
diff changeset
89
29730
2dd8c225e94c vfs: use propertycache for open
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 29727
diff changeset
90 @util.propertycache
2dd8c225e94c vfs: use propertycache for open
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 29727
diff changeset
91 def open(self):
45957
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 44575
diff changeset
92 """Open ``path`` file, which is relative to vfs root.
23370
46265d0f0c7b vfs: add "notindexed" argument to invoke "ensuredir" with it in write mode
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 23368
diff changeset
93
46265d0f0c7b vfs: add "notindexed" argument to invoke "ensuredir" with it in write mode
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 23368
diff changeset
94 Newly created directories are marked as "not to be indexed by
46265d0f0c7b vfs: add "notindexed" argument to invoke "ensuredir" with it in write mode
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 23368
diff changeset
95 the content indexing service", if ``notindexed`` is specified
46265d0f0c7b vfs: add "notindexed" argument to invoke "ensuredir" with it in write mode
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 23368
diff changeset
96 for "write" mode access.
45957
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 44575
diff changeset
97 """
29730
2dd8c225e94c vfs: use propertycache for open
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 29727
diff changeset
98 return self.__call__
19897
896a4568def7 vfs: add "open()" for newly added code paths which open files via vfs
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 19896
diff changeset
99
49722
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
100 def read(self, path: bytes) -> bytes:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
101 with self(path, b'rb') as fp:
14097
ca3376f044f8 opener: add read & write utility methods
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 14090
diff changeset
102 return fp.read()
ca3376f044f8 opener: add read & write utility methods
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 14090
diff changeset
103
49722
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
104 def readlines(self, path: bytes, mode: bytes = b'rb'):
27706
22e362da27cf scmutil: use context managers for file handles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27651
diff changeset
105 with self(path, mode=mode) as fp:
23368
bf8c3172255c vfs: add "readlines" and "tryreadlines"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 23259
diff changeset
106 return fp.readlines()
bf8c3172255c vfs: add "readlines" and "tryreadlines"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 23259
diff changeset
107
49722
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
108 def write(
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
109 self, path: bytes, data: bytes, backgroundclose=False, **kwargs
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
110 ) -> int:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
111 with self(path, b'wb', backgroundclose=backgroundclose, **kwargs) as fp:
14167
0e4753807c93 util & scmutil: adapt read/write helpers as request by mpm
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 14138
diff changeset
112 return fp.write(data)
0e4753807c93 util & scmutil: adapt read/write helpers as request by mpm
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 14138
diff changeset
113
49722
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
114 def writelines(
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
115 self, path: bytes, data: bytes, mode: bytes = b'wb', notindexed=False
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
116 ) -> None:
27706
22e362da27cf scmutil: use context managers for file handles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27651
diff changeset
117 with self(path, mode=mode, notindexed=notindexed) as fp:
23371
1df6519eb3ab vfs: add "writelines"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 23370
diff changeset
118 return fp.writelines(data)
1df6519eb3ab vfs: add "writelines"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 23370
diff changeset
119
49722
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
120 def append(self, path: bytes, data: bytes) -> int:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
121 with self(path, b'ab') as fp:
14097
ca3376f044f8 opener: add read & write utility methods
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 14090
diff changeset
122 return fp.write(data)
ca3376f044f8 opener: add read & write utility methods
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 14090
diff changeset
123
49722
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
124 def basename(self, path: bytes) -> bytes:
25770
39de2e9cc6f4 vfs: add basename
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25739
diff changeset
125 """return base element of a path (as os.path.basename would do)
39de2e9cc6f4 vfs: add basename
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25739
diff changeset
126
39de2e9cc6f4 vfs: add basename
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25739
diff changeset
127 This exists to allow handling of strange encoding if needed."""
39de2e9cc6f4 vfs: add basename
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25739
diff changeset
128 return os.path.basename(path)
39de2e9cc6f4 vfs: add basename
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25739
diff changeset
129
49722
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
130 def chmod(self, path: bytes, mode: int) -> None:
20086
f3df2612f3c3 vfs: add "chmod()"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 20085
diff changeset
131 return os.chmod(self.join(path), mode)
f3df2612f3c3 vfs: add "chmod()"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 20085
diff changeset
132
49722
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
133 def dirname(self, path: bytes) -> bytes:
25772
5471965af5cb vfs: add dirname
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25770
diff changeset
134 """return dirname element of a path (as os.path.dirname would do)
5471965af5cb vfs: add dirname
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25770
diff changeset
135
5471965af5cb vfs: add dirname
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25770
diff changeset
136 This exists to allow handling of strange encoding if needed."""
5471965af5cb vfs: add dirname
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25770
diff changeset
137 return os.path.dirname(path)
5471965af5cb vfs: add dirname
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25770
diff changeset
138
49722
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
139 def exists(self, path: Optional[bytes] = None) -> bool:
17161
be016e96117a localrepo: use file API via vfs while ensuring repository directory
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17157
diff changeset
140 return os.path.exists(self.join(path))
be016e96117a localrepo: use file API via vfs while ensuring repository directory
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17157
diff changeset
141
19899
8c3dcbbfb5de changelog: use "vfs.fstat()" instead of "util.fstat()"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 19897
diff changeset
142 def fstat(self, fp):
8c3dcbbfb5de changelog: use "vfs.fstat()" instead of "util.fstat()"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 19897
diff changeset
143 return util.fstat(fp)
8c3dcbbfb5de changelog: use "vfs.fstat()" instead of "util.fstat()"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 19897
diff changeset
144
49722
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
145 def isdir(self, path: Optional[bytes] = None) -> bool:
17161
be016e96117a localrepo: use file API via vfs while ensuring repository directory
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17157
diff changeset
146 return os.path.isdir(self.join(path))
be016e96117a localrepo: use file API via vfs while ensuring repository directory
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17157
diff changeset
147
49722
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
148 def isfile(self, path: Optional[bytes] = None) -> bool:
20085
589d6bb5b18d vfs: add "isfile()"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 20045
diff changeset
149 return os.path.isfile(self.join(path))
589d6bb5b18d vfs: add "isfile()"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 20045
diff changeset
150
49722
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
151 def islink(self, path: Optional[bytes] = None) -> bool:
18949
138978f20180 localrepo: use "vfs.islink()" instead of "os.path.islink()"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 18948
diff changeset
152 return os.path.islink(self.join(path))
138978f20180 localrepo: use "vfs.islink()" instead of "os.path.islink()"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 18948
diff changeset
153
49722
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
154 def isfileorlink(self, path: Optional[bytes] = None) -> bool:
45957
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 44575
diff changeset
155 """return whether path is a regular file or a symlink
27571
6a6e78f84cc6 merge: while checking for unknown files don't follow symlinks (issue5027)
Siddharth Agarwal <sid0@fb.com>
parents: 26836
diff changeset
156
45957
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 44575
diff changeset
157 Unlike isfile, this doesn't follow symlinks."""
27571
6a6e78f84cc6 merge: while checking for unknown files don't follow symlinks (issue5027)
Siddharth Agarwal <sid0@fb.com>
parents: 26836
diff changeset
158 try:
6a6e78f84cc6 merge: while checking for unknown files don't follow symlinks (issue5027)
Siddharth Agarwal <sid0@fb.com>
parents: 26836
diff changeset
159 st = self.lstat(path)
6a6e78f84cc6 merge: while checking for unknown files don't follow symlinks (issue5027)
Siddharth Agarwal <sid0@fb.com>
parents: 26836
diff changeset
160 except OSError:
6a6e78f84cc6 merge: while checking for unknown files don't follow symlinks (issue5027)
Siddharth Agarwal <sid0@fb.com>
parents: 26836
diff changeset
161 return False
6a6e78f84cc6 merge: while checking for unknown files don't follow symlinks (issue5027)
Siddharth Agarwal <sid0@fb.com>
parents: 26836
diff changeset
162 mode = st.st_mode
6a6e78f84cc6 merge: while checking for unknown files don't follow symlinks (issue5027)
Siddharth Agarwal <sid0@fb.com>
parents: 26836
diff changeset
163 return stat.S_ISREG(mode) or stat.S_ISLNK(mode)
6a6e78f84cc6 merge: while checking for unknown files don't follow symlinks (issue5027)
Siddharth Agarwal <sid0@fb.com>
parents: 26836
diff changeset
164
49722
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
165 def _join(self, *paths: bytes) -> bytes:
47804
98c3fa6a3ac2 vfs: always use / as file separator (issue6546)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47449
diff changeset
166 root_idx = 0
98c3fa6a3ac2 vfs: always use / as file separator (issue6546)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47449
diff changeset
167 for idx, p in enumerate(paths):
98c3fa6a3ac2 vfs: always use / as file separator (issue6546)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47449
diff changeset
168 if os.path.isabs(p) or p.startswith(self._dir_sep):
98c3fa6a3ac2 vfs: always use / as file separator (issue6546)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47449
diff changeset
169 root_idx = idx
98c3fa6a3ac2 vfs: always use / as file separator (issue6546)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47449
diff changeset
170 if root_idx != 0:
98c3fa6a3ac2 vfs: always use / as file separator (issue6546)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47449
diff changeset
171 paths = paths[root_idx:]
98c3fa6a3ac2 vfs: always use / as file separator (issue6546)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47449
diff changeset
172 paths = [p for p in paths if p]
98c3fa6a3ac2 vfs: always use / as file separator (issue6546)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47449
diff changeset
173 return self._dir_sep.join(paths)
98c3fa6a3ac2 vfs: always use / as file separator (issue6546)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47449
diff changeset
174
49722
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
175 def reljoin(self, *paths: bytes) -> bytes:
23581
aed981c7bebf vfs: add a 'reljoin' function for joining relative paths
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23539
diff changeset
176 """join various elements of a path together (as os.path.join would do)
aed981c7bebf vfs: add a 'reljoin' function for joining relative paths
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23539
diff changeset
177
aed981c7bebf vfs: add a 'reljoin' function for joining relative paths
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23539
diff changeset
178 The vfs base is not injected so that path stay relative. This exists
aed981c7bebf vfs: add a 'reljoin' function for joining relative paths
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23539
diff changeset
179 to allow handling of strange encoding if needed."""
47804
98c3fa6a3ac2 vfs: always use / as file separator (issue6546)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47449
diff changeset
180 return self._join(*paths)
23581
aed981c7bebf vfs: add a 'reljoin' function for joining relative paths
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23539
diff changeset
181
49722
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
182 def split(self, path: bytes):
23582
7559dc8c4238 vfs: add a 'split' method
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23581
diff changeset
183 """split top-most element of a path (as os.path.split would do)
7559dc8c4238 vfs: add a 'split' method
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23581
diff changeset
184
7559dc8c4238 vfs: add a 'split' method
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23581
diff changeset
185 This exists to allow handling of strange encoding if needed."""
7559dc8c4238 vfs: add a 'split' method
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23581
diff changeset
186 return os.path.split(path)
7559dc8c4238 vfs: add a 'split' method
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23581
diff changeset
187
49722
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
188 def lexists(self, path: Optional[bytes] = None) -> bool:
21563
764b691b8bda vfs: add lexists() in current api
Chinmay Joshi <c@chinmayjoshi.com>
parents: 21111
diff changeset
189 return os.path.lexists(self.join(path))
764b691b8bda vfs: add lexists() in current api
Chinmay Joshi <c@chinmayjoshi.com>
parents: 21111
diff changeset
190
49722
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
191 def lstat(self, path: Optional[bytes] = None):
19900
7c21e3398931 context: use "vfs.lstat()" instead of "os.lstat()"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 19899
diff changeset
192 return os.lstat(self.join(path))
7c21e3398931 context: use "vfs.lstat()" instead of "os.lstat()"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 19899
diff changeset
193
51649
ba205f944cb4 mmap: add a `is_mmap_safe` method to vfs
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51618
diff changeset
194 def is_mmap_safe(self, path: Optional[bytes] = None) -> bool:
ba205f944cb4 mmap: add a `is_mmap_safe` method to vfs
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51618
diff changeset
195 """return True if it is safe to read a file content as mmap
ba205f944cb4 mmap: add a `is_mmap_safe` method to vfs
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51618
diff changeset
196
ba205f944cb4 mmap: add a `is_mmap_safe` method to vfs
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51618
diff changeset
197 This focus on the file system aspect of such safety, the application
ba205f944cb4 mmap: add a `is_mmap_safe` method to vfs
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51618
diff changeset
198 logic around that file is not taken into account, so caller need to
ba205f944cb4 mmap: add a `is_mmap_safe` method to vfs
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51618
diff changeset
199 make sure the file won't be truncated in a way that will create SIGBUS
ba205f944cb4 mmap: add a `is_mmap_safe` method to vfs
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51618
diff changeset
200 on access.
ba205f944cb4 mmap: add a `is_mmap_safe` method to vfs
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51618
diff changeset
201
ba205f944cb4 mmap: add a `is_mmap_safe` method to vfs
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51618
diff changeset
202
ba205f944cb4 mmap: add a `is_mmap_safe` method to vfs
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51618
diff changeset
203 The initial motivation for this logic is that if mmap is used on NFS
ba205f944cb4 mmap: add a `is_mmap_safe` method to vfs
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51618
diff changeset
204 and somebody deletes the mapped file (e.g. by renaming on top of it),
ba205f944cb4 mmap: add a `is_mmap_safe` method to vfs
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51618
diff changeset
205 then you get SIGBUS, which can be pretty disruptive: we get core dump
ba205f944cb4 mmap: add a `is_mmap_safe` method to vfs
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51618
diff changeset
206 reports, and the process terminates without writing to the blackbox.
ba205f944cb4 mmap: add a `is_mmap_safe` method to vfs
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51618
diff changeset
207
ba205f944cb4 mmap: add a `is_mmap_safe` method to vfs
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51618
diff changeset
208 Instead in this situation we prefer to read the file normally.
ba205f944cb4 mmap: add a `is_mmap_safe` method to vfs
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51618
diff changeset
209 The risk of ESTALE in the middle of the read remains, but it's
ba205f944cb4 mmap: add a `is_mmap_safe` method to vfs
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51618
diff changeset
210 smaller because we read sooner and the error should be reported
ba205f944cb4 mmap: add a `is_mmap_safe` method to vfs
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51618
diff changeset
211 just as any other error.
ba205f944cb4 mmap: add a `is_mmap_safe` method to vfs
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51618
diff changeset
212
ba205f944cb4 mmap: add a `is_mmap_safe` method to vfs
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51618
diff changeset
213 Note that python standard library does not offer the necessary function
ba205f944cb4 mmap: add a `is_mmap_safe` method to vfs
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51618
diff changeset
214 to detect the file stem bits. So this detection rely on compiled bits
ba205f944cb4 mmap: add a `is_mmap_safe` method to vfs
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51618
diff changeset
215 and is not available in pure python.
ba205f944cb4 mmap: add a `is_mmap_safe` method to vfs
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51618
diff changeset
216 """
ba205f944cb4 mmap: add a `is_mmap_safe` method to vfs
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51618
diff changeset
217 # XXX Since we already assume a vfs to address a consistent file system
ba205f944cb4 mmap: add a `is_mmap_safe` method to vfs
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51618
diff changeset
218 # in other location, we could determine the fstype once for the root
ba205f944cb4 mmap: add a `is_mmap_safe` method to vfs
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51618
diff changeset
219 # and cache that value.
ba205f944cb4 mmap: add a `is_mmap_safe` method to vfs
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51618
diff changeset
220 fstype = util.getfstype(self.join(path))
ba205f944cb4 mmap: add a `is_mmap_safe` method to vfs
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51618
diff changeset
221 return fstype is not None and fstype != b'nfs'
ba205f944cb4 mmap: add a `is_mmap_safe` method to vfs
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51618
diff changeset
222
49722
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
223 def listdir(self, path: Optional[bytes] = None):
21799
dfacdd6a111e vfs: add listdir for os.listdir in vfs
Chinmay Joshi <c@chinmayjoshi.com>
parents: 21716
diff changeset
224 return os.listdir(self.join(path))
dfacdd6a111e vfs: add listdir for os.listdir in vfs
Chinmay Joshi <c@chinmayjoshi.com>
parents: 21716
diff changeset
225
49722
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
226 def makedir(self, path: Optional[bytes] = None, notindexed=True):
17161
be016e96117a localrepo: use file API via vfs while ensuring repository directory
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17157
diff changeset
227 return util.makedir(self.join(path), notindexed)
be016e96117a localrepo: use file API via vfs while ensuring repository directory
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17157
diff changeset
228
49722
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
229 def makedirs(
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
230 self, path: Optional[bytes] = None, mode: Optional[int] = None
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
231 ):
17161
be016e96117a localrepo: use file API via vfs while ensuring repository directory
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17157
diff changeset
232 return util.makedirs(self.join(path), mode)
be016e96117a localrepo: use file API via vfs while ensuring repository directory
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17157
diff changeset
233
49722
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
234 def makelock(self, info, path: bytes):
20090
88d8e568add1 vfs: add "makelock()" and "readlock()"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 20086
diff changeset
235 return util.makelock(info, self.join(path))
88d8e568add1 vfs: add "makelock()" and "readlock()"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 20086
diff changeset
236
49722
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
237 def mkdir(self, path: Optional[bytes] = None):
17723
ab23768746fd scmutil: reorder newly added functions for vfs support in dictionary order
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17681
diff changeset
238 return os.mkdir(self.join(path))
ab23768746fd scmutil: reorder newly added functions for vfs support in dictionary order
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17681
diff changeset
239
49722
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
240 def mkstemp(
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
241 self,
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
242 suffix: bytes = b'',
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
243 prefix: bytes = b'tmp',
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
244 dir: Optional[bytes] = None,
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
245 ):
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43027
diff changeset
246 fd, name = pycompat.mkstemp(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43027
diff changeset
247 suffix=suffix, prefix=prefix, dir=self.join(dir)
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43027
diff changeset
248 )
20980
6fb4c94ae300 vfs: add "mkstemp()"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 20862
diff changeset
249 dname, fname = util.split(name)
6fb4c94ae300 vfs: add "mkstemp()"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 20862
diff changeset
250 if dir:
6fb4c94ae300 vfs: add "mkstemp()"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 20862
diff changeset
251 return fd, os.path.join(dir, fname)
6fb4c94ae300 vfs: add "mkstemp()"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 20862
diff changeset
252 else:
6fb4c94ae300 vfs: add "mkstemp()"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 20862
diff changeset
253 return fd, fname
6fb4c94ae300 vfs: add "mkstemp()"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 20862
diff changeset
254
49722
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
255 def readdir(self, path: Optional[bytes] = None, stat=None, skip=None):
32248
d74b0cff94a9 osutil: proxy through util (and platform) modules (API)
Yuya Nishihara <yuya@tcha.org>
parents: 31649
diff changeset
256 return util.listdir(self.join(path), stat, skip)
17747
aad3bce98f76 store: invoke "osutil.listdir()" via vfs
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17726
diff changeset
257
49722
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
258 def readlock(self, path: bytes) -> bytes:
20090
88d8e568add1 vfs: add "makelock()" and "readlock()"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 20086
diff changeset
259 return util.readlock(self.join(path))
88d8e568add1 vfs: add "makelock()" and "readlock()"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 20086
diff changeset
260
49722
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
261 def rename(self, src: bytes, dst: bytes, checkambig=False):
29367
4e6e280e238f doc: describe detail about checkambig optional argument
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29341
diff changeset
262 """Rename from src to dst
4e6e280e238f doc: describe detail about checkambig optional argument
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29341
diff changeset
263
4e6e280e238f doc: describe detail about checkambig optional argument
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29341
diff changeset
264 checkambig argument is used with util.filestat, and is useful
4e6e280e238f doc: describe detail about checkambig optional argument
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29341
diff changeset
265 only if destination file is guarded by any lock
4e6e280e238f doc: describe detail about checkambig optional argument
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29341
diff changeset
266 (e.g. repo.lock or repo.wlock).
33282
d1db7af81548 vfs: add explanation about cost of checkambig=True in corner case
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 33281
diff changeset
267
d1db7af81548 vfs: add explanation about cost of checkambig=True in corner case
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 33281
diff changeset
268 To avoid file stat ambiguity forcibly, checkambig=True involves
d1db7af81548 vfs: add explanation about cost of checkambig=True in corner case
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 33281
diff changeset
269 copying ``src`` file, if it is owned by another. Therefore, use
d1db7af81548 vfs: add explanation about cost of checkambig=True in corner case
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 33281
diff changeset
270 checkambig=True only in limited cases (see also issue5418 and
d1db7af81548 vfs: add explanation about cost of checkambig=True in corner case
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 33281
diff changeset
271 issue5584 for detail).
29367
4e6e280e238f doc: describe detail about checkambig optional argument
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29341
diff changeset
272 """
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
273 self._auditpath(dst, b'w')
32768
ed66ec39933f vfs: create copy at renaming to avoid file stat ambiguity if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 32767
diff changeset
274 srcpath = self.join(src)
29203
731ced087a4b vfs: make rename avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29202
diff changeset
275 dstpath = self.join(dst)
32792
7ad95626f6a7 filestat: move __init__ to frompath constructor
Siddharth Agarwal <sid0@fb.com>
parents: 32768
diff changeset
276 oldstat = checkambig and util.filestat.frompath(dstpath)
29203
731ced087a4b vfs: make rename avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29202
diff changeset
277 if oldstat and oldstat.stat:
33281
6af0f023d014 vfs: replace avoiding ambiguity in abstractvfs.rename with _avoidambig
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 33280
diff changeset
278 ret = util.rename(srcpath, dstpath)
6af0f023d014 vfs: replace avoiding ambiguity in abstractvfs.rename with _avoidambig
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 33280
diff changeset
279 _avoidambig(dstpath, oldstat)
29203
731ced087a4b vfs: make rename avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29202
diff changeset
280 return ret
32768
ed66ec39933f vfs: create copy at renaming to avoid file stat ambiguity if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 32767
diff changeset
281 return util.rename(srcpath, dstpath)
18948
2f05fa162316 localrepo: use "vfs.rename()" instead of "util.rename()"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 18945
diff changeset
282
49722
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
283 def readlink(self, path: bytes) -> bytes:
39909
5fe0b880200e py3: convert os.readlink() path to native strings on Windows
Matt Harbison <matt_harbison@yahoo.com>
parents: 39481
diff changeset
284 return util.readlink(self.join(path))
18950
647e3b0c8751 localrepo: use "vfs.readlink()" instead of "os.readlink()"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 18949
diff changeset
285
49722
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
286 def removedirs(self, path: Optional[bytes] = None):
45957
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 44575
diff changeset
287 """Remove a leaf directory and all empty intermediate ones"""
24693
0d28b0df77ea vfs: add removedirs
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 24689
diff changeset
288 return util.removedirs(self.join(path))
0d28b0df77ea vfs: add removedirs
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 24689
diff changeset
289
49722
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
290 def rmdir(self, path: Optional[bytes] = None):
39481
3dd34b401bc2 merge: use vfs methods for I/O
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38497
diff changeset
291 """Remove an empty directory."""
3dd34b401bc2 merge: use vfs methods for I/O
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38497
diff changeset
292 return os.rmdir(self.join(path))
3dd34b401bc2 merge: use vfs methods for I/O
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38497
diff changeset
293
49722
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
294 def rmtree(
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
295 self, path: Optional[bytes] = None, ignore_errors=False, forcibly=False
51739
be6d8ea6d3d2 typing: add a trivial type hint to `mercurial/vfs.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 51649
diff changeset
296 ) -> None:
24689
ca3a90096c95 vfs: add rmtree
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 24635
diff changeset
297 """Remove a directory tree recursively
ca3a90096c95 vfs: add rmtree
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 24635
diff changeset
298
ca3a90096c95 vfs: add rmtree
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 24635
diff changeset
299 If ``forcibly``, this tries to remove READ-ONLY files, too.
ca3a90096c95 vfs: add rmtree
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 24635
diff changeset
300 """
ca3a90096c95 vfs: add rmtree
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 24635
diff changeset
301 if forcibly:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43027
diff changeset
302
50750
f173c2c23289 vfs: handle shutil.rmtree deprecation of onerror in Python 3.12
Mads Kiilerich <mads@kiilerich.com>
parents: 50365
diff changeset
303 def onexc(function, path, excinfo):
24689
ca3a90096c95 vfs: add rmtree
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 24635
diff changeset
304 if function is not os.remove:
ca3a90096c95 vfs: add rmtree
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 24635
diff changeset
305 raise
ca3a90096c95 vfs: add rmtree
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 24635
diff changeset
306 # read-only files cannot be unlinked under Windows
ca3a90096c95 vfs: add rmtree
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 24635
diff changeset
307 s = os.stat(path)
ca3a90096c95 vfs: add rmtree
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 24635
diff changeset
308 if (s.st_mode & stat.S_IWRITE) != 0:
ca3a90096c95 vfs: add rmtree
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 24635
diff changeset
309 raise
ca3a90096c95 vfs: add rmtree
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 24635
diff changeset
310 os.chmod(path, stat.S_IMODE(s.st_mode) | stat.S_IWRITE)
ca3a90096c95 vfs: add rmtree
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 24635
diff changeset
311 os.remove(path)
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43027
diff changeset
312
24689
ca3a90096c95 vfs: add rmtree
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 24635
diff changeset
313 else:
50750
f173c2c23289 vfs: handle shutil.rmtree deprecation of onerror in Python 3.12
Mads Kiilerich <mads@kiilerich.com>
parents: 50365
diff changeset
314 onexc = None
f173c2c23289 vfs: handle shutil.rmtree deprecation of onerror in Python 3.12
Mads Kiilerich <mads@kiilerich.com>
parents: 50365
diff changeset
315 try:
f173c2c23289 vfs: handle shutil.rmtree deprecation of onerror in Python 3.12
Mads Kiilerich <mads@kiilerich.com>
parents: 50365
diff changeset
316 # pytype: disable=wrong-keyword-args
f173c2c23289 vfs: handle shutil.rmtree deprecation of onerror in Python 3.12
Mads Kiilerich <mads@kiilerich.com>
parents: 50365
diff changeset
317 return shutil.rmtree(
f173c2c23289 vfs: handle shutil.rmtree deprecation of onerror in Python 3.12
Mads Kiilerich <mads@kiilerich.com>
parents: 50365
diff changeset
318 self.join(path), ignore_errors=ignore_errors, onexc=onexc
f173c2c23289 vfs: handle shutil.rmtree deprecation of onerror in Python 3.12
Mads Kiilerich <mads@kiilerich.com>
parents: 50365
diff changeset
319 )
f173c2c23289 vfs: handle shutil.rmtree deprecation of onerror in Python 3.12
Mads Kiilerich <mads@kiilerich.com>
parents: 50365
diff changeset
320 # pytype: enable=wrong-keyword-args
f173c2c23289 vfs: handle shutil.rmtree deprecation of onerror in Python 3.12
Mads Kiilerich <mads@kiilerich.com>
parents: 50365
diff changeset
321 except TypeError: # onexc was introduced in Python 3.12
f173c2c23289 vfs: handle shutil.rmtree deprecation of onerror in Python 3.12
Mads Kiilerich <mads@kiilerich.com>
parents: 50365
diff changeset
322 return shutil.rmtree(
f173c2c23289 vfs: handle shutil.rmtree deprecation of onerror in Python 3.12
Mads Kiilerich <mads@kiilerich.com>
parents: 50365
diff changeset
323 self.join(path), ignore_errors=ignore_errors, onerror=onexc
f173c2c23289 vfs: handle shutil.rmtree deprecation of onerror in Python 3.12
Mads Kiilerich <mads@kiilerich.com>
parents: 50365
diff changeset
324 )
24689
ca3a90096c95 vfs: add rmtree
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 24635
diff changeset
325
49722
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
326 def setflags(self, path: bytes, l: bool, x: bool):
18951
d13916a00b7e localrepo: use "vfs.setflags()" instead of "util.setflags()"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 18950
diff changeset
327 return util.setflags(self.join(path), l, x)
d13916a00b7e localrepo: use "vfs.setflags()" instead of "util.setflags()"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 18950
diff changeset
328
49722
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
329 def stat(self, path: Optional[bytes] = None):
17726
7cb7e17c23b2 store: invoke "os.stat()" for "createmode" initialization via vfs
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17725
diff changeset
330 return os.stat(self.join(path))
7cb7e17c23b2 store: invoke "os.stat()" for "createmode" initialization via vfs
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17725
diff changeset
331
49722
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
332 def unlink(self, path: Optional[bytes] = None):
19895
37c0d93fb166 bookmarks: use "vfs.unlink()" instead of "util.unlink()"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 19655
diff changeset
333 return util.unlink(self.join(path))
37c0d93fb166 bookmarks: use "vfs.unlink()" instead of "util.unlink()"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 19655
diff changeset
334
49722
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
335 def tryunlink(self, path: Optional[bytes] = None):
31548
fad440db3565 vfs: add tryunlink method
Ryan McElroy <rmcelroy@fb.com>
parents: 31318
diff changeset
336 """Attempt to remove a file, ignoring missing file errors."""
51428
187c5769a629 vfs: have tryunlink tell what it did
Georges Racinet <georges.racinet@octobus.net>
parents: 50952
diff changeset
337 return util.tryunlink(self.join(path))
31548
fad440db3565 vfs: add tryunlink method
Ryan McElroy <rmcelroy@fb.com>
parents: 31318
diff changeset
338
49722
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
339 def unlinkpath(
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
340 self, path: Optional[bytes] = None, ignoremissing=False, rmdir=True
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
341 ):
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43027
diff changeset
342 return util.unlinkpath(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43027
diff changeset
343 self.join(path), ignoremissing=ignoremissing, rmdir=rmdir
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43027
diff changeset
344 )
21716
90f9be5adade vfs: add unlinkpath to vfs
Chinmay Joshi <c@chinmayjoshi.com>
parents: 21563
diff changeset
345
49722
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
346 def utime(self, path: Optional[bytes] = None, t=None):
19896
af03279c766a bookmarks: use "vfs.utime()" instead of "os.utime()"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 19895
diff changeset
347 return os.utime(self.join(path), t)
af03279c766a bookmarks: use "vfs.utime()" instead of "os.utime()"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 19895
diff changeset
348
49722
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
349 def walk(self, path: Optional[bytes] = None, onerror=None):
24725
ee751d47cf2c vfs: add walk
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 24723
diff changeset
350 """Yield (dirpath, dirs, files) tuple for each directories under path
ee751d47cf2c vfs: add walk
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 24723
diff changeset
351
ee751d47cf2c vfs: add walk
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 24723
diff changeset
352 ``dirpath`` is relative one from the root of this vfs. This
ee751d47cf2c vfs: add walk
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 24723
diff changeset
353 uses ``os.sep`` as path separator, even you specify POSIX
ee751d47cf2c vfs: add walk
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 24723
diff changeset
354 style ``path``.
ee751d47cf2c vfs: add walk
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 24723
diff changeset
355
ee751d47cf2c vfs: add walk
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 24723
diff changeset
356 "The root of this vfs" is represented as empty ``dirpath``.
ee751d47cf2c vfs: add walk
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 24723
diff changeset
357 """
ee751d47cf2c vfs: add walk
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 24723
diff changeset
358 root = os.path.normpath(self.join(None))
ee751d47cf2c vfs: add walk
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 24723
diff changeset
359 # when dirpath == root, dirpath[prefixlen:] becomes empty
ee751d47cf2c vfs: add walk
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 24723
diff changeset
360 # because len(dirpath) < prefixlen.
ee751d47cf2c vfs: add walk
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 24723
diff changeset
361 prefixlen = len(pathutil.normasprefix(root))
ee751d47cf2c vfs: add walk
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 24723
diff changeset
362 for dirpath, dirs, files in os.walk(self.join(path), onerror=onerror):
ee751d47cf2c vfs: add walk
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 24723
diff changeset
363 yield (dirpath[prefixlen:], dirs, files)
ee751d47cf2c vfs: add walk
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 24723
diff changeset
364
27895
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
365 @contextlib.contextmanager
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
366 def backgroundclosing(self, ui, expectedcount=-1):
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
367 """Allow files to be closed asynchronously.
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
368
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
369 When this context manager is active, ``backgroundclose`` can be passed
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
370 to ``__call__``/``open`` to result in the file possibly being closed
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
371 asynchronously, on a background thread.
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
372 """
35432
60f2a215faa7 workers: don't use backgroundfilecloser in threads
Wojciech Lis <wlis@fb.com>
parents: 34645
diff changeset
373 # Sharing backgroundfilecloser between threads is complex and using
60f2a215faa7 workers: don't use backgroundfilecloser in threads
Wojciech Lis <wlis@fb.com>
parents: 34645
diff changeset
374 # multiple instances puts us at risk of running out of file descriptors
60f2a215faa7 workers: don't use backgroundfilecloser in threads
Wojciech Lis <wlis@fb.com>
parents: 34645
diff changeset
375 # only allow to use backgroundfilecloser when in main thread.
51920
1edac12af730 vfs: modernize the detection of the main thread
Matt Harbison <matt_harbison@yahoo.com>
parents: 51901
diff changeset
376 if threading.current_thread() is not threading.main_thread():
35432
60f2a215faa7 workers: don't use backgroundfilecloser in threads
Wojciech Lis <wlis@fb.com>
parents: 34645
diff changeset
377 yield
60f2a215faa7 workers: don't use backgroundfilecloser in threads
Wojciech Lis <wlis@fb.com>
parents: 34645
diff changeset
378 return
27895
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
379 vfs = getattr(self, 'vfs', self)
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
380 if getattr(vfs, '_backgroundfilecloser', None):
29389
98e8313dcd9e i18n: translate abort messages
liscju <piotr.listkiewicz@gmail.com>
parents: 29373
diff changeset
381 raise error.Abort(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
382 _(b'can only have 1 active background file closer')
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43027
diff changeset
383 )
27895
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
384
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
385 with backgroundfilecloser(ui, expectedcount=expectedcount) as bfc:
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
386 try:
43530
fc19f8ab8199 vfs: suppress some pytype errors around us using a private attribute
Augie Fackler <augie@google.com>
parents: 43516
diff changeset
387 vfs._backgroundfilecloser = (
fc19f8ab8199 vfs: suppress some pytype errors around us using a private attribute
Augie Fackler <augie@google.com>
parents: 43516
diff changeset
388 bfc # pytype: disable=attribute-error
fc19f8ab8199 vfs: suppress some pytype errors around us using a private attribute
Augie Fackler <augie@google.com>
parents: 43516
diff changeset
389 )
27895
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
390 yield bfc
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
391 finally:
43530
fc19f8ab8199 vfs: suppress some pytype errors around us using a private attribute
Augie Fackler <augie@google.com>
parents: 43516
diff changeset
392 vfs._backgroundfilecloser = (
fc19f8ab8199 vfs: suppress some pytype errors around us using a private attribute
Augie Fackler <augie@google.com>
parents: 43516
diff changeset
393 None # pytype: disable=attribute-error
fc19f8ab8199 vfs: suppress some pytype errors around us using a private attribute
Augie Fackler <augie@google.com>
parents: 43516
diff changeset
394 )
27895
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
395
47449
9ab54aa56982 vfs: add a `register_file` method on the vfs class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47192
diff changeset
396 def register_file(self, path):
9ab54aa56982 vfs: add a `register_file` method on the vfs class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47192
diff changeset
397 """generic hook point to lets fncache steer its stew"""
9ab54aa56982 vfs: add a `register_file` method on the vfs class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47192
diff changeset
398
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43027
diff changeset
399
17649
f65c6a5f256c scmutil: rename classes from "opener" to "vfs"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17561
diff changeset
400 class vfs(abstractvfs):
45957
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 44575
diff changeset
401 """Operate files relative to a base directory
13970
d13913355390 move opener from util to scmutil
Adrian Buehlmann <adrian@cadifra.com>
parents: 13962
diff changeset
402
d13913355390 move opener from util to scmutil
Adrian Buehlmann <adrian@cadifra.com>
parents: 13962
diff changeset
403 This class is used to hide the details of COW semantics and
d13913355390 move opener from util to scmutil
Adrian Buehlmann <adrian@cadifra.com>
parents: 13962
diff changeset
404 remote file access from higher level code.
33647
377e8ddaebef pathauditor: disable cache of audited paths by default (issue5628)
Yuya Nishihara <yuya@tcha.org>
parents: 33435
diff changeset
405
377e8ddaebef pathauditor: disable cache of audited paths by default (issue5628)
Yuya Nishihara <yuya@tcha.org>
parents: 33435
diff changeset
406 'cacheaudited' should be enabled only if (a) vfs object is short-lived, or
377e8ddaebef pathauditor: disable cache of audited paths by default (issue5628)
Yuya Nishihara <yuya@tcha.org>
parents: 33435
diff changeset
407 (b) the base directory is managed by hg and considered sort-of append-only.
377e8ddaebef pathauditor: disable cache of audited paths by default (issue5628)
Yuya Nishihara <yuya@tcha.org>
parents: 33435
diff changeset
408 See pathutil.pathauditor() for details.
45957
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 44575
diff changeset
409 """
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43027
diff changeset
410
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43027
diff changeset
411 def __init__(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43027
diff changeset
412 self,
49722
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
413 base: bytes,
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43027
diff changeset
414 audit=True,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43027
diff changeset
415 cacheaudited=False,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43027
diff changeset
416 expandpath=False,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43027
diff changeset
417 realpath=False,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43027
diff changeset
418 ):
18945
e75b72fffdfe vfs: split "expand" into "realpath"/"expandpath" to apply each separately
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 18900
diff changeset
419 if expandpath:
e75b72fffdfe vfs: split "expand" into "realpath"/"expandpath" to apply each separately
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 18900
diff changeset
420 base = util.expandpath(base)
e75b72fffdfe vfs: split "expand" into "realpath"/"expandpath" to apply each separately
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 18900
diff changeset
421 if realpath:
e75b72fffdfe vfs: split "expand" into "realpath"/"expandpath" to apply each separately
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 18900
diff changeset
422 base = os.path.realpath(base)
13970
d13913355390 move opener from util to scmutil
Adrian Buehlmann <adrian@cadifra.com>
parents: 13962
diff changeset
423 self.base = base
33259
6fb5c5096887 vfs: drop the 'mustaudit' API
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 33257
diff changeset
424 self._audit = audit
6fb5c5096887 vfs: drop the 'mustaudit' API
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 33257
diff changeset
425 if audit:
33647
377e8ddaebef pathauditor: disable cache of audited paths by default (issue5628)
Yuya Nishihara <yuya@tcha.org>
parents: 33435
diff changeset
426 self.audit = pathutil.pathauditor(self.base, cached=cacheaudited)
13970
d13913355390 move opener from util to scmutil
Adrian Buehlmann <adrian@cadifra.com>
parents: 13962
diff changeset
427 else:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43027
diff changeset
428 self.audit = lambda path, mode=None: True
33259
6fb5c5096887 vfs: drop the 'mustaudit' API
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 33257
diff changeset
429 self.createmode = None
6fb5c5096887 vfs: drop the 'mustaudit' API
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 33257
diff changeset
430 self._trustnlink = None
43027
3518da504303 vfs: give all vfs an options attribute by default
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41098
diff changeset
431 self.options = {}
17554
5450c8ad9d98 scmutil: turn opener._audit into a property, mustaudit
Bryan O'Sullivan <bryano@fb.com>
parents: 17248
diff changeset
432
13970
d13913355390 move opener from util to scmutil
Adrian Buehlmann <adrian@cadifra.com>
parents: 13962
diff changeset
433 @util.propertycache
49722
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
434 def _cansymlink(self) -> bool:
13970
d13913355390 move opener from util to scmutil
Adrian Buehlmann <adrian@cadifra.com>
parents: 13962
diff changeset
435 return util.checklink(self.base)
d13913355390 move opener from util to scmutil
Adrian Buehlmann <adrian@cadifra.com>
parents: 13962
diff changeset
436
18192
f9a89bdd64a6 scmutil: don't try to match modes on filesystems without modes (issue3740)
Matt Mackall <mpm@selenic.com>
parents: 17850
diff changeset
437 @util.propertycache
f9a89bdd64a6 scmutil: don't try to match modes on filesystems without modes (issue3740)
Matt Mackall <mpm@selenic.com>
parents: 17850
diff changeset
438 def _chmod(self):
f9a89bdd64a6 scmutil: don't try to match modes on filesystems without modes (issue3740)
Matt Mackall <mpm@selenic.com>
parents: 17850
diff changeset
439 return util.checkexec(self.base)
f9a89bdd64a6 scmutil: don't try to match modes on filesystems without modes (issue3740)
Matt Mackall <mpm@selenic.com>
parents: 17850
diff changeset
440
17763
13070de77c86 vfs: backout fchmod change from 76b73ce0ffac
Matt Mackall <mpm@selenic.com>
parents: 17752
diff changeset
441 def _fixfilemode(self, name):
18192
f9a89bdd64a6 scmutil: don't try to match modes on filesystems without modes (issue3740)
Matt Mackall <mpm@selenic.com>
parents: 17850
diff changeset
442 if self.createmode is None or not self._chmod:
13970
d13913355390 move opener from util to scmutil
Adrian Buehlmann <adrian@cadifra.com>
parents: 13962
diff changeset
443 return
25658
e93036747902 global: mass rewrite to use modern octal syntax
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25629
diff changeset
444 os.chmod(name, self.createmode & 0o666)
13970
d13913355390 move opener from util to scmutil
Adrian Buehlmann <adrian@cadifra.com>
parents: 13962
diff changeset
445
49722
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
446 def _auditpath(self, path, mode) -> None:
40788
34f15db81cf0 vfs: extract the audit path logic into a submethod
Boris Feld <boris.feld@octobus.net>
parents: 39909
diff changeset
447 if self._audit:
41093
c8006a25b845 vfs: makes all audited path relative
Boris Feld <boris.feld@octobus.net>
parents: 40999
diff changeset
448 if os.path.isabs(path) and path.startswith(self.base):
c8006a25b845 vfs: makes all audited path relative
Boris Feld <boris.feld@octobus.net>
parents: 40999
diff changeset
449 path = os.path.relpath(path, self.base)
40788
34f15db81cf0 vfs: extract the audit path logic into a submethod
Boris Feld <boris.feld@octobus.net>
parents: 39909
diff changeset
450 r = util.checkosfilename(path)
34f15db81cf0 vfs: extract the audit path logic into a submethod
Boris Feld <boris.feld@octobus.net>
parents: 39909
diff changeset
451 if r:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
452 raise error.Abort(b"%s: %r" % (r, path))
40788
34f15db81cf0 vfs: extract the audit path logic into a submethod
Boris Feld <boris.feld@octobus.net>
parents: 39909
diff changeset
453 self.audit(path, mode=mode)
34f15db81cf0 vfs: extract the audit path logic into a submethod
Boris Feld <boris.feld@octobus.net>
parents: 39909
diff changeset
454
49982
b7cf91ef03ba merge: skip syntactic path checks in [_checkunknownfile]
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 49723
diff changeset
455 def isfileorlink_checkdir(
b7cf91ef03ba merge: skip syntactic path checks in [_checkunknownfile]
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 49723
diff changeset
456 self, dircache, path: Optional[bytes] = None
b7cf91ef03ba merge: skip syntactic path checks in [_checkunknownfile]
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 49723
diff changeset
457 ) -> bool:
b7cf91ef03ba merge: skip syntactic path checks in [_checkunknownfile]
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 49723
diff changeset
458 """return True if the path is a regular file or a symlink and
b7cf91ef03ba merge: skip syntactic path checks in [_checkunknownfile]
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 49723
diff changeset
459 the directories along the path are "normal", that is
49985
bc83ebe07bf0 pathauditor: make _checkfs_exists a static method
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 49982
diff changeset
460 not symlinks or nested hg repositories.
bc83ebe07bf0 pathauditor: make _checkfs_exists a static method
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 49982
diff changeset
461
bc83ebe07bf0 pathauditor: make _checkfs_exists a static method
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 49982
diff changeset
462 Ignores the `_audit` setting, and checks the directories regardless.
bc83ebe07bf0 pathauditor: make _checkfs_exists a static method
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 49982
diff changeset
463 `dircache` is used to cache the directory checks.
bc83ebe07bf0 pathauditor: make _checkfs_exists a static method
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 49982
diff changeset
464 """
49982
b7cf91ef03ba merge: skip syntactic path checks in [_checkunknownfile]
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 49723
diff changeset
465 try:
b7cf91ef03ba merge: skip syntactic path checks in [_checkunknownfile]
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 49723
diff changeset
466 for prefix in pathutil.finddirs_rev_noroot(util.localpath(path)):
b7cf91ef03ba merge: skip syntactic path checks in [_checkunknownfile]
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 49723
diff changeset
467 if prefix in dircache:
b7cf91ef03ba merge: skip syntactic path checks in [_checkunknownfile]
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 49723
diff changeset
468 res = dircache[prefix]
b7cf91ef03ba merge: skip syntactic path checks in [_checkunknownfile]
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 49723
diff changeset
469 else:
49985
bc83ebe07bf0 pathauditor: make _checkfs_exists a static method
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 49982
diff changeset
470 res = pathutil.pathauditor._checkfs_exists(
bc83ebe07bf0 pathauditor: make _checkfs_exists a static method
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 49982
diff changeset
471 self.base, prefix, path
bc83ebe07bf0 pathauditor: make _checkfs_exists a static method
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 49982
diff changeset
472 )
49982
b7cf91ef03ba merge: skip syntactic path checks in [_checkunknownfile]
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 49723
diff changeset
473 dircache[prefix] = res
b7cf91ef03ba merge: skip syntactic path checks in [_checkunknownfile]
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 49723
diff changeset
474 if not res:
b7cf91ef03ba merge: skip syntactic path checks in [_checkunknownfile]
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 49723
diff changeset
475 return False
b7cf91ef03ba merge: skip syntactic path checks in [_checkunknownfile]
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 49723
diff changeset
476 except (OSError, error.Abort):
b7cf91ef03ba merge: skip syntactic path checks in [_checkunknownfile]
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 49723
diff changeset
477 return False
b7cf91ef03ba merge: skip syntactic path checks in [_checkunknownfile]
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 49723
diff changeset
478 return self.isfileorlink(path)
b7cf91ef03ba merge: skip syntactic path checks in [_checkunknownfile]
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 49723
diff changeset
479
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43027
diff changeset
480 def __call__(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43027
diff changeset
481 self,
49722
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
482 path: bytes,
49723
2506c3ac73f4 vfs: make the default opener mode binary
Matt Harbison <matt_harbison@yahoo.com>
parents: 49722
diff changeset
483 mode: bytes = b"rb",
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43027
diff changeset
484 atomictemp=False,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43027
diff changeset
485 notindexed=False,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43027
diff changeset
486 backgroundclose=False,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43027
diff changeset
487 checkambig=False,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43027
diff changeset
488 auditpath=True,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43027
diff changeset
489 makeparentdirs=True,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43027
diff changeset
490 ):
45957
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 44575
diff changeset
491 """Open ``path`` file, which is relative to vfs root.
23370
46265d0f0c7b vfs: add "notindexed" argument to invoke "ensuredir" with it in write mode
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 23368
diff changeset
492
40825
03bca908d9fb vfs: add option to not create parent directories implicitly
Yuya Nishihara <yuya@tcha.org>
parents: 40788
diff changeset
493 By default, parent directories are created as needed. Newly created
03bca908d9fb vfs: add option to not create parent directories implicitly
Yuya Nishihara <yuya@tcha.org>
parents: 40788
diff changeset
494 directories are marked as "not to be indexed by the content indexing
03bca908d9fb vfs: add option to not create parent directories implicitly
Yuya Nishihara <yuya@tcha.org>
parents: 40788
diff changeset
495 service", if ``notindexed`` is specified for "write" mode access.
03bca908d9fb vfs: add option to not create parent directories implicitly
Yuya Nishihara <yuya@tcha.org>
parents: 40788
diff changeset
496 Set ``makeparentdirs=False`` to not create directories implicitly.
27895
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
497
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
498 If ``backgroundclose`` is passed, the file may be closed asynchronously.
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
499 It can only be used if the ``self.backgroundclosing()`` context manager
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
500 is active. This should only be specified if the following criteria hold:
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
501
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
502 1. There is a potential for writing thousands of files. Unless you
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
503 are writing thousands of files, the performance benefits of
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
504 asynchronously closing files is not realized.
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
505 2. Files are opened exactly once for the ``backgroundclosing``
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
506 active duration and are therefore free of race conditions between
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
507 closing a file on a background thread and reopening it. (If the
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
508 file were opened multiple times, there could be unflushed data
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
509 because the original file handle hasn't been flushed/closed yet.)
29202
76f1ea360c7e vfs: make atomictempfile avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29119
diff changeset
510
44575
77d48738b8e0 vfs: fix typo in comment (remove extra "l")
Kyle Lippincott <spectral@google.com>
parents: 43554
diff changeset
511 ``checkambig`` argument is passed to atomictempfile (valid
29367
4e6e280e238f doc: describe detail about checkambig optional argument
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29341
diff changeset
512 only for writing), and is useful only if target file is
4e6e280e238f doc: describe detail about checkambig optional argument
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29341
diff changeset
513 guarded by any lock (e.g. repo.lock or repo.wlock).
33282
d1db7af81548 vfs: add explanation about cost of checkambig=True in corner case
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 33281
diff changeset
514
d1db7af81548 vfs: add explanation about cost of checkambig=True in corner case
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 33281
diff changeset
515 To avoid file stat ambiguity forcibly, checkambig=True involves
d1db7af81548 vfs: add explanation about cost of checkambig=True in corner case
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 33281
diff changeset
516 copying ``path`` file opened in "append" mode (e.g. for
d1db7af81548 vfs: add explanation about cost of checkambig=True in corner case
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 33281
diff changeset
517 truncation), if it is owned by another. Therefore, use
d1db7af81548 vfs: add explanation about cost of checkambig=True in corner case
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 33281
diff changeset
518 combination of append mode and checkambig=True only in limited
d1db7af81548 vfs: add explanation about cost of checkambig=True in corner case
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 33281
diff changeset
519 cases (see also issue5418 and issue5584 for detail).
45957
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 44575
diff changeset
520 """
33257
15e9cbe6ae49 vfs: simplify path audit disabling in stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 33234
diff changeset
521 if auditpath:
40788
34f15db81cf0 vfs: extract the audit path logic into a submethod
Boris Feld <boris.feld@octobus.net>
parents: 39909
diff changeset
522 self._auditpath(path, mode)
16199
8181bd808dc5 scmutil: add join method to opener to construct path relative to base
Idan Kamara <idankk86@gmail.com>
parents: 16198
diff changeset
523 f = self.join(path)
13970
d13913355390 move opener from util to scmutil
Adrian Buehlmann <adrian@cadifra.com>
parents: 13962
diff changeset
524
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
525 if b"b" not in mode:
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
526 mode += b"b" # for that other OS
13970
d13913355390 move opener from util to scmutil
Adrian Buehlmann <adrian@cadifra.com>
parents: 13962
diff changeset
527
d13913355390 move opener from util to scmutil
Adrian Buehlmann <adrian@cadifra.com>
parents: 13962
diff changeset
528 nlink = -1
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
529 if mode not in (b'r', b'rb'):
17937
3cb032d50447 vfs: optimize __call__ by not calling util.split for reads
Adrian Buehlmann <adrian@cadifra.com>
parents: 17850
diff changeset
530 dirname, basename = util.split(f)
3cb032d50447 vfs: optimize __call__ by not calling util.split for reads
Adrian Buehlmann <adrian@cadifra.com>
parents: 17850
diff changeset
531 # If basename is empty, then the path is malformed because it points
3cb032d50447 vfs: optimize __call__ by not calling util.split for reads
Adrian Buehlmann <adrian@cadifra.com>
parents: 17850
diff changeset
532 # to a directory. Let the posixfile() call below raise IOError.
3cb032d50447 vfs: optimize __call__ by not calling util.split for reads
Adrian Buehlmann <adrian@cadifra.com>
parents: 17850
diff changeset
533 if basename:
3cb032d50447 vfs: optimize __call__ by not calling util.split for reads
Adrian Buehlmann <adrian@cadifra.com>
parents: 17850
diff changeset
534 if atomictemp:
40825
03bca908d9fb vfs: add option to not create parent directories implicitly
Yuya Nishihara <yuya@tcha.org>
parents: 40788
diff changeset
535 if makeparentdirs:
03bca908d9fb vfs: add option to not create parent directories implicitly
Yuya Nishihara <yuya@tcha.org>
parents: 40788
diff changeset
536 util.makedirs(dirname, self.createmode, notindexed)
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43027
diff changeset
537 return util.atomictempfile(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43027
diff changeset
538 f, mode, self.createmode, checkambig=checkambig
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43027
diff changeset
539 )
17937
3cb032d50447 vfs: optimize __call__ by not calling util.split for reads
Adrian Buehlmann <adrian@cadifra.com>
parents: 17850
diff changeset
540 try:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
541 if b'w' in mode:
17937
3cb032d50447 vfs: optimize __call__ by not calling util.split for reads
Adrian Buehlmann <adrian@cadifra.com>
parents: 17850
diff changeset
542 util.unlink(f)
3cb032d50447 vfs: optimize __call__ by not calling util.split for reads
Adrian Buehlmann <adrian@cadifra.com>
parents: 17850
diff changeset
543 nlink = 0
3cb032d50447 vfs: optimize __call__ by not calling util.split for reads
Adrian Buehlmann <adrian@cadifra.com>
parents: 17850
diff changeset
544 else:
3cb032d50447 vfs: optimize __call__ by not calling util.split for reads
Adrian Buehlmann <adrian@cadifra.com>
parents: 17850
diff changeset
545 # nlinks() may behave differently for files on Windows
3cb032d50447 vfs: optimize __call__ by not calling util.split for reads
Adrian Buehlmann <adrian@cadifra.com>
parents: 17850
diff changeset
546 # shares if the file is open.
27706
22e362da27cf scmutil: use context managers for file handles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27651
diff changeset
547 with util.posixfile(f):
22e362da27cf scmutil: use context managers for file handles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27651
diff changeset
548 nlink = util.nlinks(f)
22e362da27cf scmutil: use context managers for file handles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27651
diff changeset
549 if nlink < 1:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43027
diff changeset
550 nlink = 2 # force mktempcopy (issue1922)
49314
2e726c934fcd py3: catch FileNotFoundError instead of checking errno == ENOENT
Manuel Jacob <me@manueljacob.de>
parents: 49037
diff changeset
551 except FileNotFoundError:
13970
d13913355390 move opener from util to scmutil
Adrian Buehlmann <adrian@cadifra.com>
parents: 13962
diff changeset
552 nlink = 0
40825
03bca908d9fb vfs: add option to not create parent directories implicitly
Yuya Nishihara <yuya@tcha.org>
parents: 40788
diff changeset
553 if makeparentdirs:
03bca908d9fb vfs: add option to not create parent directories implicitly
Yuya Nishihara <yuya@tcha.org>
parents: 40788
diff changeset
554 util.makedirs(dirname, self.createmode, notindexed)
17937
3cb032d50447 vfs: optimize __call__ by not calling util.split for reads
Adrian Buehlmann <adrian@cadifra.com>
parents: 17850
diff changeset
555 if nlink > 0:
3cb032d50447 vfs: optimize __call__ by not calling util.split for reads
Adrian Buehlmann <adrian@cadifra.com>
parents: 17850
diff changeset
556 if self._trustnlink is None:
3cb032d50447 vfs: optimize __call__ by not calling util.split for reads
Adrian Buehlmann <adrian@cadifra.com>
parents: 17850
diff changeset
557 self._trustnlink = nlink > 1 or util.checknlink(f)
3cb032d50447 vfs: optimize __call__ by not calling util.split for reads
Adrian Buehlmann <adrian@cadifra.com>
parents: 17850
diff changeset
558 if nlink > 1 or not self._trustnlink:
3cb032d50447 vfs: optimize __call__ by not calling util.split for reads
Adrian Buehlmann <adrian@cadifra.com>
parents: 17850
diff changeset
559 util.rename(util.mktempcopy(f), f)
13970
d13913355390 move opener from util to scmutil
Adrian Buehlmann <adrian@cadifra.com>
parents: 13962
diff changeset
560 fp = util.posixfile(f, mode)
d13913355390 move opener from util to scmutil
Adrian Buehlmann <adrian@cadifra.com>
parents: 13962
diff changeset
561 if nlink == 0:
17763
13070de77c86 vfs: backout fchmod change from 76b73ce0ffac
Matt Mackall <mpm@selenic.com>
parents: 17752
diff changeset
562 self._fixfilemode(f)
27895
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
563
30000
9766d88c2465 vfs: use checkambigatclosing in checkambig=True but atomictemp=False case
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29999
diff changeset
564 if checkambig:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
565 if mode in (b'r', b'rb'):
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43027
diff changeset
566 raise error.Abort(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43027
diff changeset
567 _(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
568 b'implementation error: mode %s is not'
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
569 b' valid for checkambig=True'
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43027
diff changeset
570 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43027
diff changeset
571 % mode
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43027
diff changeset
572 )
30000
9766d88c2465 vfs: use checkambigatclosing in checkambig=True but atomictemp=False case
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29999
diff changeset
573 fp = checkambigatclosing(fp)
9766d88c2465 vfs: use checkambigatclosing in checkambig=True but atomictemp=False case
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29999
diff changeset
574
51920
1edac12af730 vfs: modernize the detection of the main thread
Matt Harbison <matt_harbison@yahoo.com>
parents: 51901
diff changeset
575 if (
1edac12af730 vfs: modernize the detection of the main thread
Matt Harbison <matt_harbison@yahoo.com>
parents: 51901
diff changeset
576 backgroundclose
1edac12af730 vfs: modernize the detection of the main thread
Matt Harbison <matt_harbison@yahoo.com>
parents: 51901
diff changeset
577 and threading.current_thread() is threading.main_thread()
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43027
diff changeset
578 ):
43532
da9ecbb10368 vfs: more attribute suppressions
Augie Fackler <augie@google.com>
parents: 43530
diff changeset
579 if (
da9ecbb10368 vfs: more attribute suppressions
Augie Fackler <augie@google.com>
parents: 43530
diff changeset
580 not self._backgroundfilecloser # pytype: disable=attribute-error
da9ecbb10368 vfs: more attribute suppressions
Augie Fackler <augie@google.com>
parents: 43530
diff changeset
581 ):
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43027
diff changeset
582 raise error.Abort(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43027
diff changeset
583 _(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
584 b'backgroundclose can only be used when a '
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
585 b'backgroundclosing context manager is active'
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43027
diff changeset
586 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43027
diff changeset
587 )
27895
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
588
43532
da9ecbb10368 vfs: more attribute suppressions
Augie Fackler <augie@google.com>
parents: 43530
diff changeset
589 fp = delayclosedfile(
da9ecbb10368 vfs: more attribute suppressions
Augie Fackler <augie@google.com>
parents: 43530
diff changeset
590 fp,
da9ecbb10368 vfs: more attribute suppressions
Augie Fackler <augie@google.com>
parents: 43530
diff changeset
591 self._backgroundfilecloser, # pytype: disable=attribute-error
da9ecbb10368 vfs: more attribute suppressions
Augie Fackler <augie@google.com>
parents: 43530
diff changeset
592 )
27895
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
593
13970
d13913355390 move opener from util to scmutil
Adrian Buehlmann <adrian@cadifra.com>
parents: 13962
diff changeset
594 return fp
d13913355390 move opener from util to scmutil
Adrian Buehlmann <adrian@cadifra.com>
parents: 13962
diff changeset
595
49722
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
596 def symlink(self, src: bytes, dst: bytes) -> None:
18327
4aecdb91443c scmutil: simplify vfs.audit - drop wrapped vfs.auditor
Mads Kiilerich <mads@kiilerich.com>
parents: 18316
diff changeset
597 self.audit(dst)
16199
8181bd808dc5 scmutil: add join method to opener to construct path relative to base
Idan Kamara <idankk86@gmail.com>
parents: 16198
diff changeset
598 linkname = self.join(dst)
31555
18b9d9b95719 vfs: use tryunlink
Ryan McElroy <rmcelroy@fb.com>
parents: 31548
diff changeset
599 util.tryunlink(linkname)
13970
d13913355390 move opener from util to scmutil
Adrian Buehlmann <adrian@cadifra.com>
parents: 13962
diff changeset
600
29017
07be86828e79 util: fix race in makedirs()
Adam Simpkins <simpkins@fb.com>
parents: 28819
diff changeset
601 util.makedirs(os.path.dirname(linkname), self.createmode)
13970
d13913355390 move opener from util to scmutil
Adrian Buehlmann <adrian@cadifra.com>
parents: 13962
diff changeset
602
14261
e3649bcca3f6 opener: rename _can_symlink to _cansymlink
Adrian Buehlmann <adrian@cadifra.com>
parents: 14236
diff changeset
603 if self._cansymlink:
13970
d13913355390 move opener from util to scmutil
Adrian Buehlmann <adrian@cadifra.com>
parents: 13962
diff changeset
604 try:
d13913355390 move opener from util to scmutil
Adrian Buehlmann <adrian@cadifra.com>
parents: 13962
diff changeset
605 os.symlink(src, linkname)
25660
328739ea70c3 global: mass rewrite to use modern exception syntax
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25658
diff changeset
606 except OSError as err:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43027
diff changeset
607 raise OSError(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43027
diff changeset
608 err.errno,
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
609 _(b'could not symlink to %r: %s')
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43027
diff changeset
610 % (src, encoding.strtolocal(err.strerror)),
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43027
diff changeset
611 linkname,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43027
diff changeset
612 )
13970
d13913355390 move opener from util to scmutil
Adrian Buehlmann <adrian@cadifra.com>
parents: 13962
diff changeset
613 else:
17768
9837cafc25b1 vfs: use self.write to write symlink placeholders
Matt Mackall <mpm@selenic.com>
parents: 17763
diff changeset
614 self.write(dst, src)
13971
bfeaa88b875d move canonpath from util to scmutil
Adrian Buehlmann <adrian@cadifra.com>
parents: 13970
diff changeset
615
49722
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
616 def join(self, path: Optional[bytes], *insidef: bytes) -> bytes:
17161
be016e96117a localrepo: use file API via vfs while ensuring repository directory
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17157
diff changeset
617 if path:
47804
98c3fa6a3ac2 vfs: always use / as file separator (issue6546)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47449
diff changeset
618 parts = [self.base, path]
98c3fa6a3ac2 vfs: always use / as file separator (issue6546)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47449
diff changeset
619 parts.extend(insidef)
98c3fa6a3ac2 vfs: always use / as file separator (issue6546)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47449
diff changeset
620 return self._join(*parts)
17681
a41fd730f230 scmutil: backout 83785bb56062 (issue3643)
Matt Mackall <mpm@selenic.com>
parents: 17675
diff changeset
621 else:
a41fd730f230 scmutil: backout 83785bb56062 (issue3643)
Matt Mackall <mpm@selenic.com>
parents: 17675
diff changeset
622 return self.base
16199
8181bd808dc5 scmutil: add join method to opener to construct path relative to base
Idan Kamara <idankk86@gmail.com>
parents: 16198
diff changeset
623
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43027
diff changeset
624
17649
f65c6a5f256c scmutil: rename classes from "opener" to "vfs"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17561
diff changeset
625 opener = vfs
f65c6a5f256c scmutil: rename classes from "opener" to "vfs"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17561
diff changeset
626
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43027
diff changeset
627
51921
f79f98733a5b vfs: use @abstractmethod instead of homebrewing abstract methods
Matt Harbison <matt_harbison@yahoo.com>
parents: 51920
diff changeset
628 class proxyvfs(abstractvfs, abc.ABC):
49722
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
629 def __init__(self, vfs: "vfs"):
17845
408ded42c5ec scmutil: abstract out mustaudit delegation
Bryan O'Sullivan <bryano@fb.com>
parents: 17821
diff changeset
630 self.vfs = vfs
408ded42c5ec scmutil: abstract out mustaudit delegation
Bryan O'Sullivan <bryano@fb.com>
parents: 17821
diff changeset
631
50365
d1d458fb96a5 vfsproxy: inherit the `createmode` attribute too
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 49985
diff changeset
632 @property
d1d458fb96a5 vfsproxy: inherit the `createmode` attribute too
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 49985
diff changeset
633 def createmode(self):
d1d458fb96a5 vfsproxy: inherit the `createmode` attribute too
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 49985
diff changeset
634 return self.vfs.createmode
d1d458fb96a5 vfsproxy: inherit the `createmode` attribute too
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 49985
diff changeset
635
41097
adee334d94cd vfs: handle _auditpath in proxyvfs
Boris Feld <boris.feld@octobus.net>
parents: 41096
diff changeset
636 def _auditpath(self, path, mode):
adee334d94cd vfs: handle _auditpath in proxyvfs
Boris Feld <boris.feld@octobus.net>
parents: 41096
diff changeset
637 return self.vfs._auditpath(path, mode)
adee334d94cd vfs: handle _auditpath in proxyvfs
Boris Feld <boris.feld@octobus.net>
parents: 41096
diff changeset
638
27879
52a4ad62b006 cleanup: use modern @property/@foo.setter property specification
Augie Fackler <augie@google.com>
parents: 27851
diff changeset
639 @property
29727
69109052d9ac auditvfs: forward options property from nested vfs
Augie Fackler <augie@google.com>
parents: 29417
diff changeset
640 def options(self):
69109052d9ac auditvfs: forward options property from nested vfs
Augie Fackler <augie@google.com>
parents: 29417
diff changeset
641 return self.vfs.options
69109052d9ac auditvfs: forward options property from nested vfs
Augie Fackler <augie@google.com>
parents: 29417
diff changeset
642
69109052d9ac auditvfs: forward options property from nested vfs
Augie Fackler <augie@google.com>
parents: 29417
diff changeset
643 @options.setter
69109052d9ac auditvfs: forward options property from nested vfs
Augie Fackler <augie@google.com>
parents: 29417
diff changeset
644 def options(self, value):
69109052d9ac auditvfs: forward options property from nested vfs
Augie Fackler <augie@google.com>
parents: 29417
diff changeset
645 self.vfs.options = value
69109052d9ac auditvfs: forward options property from nested vfs
Augie Fackler <augie@google.com>
parents: 29417
diff changeset
646
51618
b5500857e173 proxy-vfs: also proxy the `audit` attribute
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51428
diff changeset
647 @property
b5500857e173 proxy-vfs: also proxy the `audit` attribute
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51428
diff changeset
648 def audit(self):
b5500857e173 proxy-vfs: also proxy the `audit` attribute
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51428
diff changeset
649 return self.vfs.audit
b5500857e173 proxy-vfs: also proxy the `audit` attribute
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51428
diff changeset
650
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43027
diff changeset
651
41096
6498f0e03526 vfs: fix proxyvfs inheritance
Boris Feld <boris.feld@octobus.net>
parents: 41095
diff changeset
652 class filtervfs(proxyvfs, abstractvfs):
17649
f65c6a5f256c scmutil: rename classes from "opener" to "vfs"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17561
diff changeset
653 '''Wrapper vfs for filtering filenames with a function.'''
14090
e24b5e3c2f27 add filteropener abstraction for store openers
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 14089
diff changeset
654
49722
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
655 def __init__(self, vfs: "vfs", filter):
33412
a42369e04aee vfs: rename auditvfs to proxyvfs
Yuya Nishihara <yuya@tcha.org>
parents: 33282
diff changeset
656 proxyvfs.__init__(self, vfs)
14090
e24b5e3c2f27 add filteropener abstraction for store openers
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 14089
diff changeset
657 self._filter = filter
e24b5e3c2f27 add filteropener abstraction for store openers
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 14089
diff changeset
658
49722
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
659 def __call__(self, path: bytes, *args, **kwargs):
17846
f42cf30873dc scmutil: add mustaudit delegation to filtervfs (issue3673)
Bryan O'Sullivan <bryano@fb.com>
parents: 17845
diff changeset
660 return self.vfs(self._filter(path), *args, **kwargs)
14090
e24b5e3c2f27 add filteropener abstraction for store openers
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 14089
diff changeset
661
49722
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
662 def join(self, path: Optional[bytes], *insidef: bytes) -> bytes:
17725
ffd589d4b785 vfs: define "join()" in each classes derived from "abstractvfs"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17723
diff changeset
663 if path:
24628
a0b47885a1c5 vfs: make it possible to pass multiple path elements to join
Matt Harbison <matt_harbison@yahoo.com>
parents: 24582
diff changeset
664 return self.vfs.join(self._filter(self.vfs.reljoin(path, *insidef)))
17725
ffd589d4b785 vfs: define "join()" in each classes derived from "abstractvfs"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17723
diff changeset
665 else:
17846
f42cf30873dc scmutil: add mustaudit delegation to filtervfs (issue3673)
Bryan O'Sullivan <bryano@fb.com>
parents: 17845
diff changeset
666 return self.vfs.join(path)
17725
ffd589d4b785 vfs: define "join()" in each classes derived from "abstractvfs"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17723
diff changeset
667
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43027
diff changeset
668
17649
f65c6a5f256c scmutil: rename classes from "opener" to "vfs"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17561
diff changeset
669 filteropener = filtervfs
f65c6a5f256c scmutil: rename classes from "opener" to "vfs"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17561
diff changeset
670
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43027
diff changeset
671
41096
6498f0e03526 vfs: fix proxyvfs inheritance
Boris Feld <boris.feld@octobus.net>
parents: 41095
diff changeset
672 class readonlyvfs(proxyvfs):
18213
c38a62af000e vfs: add a read only vfs
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 18206
diff changeset
673 '''Wrapper vfs preventing any writing.'''
c38a62af000e vfs: add a read only vfs
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 18206
diff changeset
674
49722
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
675 def __init__(self, vfs: "vfs"):
33412
a42369e04aee vfs: rename auditvfs to proxyvfs
Yuya Nishihara <yuya@tcha.org>
parents: 33282
diff changeset
676 proxyvfs.__init__(self, vfs)
18213
c38a62af000e vfs: add a read only vfs
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 18206
diff changeset
677
49723
2506c3ac73f4 vfs: make the default opener mode binary
Matt Harbison <matt_harbison@yahoo.com>
parents: 49722
diff changeset
678 def __call__(self, path: bytes, mode: bytes = b'rb', *args, **kw):
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
679 if mode not in (b'r', b'rb'):
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
680 raise error.Abort(_(b'this vfs is read only'))
18213
c38a62af000e vfs: add a read only vfs
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 18206
diff changeset
681 return self.vfs(path, mode, *args, **kw)
c38a62af000e vfs: add a read only vfs
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 18206
diff changeset
682
49722
cc9a60050a07 typing: add basic type hints to vfs.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 49314
diff changeset
683 def join(self, path: Optional[bytes], *insidef: bytes) -> bytes:
26156
a112fffdb632 scmutil.readonlyvfs: implement join
Siddharth Agarwal <sid0@fb.com>
parents: 26098
diff changeset
684 return self.vfs.join(path, *insidef)
18213
c38a62af000e vfs: add a read only vfs
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 18206
diff changeset
685
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43027
diff changeset
686
51921
f79f98733a5b vfs: use @abstractmethod instead of homebrewing abstract methods
Matt Harbison <matt_harbison@yahoo.com>
parents: 51920
diff changeset
687 class closewrapbase(abc.ABC):
29998
0c40e64d6154 scmutil: factor out common logic of delayclosedfile to reuse it
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29813
diff changeset
688 """Base class of wrapper, which hooks closing
27895
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
689
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
690 Do not instantiate outside of the vfs layer.
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
691 """
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43027
diff changeset
692
29998
0c40e64d6154 scmutil: factor out common logic of delayclosedfile to reuse it
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29813
diff changeset
693 def __init__(self, fh):
43554
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43532
diff changeset
694 object.__setattr__(self, '_origfh', fh)
27895
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
695
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
696 def __getattr__(self, attr):
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
697 return getattr(self._origfh, attr)
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
698
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
699 def __setattr__(self, attr, value):
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
700 return setattr(self._origfh, attr, value)
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
701
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
702 def __delattr__(self, attr):
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
703 return delattr(self._origfh, attr)
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
704
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
705 def __enter__(self):
40468
8d9f366b7f19 vfs: ensure closewrapbase fh doesn't escape by entering context manager
Matt Harbison <matt_harbison@yahoo.com>
parents: 39909
diff changeset
706 self._origfh.__enter__()
8d9f366b7f19 vfs: ensure closewrapbase fh doesn't escape by entering context manager
Matt Harbison <matt_harbison@yahoo.com>
parents: 39909
diff changeset
707 return self
27895
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
708
51921
f79f98733a5b vfs: use @abstractmethod instead of homebrewing abstract methods
Matt Harbison <matt_harbison@yahoo.com>
parents: 51920
diff changeset
709 @abc.abstractmethod
27895
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
710 def __exit__(self, exc_type, exc_value, exc_tb):
51921
f79f98733a5b vfs: use @abstractmethod instead of homebrewing abstract methods
Matt Harbison <matt_harbison@yahoo.com>
parents: 51920
diff changeset
711 ...
29998
0c40e64d6154 scmutil: factor out common logic of delayclosedfile to reuse it
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29813
diff changeset
712
51921
f79f98733a5b vfs: use @abstractmethod instead of homebrewing abstract methods
Matt Harbison <matt_harbison@yahoo.com>
parents: 51920
diff changeset
713 @abc.abstractmethod
29998
0c40e64d6154 scmutil: factor out common logic of delayclosedfile to reuse it
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29813
diff changeset
714 def close(self):
51921
f79f98733a5b vfs: use @abstractmethod instead of homebrewing abstract methods
Matt Harbison <matt_harbison@yahoo.com>
parents: 51920
diff changeset
715 ...
29998
0c40e64d6154 scmutil: factor out common logic of delayclosedfile to reuse it
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29813
diff changeset
716
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43027
diff changeset
717
29998
0c40e64d6154 scmutil: factor out common logic of delayclosedfile to reuse it
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29813
diff changeset
718 class delayclosedfile(closewrapbase):
0c40e64d6154 scmutil: factor out common logic of delayclosedfile to reuse it
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29813
diff changeset
719 """Proxy for a file object whose close is delayed.
0c40e64d6154 scmutil: factor out common logic of delayclosedfile to reuse it
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29813
diff changeset
720
0c40e64d6154 scmutil: factor out common logic of delayclosedfile to reuse it
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29813
diff changeset
721 Do not instantiate outside of the vfs layer.
0c40e64d6154 scmutil: factor out common logic of delayclosedfile to reuse it
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29813
diff changeset
722 """
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43027
diff changeset
723
29998
0c40e64d6154 scmutil: factor out common logic of delayclosedfile to reuse it
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29813
diff changeset
724 def __init__(self, fh, closer):
0c40e64d6154 scmutil: factor out common logic of delayclosedfile to reuse it
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29813
diff changeset
725 super(delayclosedfile, self).__init__(fh)
43554
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43532
diff changeset
726 object.__setattr__(self, '_closer', closer)
29998
0c40e64d6154 scmutil: factor out common logic of delayclosedfile to reuse it
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29813
diff changeset
727
0c40e64d6154 scmutil: factor out common logic of delayclosedfile to reuse it
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29813
diff changeset
728 def __exit__(self, exc_type, exc_value, exc_tb):
27895
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
729 self._closer.close(self._origfh)
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
730
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
731 def close(self):
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
732 self._closer.close(self._origfh)
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
733
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43027
diff changeset
734
49037
642e31cb55f0 py3: use class X: instead of class X(object):
Gregory Szorc <gregory.szorc@gmail.com>
parents: 48966
diff changeset
735 class backgroundfilecloser:
27895
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
736 """Coordinates background closing of file handles on multiple threads."""
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43027
diff changeset
737
27895
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
738 def __init__(self, ui, expectedcount=-1):
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
739 self._running = False
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
740 self._entered = False
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
741 self._threads = []
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
742 self._threadexception = None
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
743
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
744 # Only Windows/NTFS has slow file closing. So only enable by default
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
745 # on that platform. But allow to be enabled elsewhere for testing.
34645
75979c8d4572 codemod: use pycompat.iswindows
Jun Wu <quark@fb.com>
parents: 34040
diff changeset
746 defaultenabled = pycompat.iswindows
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
747 enabled = ui.configbool(b'worker', b'backgroundclose', defaultenabled)
27895
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
748
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
749 if not enabled:
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
750 return
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
751
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
752 # There is overhead to starting and stopping the background threads.
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
753 # Don't do background processing unless the file count is large enough
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
754 # to justify it.
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
755 minfilecount = ui.configint(b'worker', b'backgroundcloseminfilecount')
27895
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
756 # FUTURE dynamically start background threads after minfilecount closes.
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
757 # (We don't currently have any callers that don't know their file count)
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
758 if expectedcount > 0 and expectedcount < minfilecount:
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
759 return
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
760
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
761 maxqueue = ui.configint(b'worker', b'backgroundclosemaxqueue')
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
762 threadcount = ui.configint(b'worker', b'backgroundclosethreadcount')
27895
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
763
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43027
diff changeset
764 ui.debug(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
765 b'starting %d threads for background file closing\n' % threadcount
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43027
diff changeset
766 )
27895
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
767
37890
8fb9985382be pycompat: export queue module instead of symbols in module (API)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35725
diff changeset
768 self._queue = pycompat.queue.Queue(maxsize=maxqueue)
27895
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
769 self._running = True
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
770
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
771 for i in range(threadcount):
43510
ffd632c224c3 vfs: another bytes-str confusion on thread name
Augie Fackler <augie@google.com>
parents: 43117
diff changeset
772 t = threading.Thread(target=self._worker, name='backgroundcloser')
27895
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
773 self._threads.append(t)
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
774 t.start()
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
775
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
776 def __enter__(self):
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
777 self._entered = True
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
778 return self
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
779
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
780 def __exit__(self, exc_type, exc_value, exc_tb):
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
781 self._running = False
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
782
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
783 # Wait for threads to finish closing so open files don't linger for
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
784 # longer than lifetime of context manager.
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
785 for t in self._threads:
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
786 t.join()
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
787
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
788 def _worker(self):
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
789 """Main routine for worker thread."""
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
790 while True:
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
791 try:
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
792 fh = self._queue.get(block=True, timeout=0.100)
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
793 # Need to catch or the thread will terminate and
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
794 # we could orphan file descriptors.
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
795 try:
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
796 fh.close()
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
797 except Exception as e:
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
798 # Stash so can re-raise from main thread later.
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
799 self._threadexception = e
37890
8fb9985382be pycompat: export queue module instead of symbols in module (API)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35725
diff changeset
800 except pycompat.queue.Empty:
27895
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
801 if not self._running:
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
802 break
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
803
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
804 def close(self, fh):
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
805 """Schedule a file for closing."""
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
806 if not self._entered:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43027
diff changeset
807 raise error.Abort(
43117
8ff1ecfadcd1 cleanup: join string literals that are already on one line
Martin von Zweigbergk <martinvonz@google.com>
parents: 43090
diff changeset
808 _(b'can only call close() when context manager active')
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43027
diff changeset
809 )
27895
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
810
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
811 # If a background thread encountered an exception, raise now so we fail
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
812 # fast. Otherwise we may potentially go on for minutes until the error
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
813 # is acted on.
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
814 if self._threadexception:
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
815 e = self._threadexception
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
816 self._threadexception = None
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
817 raise e
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
818
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
819 # If we're not actively running, close synchronously.
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
820 if not self._running:
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
821 fh.close()
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
822 return
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
823
2d6a89e79b48 scmutil: support background file closing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27879
diff changeset
824 self._queue.put(fh, block=True, timeout=None)
29999
57830bd0e787 scmutil: add file object wrapper class to check ambiguity at closing
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29998
diff changeset
825
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43027
diff changeset
826
29999
57830bd0e787 scmutil: add file object wrapper class to check ambiguity at closing
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29998
diff changeset
827 class checkambigatclosing(closewrapbase):
57830bd0e787 scmutil: add file object wrapper class to check ambiguity at closing
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29998
diff changeset
828 """Proxy for a file object, to avoid ambiguity of file stat
57830bd0e787 scmutil: add file object wrapper class to check ambiguity at closing
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29998
diff changeset
829
57830bd0e787 scmutil: add file object wrapper class to check ambiguity at closing
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29998
diff changeset
830 See also util.filestat for detail about "ambiguity of file stat".
57830bd0e787 scmutil: add file object wrapper class to check ambiguity at closing
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29998
diff changeset
831
57830bd0e787 scmutil: add file object wrapper class to check ambiguity at closing
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29998
diff changeset
832 This proxy is useful only if the target file is guarded by any
57830bd0e787 scmutil: add file object wrapper class to check ambiguity at closing
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29998
diff changeset
833 lock (e.g. repo.lock or repo.wlock)
57830bd0e787 scmutil: add file object wrapper class to check ambiguity at closing
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29998
diff changeset
834
57830bd0e787 scmutil: add file object wrapper class to check ambiguity at closing
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29998
diff changeset
835 Do not instantiate outside of the vfs layer.
57830bd0e787 scmutil: add file object wrapper class to check ambiguity at closing
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29998
diff changeset
836 """
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 43027
diff changeset
837
29999
57830bd0e787 scmutil: add file object wrapper class to check ambiguity at closing
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29998
diff changeset
838 def __init__(self, fh):
57830bd0e787 scmutil: add file object wrapper class to check ambiguity at closing
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29998
diff changeset
839 super(checkambigatclosing, self).__init__(fh)
43554
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43532
diff changeset
840 object.__setattr__(self, '_oldstat', util.filestat.frompath(fh.name))
29999
57830bd0e787 scmutil: add file object wrapper class to check ambiguity at closing
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29998
diff changeset
841
57830bd0e787 scmutil: add file object wrapper class to check ambiguity at closing
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29998
diff changeset
842 def _checkambig(self):
57830bd0e787 scmutil: add file object wrapper class to check ambiguity at closing
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29998
diff changeset
843 oldstat = self._oldstat
57830bd0e787 scmutil: add file object wrapper class to check ambiguity at closing
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29998
diff changeset
844 if oldstat.stat:
33280
646352291f5b vfs: copy if EPERM to avoid file stat ambiguity forcibly at closing
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 33259
diff changeset
845 _avoidambig(self._origfh.name, oldstat)
29999
57830bd0e787 scmutil: add file object wrapper class to check ambiguity at closing
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29998
diff changeset
846
57830bd0e787 scmutil: add file object wrapper class to check ambiguity at closing
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29998
diff changeset
847 def __exit__(self, exc_type, exc_value, exc_tb):
57830bd0e787 scmutil: add file object wrapper class to check ambiguity at closing
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29998
diff changeset
848 self._origfh.__exit__(exc_type, exc_value, exc_tb)
57830bd0e787 scmutil: add file object wrapper class to check ambiguity at closing
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29998
diff changeset
849 self._checkambig()
57830bd0e787 scmutil: add file object wrapper class to check ambiguity at closing
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29998
diff changeset
850
57830bd0e787 scmutil: add file object wrapper class to check ambiguity at closing
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29998
diff changeset
851 def close(self):
57830bd0e787 scmutil: add file object wrapper class to check ambiguity at closing
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29998
diff changeset
852 self._origfh.close()
57830bd0e787 scmutil: add file object wrapper class to check ambiguity at closing
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29998
diff changeset
853 self._checkambig()