Mercurial > public > mercurial-scm > hg-stable

annotate mercurial/util.py @ 52668:5cc8deb96b48

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
pyupgrade: modernize calls to superclass methods This is the `legacy` fixer in `pyupgrade`, with the loop yielding the offset of `yield` statements commented out.
author Matt Harbison <matt_harbison@yahoo.com>
date Sun, 05 Jan 2025 22:23:31 -0500
parents 24ee91ba9aa8
children e627cc25b6f3
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
17515
b5b38d21fe99 spelling: specific
timeless@mozdev.org
parents: 17237
diff changeset
1 # util.py - Mercurial utility functions and platform specific implementations
8226
8b2cd04a6e97 put license and copyright info into comment blocks
Martin Geisler <mg@lazybytes.net>
parents: 8225
diff changeset
2 #
8b2cd04a6e97 put license and copyright info into comment blocks
Martin Geisler <mg@lazybytes.net>
parents: 8225
diff changeset
3 # Copyright 2005 K. Thananchayan <thananck@yahoo.com>
46819
d4ba4d51f85f contributor: change mentions of mpm to olivia
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46714
diff changeset
4 # Copyright 2005-2007 Olivia Mackall <olivia@selenic.com>
8226
8b2cd04a6e97 put license and copyright info into comment blocks
Martin Geisler <mg@lazybytes.net>
parents: 8225
diff changeset
5 # Copyright 2006 Vadim Gelfer <vadim.gelfer@gmail.com>
8b2cd04a6e97 put license and copyright info into comment blocks
Martin Geisler <mg@lazybytes.net>
parents: 8225
diff changeset
6 #
8b2cd04a6e97 put license and copyright info into comment blocks
Martin Geisler <mg@lazybytes.net>
parents: 8225
diff changeset
7 # This software may be used and distributed according to the terms of the
10263
25e572394f5c Update license to GPLv2+
Matt Mackall <mpm@selenic.com>
parents: 9996
diff changeset
8 # GNU General Public License version 2 or any later version.
1082
ce96e316278a Update util.py docstrings, fix walk test
mpm@selenic.com
parents: 1081
diff changeset
9
17515
b5b38d21fe99 spelling: specific
timeless@mozdev.org
parents: 17237
diff changeset
10 """Mercurial utility functions and platform specific implementations.
1082
ce96e316278a Update util.py docstrings, fix walk test
mpm@selenic.com
parents: 1081
diff changeset
11
8227
0a9542703300 turn some comments back into module docstrings
Martin Geisler <mg@lazybytes.net>
parents: 8226
diff changeset
12 This contains helper routines that are independent of the SCM core and
0a9542703300 turn some comments back into module docstrings
Martin Geisler <mg@lazybytes.net>
parents: 8226
diff changeset
13 hide platform-specific details from the core.
1082
ce96e316278a Update util.py docstrings, fix walk test
mpm@selenic.com
parents: 1081
diff changeset
14 """
419
28511fc21073 [PATCH] file seperator handling for the other 'OS'
mpm@selenic.com
parents:
diff changeset
15
51901
f4733654f144 typing: add `from __future__ import annotations` to most files
Matt Harbison <matt_harbison@yahoo.com>
parents: 51888
diff changeset
16 from __future__ import annotations
27358
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
17
33814
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33763
diff changeset
18 import abc
27358
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
19 import collections
33446
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
20 import contextlib
27358
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
21 import errno
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
22 import gc
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
23 import hashlib
48964
5aafc3c5bdec py3: use io.BytesIO directly
Gregory Szorc <gregory.szorc@gmail.com>
parents: 48961
diff changeset
24 import io
34554
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34468
diff changeset
25 import itertools
44826
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44633
diff changeset
26 import locale
34302
3bb2a9f25fe9 util: add an mmapread method
Mark Thomas <mbthomas@fb.com>
parents: 34154
diff changeset
27 import mmap
27358
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
28 import os
49068
225659936fff util: restore the util.pickle symbol
Matt Harbison <matt_harbison@yahoo.com>
parents: 49037
diff changeset
29 import pickle # provides util.pickle symbol
21907
7e5dfa00e3c2 util: rename 're' to 'remod'
Siddharth Agarwal <sid0@fb.com>
parents: 21857
diff changeset
30 import re as remod
27358
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
31 import shutil
30428
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
32 import stat
27358
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
33 import sys
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
34 import time
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
35 import traceback
51780
ed28085827ec typing: explicitly type some `mercurial.util` eol code to avoid @overload
Matt Harbison <matt_harbison@yahoo.com>
parents: 51762
diff changeset
36 import typing
31950
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
37 import warnings
3769
96095d9ff1f8 Add encoding detection
Matt Mackall <mpm@selenic.com>
parents: 3767
diff changeset
38
51302
9d3721552b6c pytype: import typing directly
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50952
diff changeset
39 from typing import (
51587
a452807df09b nocg: make the utility work are both a decorator and context manager
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51428
diff changeset
40 Any,
51780
ed28085827ec typing: explicitly type some `mercurial.util` eol code to avoid @overload
Matt Harbison <matt_harbison@yahoo.com>
parents: 51762
diff changeset
41 BinaryIO,
ed28085827ec typing: explicitly type some `mercurial.util` eol code to avoid @overload
Matt Harbison <matt_harbison@yahoo.com>
parents: 51762
diff changeset
42 Callable,
51302
9d3721552b6c pytype: import typing directly
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50952
diff changeset
43 Iterable,
9d3721552b6c pytype: import typing directly
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50952
diff changeset
44 Iterator,
9d3721552b6c pytype: import typing directly
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50952
diff changeset
45 List,
9d3721552b6c pytype: import typing directly
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50952
diff changeset
46 Optional,
9d3721552b6c pytype: import typing directly
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50952
diff changeset
47 Tuple,
51924
1d95a87813ad typing: add type annotations to the `mercurial.util.filestat` class
Matt Harbison <matt_harbison@yahoo.com>
parents: 51917
diff changeset
48 Type,
1d95a87813ad typing: add type annotations to the `mercurial.util.filestat` class
Matt Harbison <matt_harbison@yahoo.com>
parents: 51917
diff changeset
49 TypeVar,
51302
9d3721552b6c pytype: import typing directly
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50952
diff changeset
50 )
9d3721552b6c pytype: import typing directly
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50952
diff changeset
51
47055
d55b71393907 node: replace nullid and friends with nodeconstants class
Joerg Sonnenberger <joerg@bec.de>
parents: 46926
diff changeset
52 from .node import hex
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
53 from .thirdparty import attr
51788
278af66e6595 typing: induce pytype to use the standard `attr` instead of the vendored copy
Matt Harbison <matt_harbison@yahoo.com>
parents: 51780
diff changeset
54
278af66e6595 typing: induce pytype to use the standard `attr` instead of the vendored copy
Matt Harbison <matt_harbison@yahoo.com>
parents: 51780
diff changeset
55 # Force pytype to use the non-vendored package
278af66e6595 typing: induce pytype to use the standard `attr` instead of the vendored copy
Matt Harbison <matt_harbison@yahoo.com>
parents: 51780
diff changeset
56 if typing.TYPE_CHECKING:
278af66e6595 typing: induce pytype to use the standard `attr` instead of the vendored copy
Matt Harbison <matt_harbison@yahoo.com>
parents: 51780
diff changeset
57 # noinspection PyPackageRequirements
278af66e6595 typing: induce pytype to use the standard `attr` instead of the vendored copy
Matt Harbison <matt_harbison@yahoo.com>
parents: 51780
diff changeset
58 import attr
278af66e6595 typing: induce pytype to use the standard `attr` instead of the vendored copy
Matt Harbison <matt_harbison@yahoo.com>
parents: 51780
diff changeset
59
39287
e00123f63410 util: make timedcm require the label (API)
Augie Fackler <augie@google.com>
parents: 39286
diff changeset
60 from hgdemandimport import tracing
27358
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
61 from . import (
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
62 encoding,
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
63 error,
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
64 i18n,
32406
a9c71d578a1c osutil: switch to policy importer
Yuya Nishihara <yuya@tcha.org>
parents: 32346
diff changeset
65 policy,
28818
6041fb8f2da8 pycompat: add empty and queue to handle py3 divergence
timeless <timeless@mozdev.org>
parents: 28497
diff changeset
66 pycompat,
51780
ed28085827ec typing: explicitly type some `mercurial.util` eol code to avoid @overload
Matt Harbison <matt_harbison@yahoo.com>
parents: 51762
diff changeset
67 typelib,
34468
192f7b126ed2 urllibcompat: move some adapters from pycompat to urllibcompat
Augie Fackler <augie@google.com>
parents: 34436
diff changeset
68 urllibcompat,
27358
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
69 )
51974
fa7059f031a9 interfaces: introduce and use a protocol class for the `base85` module
Matt Harbison <matt_harbison@yahoo.com>
parents: 51967
diff changeset
70 from .interfaces import (
fa7059f031a9 interfaces: introduce and use a protocol class for the `base85` module
Matt Harbison <matt_harbison@yahoo.com>
parents: 51967
diff changeset
71 modules as intmod,
fa7059f031a9 interfaces: introduce and use a protocol class for the `base85` module
Matt Harbison <matt_harbison@yahoo.com>
parents: 51967
diff changeset
72 )
37086
f99d64e8a4e4 stringutil: move generic string helpers to new module
Yuya Nishihara <yuya@tcha.org>
parents: 37085
diff changeset
73 from .utils import (
42041
3e47d1ec9da5 util: extract compression code in `mercurial.utils.compression`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41839
diff changeset
74 compression,
44060
a61287a95dc3 core: migrate uses of hashlib.sha1 to hashutil.sha1
Augie Fackler <augie@google.com>
parents: 44036
diff changeset
75 hashutil,
37121
5be286db5fb5 procutil: move process/executable management functions to new module
Yuya Nishihara <yuya@tcha.org>
parents: 37120
diff changeset
76 procutil,
37086
f99d64e8a4e4 stringutil: move generic string helpers to new module
Yuya Nishihara <yuya@tcha.org>
parents: 37085
diff changeset
77 stringutil,
f99d64e8a4e4 stringutil: move generic string helpers to new module
Yuya Nishihara <yuya@tcha.org>
parents: 37085
diff changeset
78 )
3769
96095d9ff1f8 Add encoding detection
Matt Mackall <mpm@selenic.com>
parents: 3767
diff changeset
79
51302
9d3721552b6c pytype: import typing directly
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50952
diff changeset
80 # keeps pyflakes happy
9d3721552b6c pytype: import typing directly
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50952
diff changeset
81 assert [
9d3721552b6c pytype: import typing directly
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50952
diff changeset
82 Iterable,
9d3721552b6c pytype: import typing directly
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50952
diff changeset
83 Iterator,
9d3721552b6c pytype: import typing directly
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50952
diff changeset
84 List,
9d3721552b6c pytype: import typing directly
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50952
diff changeset
85 Optional,
9d3721552b6c pytype: import typing directly
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50952
diff changeset
86 Tuple,
9d3721552b6c pytype: import typing directly
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50952
diff changeset
87 ]
46684
7711853110b9 typing: add some type annotations to mercurial/util.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 46114
diff changeset
88
7711853110b9 typing: add some type annotations to mercurial/util.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 46114
diff changeset
89
51974
fa7059f031a9 interfaces: introduce and use a protocol class for the `base85` module
Matt Harbison <matt_harbison@yahoo.com>
parents: 51967
diff changeset
90 base85: intmod.Base85 = policy.importmod('base85')
43554
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
91 osutil = policy.importmod('osutil')
32406
a9c71d578a1c osutil: switch to policy importer
Yuya Nishihara <yuya@tcha.org>
parents: 32346
diff changeset
92
32245
4462a981e8df base85: proxy through util module
Yuya Nishihara <yuya@tcha.org>
parents: 32203
diff changeset
93 b85decode = base85.b85decode
4462a981e8df base85: proxy through util module
Yuya Nishihara <yuya@tcha.org>
parents: 32203
diff changeset
94 b85encode = base85.b85encode
4462a981e8df base85: proxy through util module
Yuya Nishihara <yuya@tcha.org>
parents: 32203
diff changeset
95
31934
12aca6770046 util: make cookielib module available
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31878
diff changeset
96 cookielib = pycompat.cookielib
30480
00c9ac4ce816 util: rewrite pycompat imports to make pyflakes always happy
Yuya Nishihara <yuya@tcha.org>
parents: 30451
diff changeset
97 httplib = pycompat.httplib
37102
6ca5f825a0ca util: make safehasattr() a pycompat function
Yuya Nishihara <yuya@tcha.org>
parents: 37101
diff changeset
98 safehasattr = pycompat.safehasattr
30480
00c9ac4ce816 util: rewrite pycompat imports to make pyflakes always happy
Yuya Nishihara <yuya@tcha.org>
parents: 30451
diff changeset
99 socketserver = pycompat.socketserver
48964
5aafc3c5bdec py3: use io.BytesIO directly
Gregory Szorc <gregory.szorc@gmail.com>
parents: 48961
diff changeset
100 bytesio = io.BytesIO
36964
644a02f6b34f util: prefer "bytesio" to "stringio"
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36952
diff changeset
101 # TODO deprecate stringio name, as it is a lie on Python 3.
644a02f6b34f util: prefer "bytesio" to "stringio"
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36952
diff changeset
102 stringio = bytesio
30480
00c9ac4ce816 util: rewrite pycompat imports to make pyflakes always happy
Yuya Nishihara <yuya@tcha.org>
parents: 30451
diff changeset
103 xmlrpclib = pycompat.xmlrpclib
28883
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents: 28882
diff changeset
104
34468
192f7b126ed2 urllibcompat: move some adapters from pycompat to urllibcompat
Augie Fackler <augie@google.com>
parents: 34436
diff changeset
105 httpserver = urllibcompat.httpserver
192f7b126ed2 urllibcompat: move some adapters from pycompat to urllibcompat
Augie Fackler <augie@google.com>
parents: 34436
diff changeset
106 urlerr = urllibcompat.urlerr
192f7b126ed2 urllibcompat: move some adapters from pycompat to urllibcompat
Augie Fackler <augie@google.com>
parents: 34436
diff changeset
107 urlreq = urllibcompat.urlreq
192f7b126ed2 urllibcompat: move some adapters from pycompat to urllibcompat
Augie Fackler <augie@google.com>
parents: 34436
diff changeset
108
32131
377c74ef008d win32mbcs: avoid unintentional failure at colorization
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 31952
diff changeset
109 # workaround for win32mbcs
377c74ef008d win32mbcs: avoid unintentional failure at colorization
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 31952
diff changeset
110 _filenamebytestr = pycompat.bytestr
377c74ef008d win32mbcs: avoid unintentional failure at colorization
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 31952
diff changeset
111
34645
75979c8d4572 codemod: use pycompat.iswindows
Jun Wu <quark@fb.com>
parents: 34554
diff changeset
112 if pycompat.iswindows:
27358
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
113 from . import windows as platform
14912
ec46a7da9f2c util: move windows and posix wildcard imports to begin of file
Adrian Buehlmann <adrian@cadifra.com>
parents: 14911
diff changeset
114 else:
27358
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
115 from . import posix as platform
14926
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
116
27358
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
117 _ = i18n._
14926
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
118
47622
bb917eea1605 windows: introduce a `util.abspath` to replace os.path.abspath
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47452
diff changeset
119 abspath = platform.abspath
29530
3239e2fdd2e2 chgserver: extract utility to bind unix domain socket to long path
Yuya Nishihara <yuya@tcha.org>
parents: 29455
diff changeset
120 bindunixsocket = platform.bindunixsocket
14927
2aa3e07b2f07 posix, windows: introduce cachestat
Idan Kamara <idankk86@gmail.com>
parents: 14926
diff changeset
121 cachestat = platform.cachestat
14926
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
122 checkexec = platform.checkexec
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
123 checklink = platform.checklink
15011
5e44e4b3a0a3 util: move copymode into posix.py and windows.py
Adrian Buehlmann <adrian@cadifra.com>
parents: 15010
diff changeset
124 copymode = platform.copymode
14926
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
125 expandglobs = platform.expandglobs
35517
c4caf530b1c7 util: add a function to show the mount point of the filesystem
Matt Harbison <matt_harbison@yahoo.com>
parents: 35515
diff changeset
126 getfsmountpoint = platform.getfsmountpoint
35513
beede158ea8a util: move getfstype() to the platform modules
Matt Harbison <matt_harbison@yahoo.com>
parents: 35464
diff changeset
127 getfstype = platform.getfstype
47038
5b3513177f2b util: avoid echoing the password to the console on Windows py3 (issue6446)
Matt Harbison <matt_harbison@yahoo.com>
parents: 47031
diff changeset
128 get_password = platform.get_password
14926
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
129 groupmembers = platform.groupmembers
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
130 groupname = platform.groupname
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
131 isexec = platform.isexec
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
132 isowner = platform.isowner
32248
d74b0cff94a9 osutil: proxy through util (and platform) modules (API)
Yuya Nishihara <yuya@tcha.org>
parents: 32245
diff changeset
133 listdir = osutil.listdir
14926
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
134 localpath = platform.localpath
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
135 lookupreg = platform.lookupreg
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
136 makedir = platform.makedir
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
137 nlinks = platform.nlinks
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
138 normpath = platform.normpath
15488
6eff984d8e76 dirstate: fix case-folding identity for traditional Unix
Matt Mackall <mpm@selenic.com>
parents: 15392
diff changeset
139 normcase = platform.normcase
24605
98744856b7d3 util: add normcase spec and fallback
Siddharth Agarwal <sid0@fb.com>
parents: 24439
diff changeset
140 normcasespec = platform.normcasespec
98744856b7d3 util: add normcase spec and fallback
Siddharth Agarwal <sid0@fb.com>
parents: 24439
diff changeset
141 normcasefallback = platform.normcasefallback
14926
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
142 openhardlinks = platform.openhardlinks
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
143 oslink = platform.oslink
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
144 parsepatchoutput = platform.parsepatchoutput
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
145 pconvert = platform.pconvert
25420
c2ec81891502 util: add a simple poll utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25406
diff changeset
146 poll = platform.poll
14926
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
147 posixfile = platform.posixfile
39909
5fe0b880200e py3: convert os.readlink() path to native strings on Windows
Matt Harbison <matt_harbison@yahoo.com>
parents: 39793
diff changeset
148 readlink = platform.readlink
14926
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
149 rename = platform.rename
24692
144883a8d0d4 util: add removedirs as platform depending function
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 24635
diff changeset
150 removedirs = platform.removedirs
14926
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
151 samedevice = platform.samedevice
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
152 samefile = platform.samefile
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
153 samestat = platform.samestat
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
154 setflags = platform.setflags
17560
9ee25d7b1aed util: implement a faster os.path.split for posix systems
Bryan O'Sullivan <bryano@fb.com>
parents: 17537
diff changeset
155 split = platform.split
18026
ddc0323db78b osutil: write a C implementation of statfiles for unix
Bryan O'Sullivan <bryano@fb.com>
parents: 18013
diff changeset
156 statfiles = getattr(osutil, 'statfiles', platform.statfiles)
18868
cafa447a7d3b util: add functions to check symlink/exec bits
Bryan O'Sullivan <bryano@fb.com>
parents: 18775
diff changeset
157 statisexec = platform.statisexec
cafa447a7d3b util: add functions to check symlink/exec bits
Bryan O'Sullivan <bryano@fb.com>
parents: 18775
diff changeset
158 statislink = platform.statislink
14926
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
159 umask = platform.umask
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
160 unlink = platform.unlink
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
161 username = platform.username
14912
ec46a7da9f2c util: move windows and posix wildcard imports to begin of file
Adrian Buehlmann <adrian@cadifra.com>
parents: 14911
diff changeset
162
44633
d37975386798 chgserver: update the umask cache before each run
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44470
diff changeset
163
51924
1d95a87813ad typing: add type annotations to the `mercurial.util.filestat` class
Matt Harbison <matt_harbison@yahoo.com>
parents: 51917
diff changeset
164 if typing.TYPE_CHECKING:
1d95a87813ad typing: add type annotations to the `mercurial.util.filestat` class
Matt Harbison <matt_harbison@yahoo.com>
parents: 51917
diff changeset
165 _Tfilestat = TypeVar('_Tfilestat', bound='filestat')
1d95a87813ad typing: add type annotations to the `mercurial.util.filestat` class
Matt Harbison <matt_harbison@yahoo.com>
parents: 51917
diff changeset
166
1d95a87813ad typing: add type annotations to the `mercurial.util.filestat` class
Matt Harbison <matt_harbison@yahoo.com>
parents: 51917
diff changeset
167
51304
f15cb5111a1e pytype: move some type comment to proper annotation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51302
diff changeset
168 def setumask(val: int) -> None:
47031
f38bf44e077f black: make codebase compatible with black v21.4b2 and v20.8b1
Kyle Lippincott <spectral@google.com>
parents: 46926
diff changeset
169 '''updates the umask. used by chg server'''
44633
d37975386798 chgserver: update the umask cache before each run
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44470
diff changeset
170 if pycompat.iswindows:
d37975386798 chgserver: update the umask cache before each run
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44470
diff changeset
171 return
d37975386798 chgserver: update the umask cache before each run
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44470
diff changeset
172 os.umask(val)
d37975386798 chgserver: update the umask cache before each run
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44470
diff changeset
173 global umask
d37975386798 chgserver: update the umask cache before each run
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44470
diff changeset
174 platform.umask = umask = val & 0o777
d37975386798 chgserver: update the umask cache before each run
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44470
diff changeset
175
d37975386798 chgserver: update the umask cache before each run
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44470
diff changeset
176
42041
3e47d1ec9da5 util: extract compression code in `mercurial.utils.compression`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41839
diff changeset
177 # small compat layer
3e47d1ec9da5 util: extract compression code in `mercurial.utils.compression`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41839
diff changeset
178 compengines = compression.compengines
3e47d1ec9da5 util: extract compression code in `mercurial.utils.compression`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41839
diff changeset
179 SERVERROLE = compression.SERVERROLE
3e47d1ec9da5 util: extract compression code in `mercurial.utils.compression`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41839
diff changeset
180 CLIENTROLE = compression.CLIENTROLE
3e47d1ec9da5 util: extract compression code in `mercurial.utils.compression`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41839
diff changeset
181
6470
ac0bcd951c2c python 2.6 compatibility: compatibility wrappers for hash functions
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents: 6339
diff changeset
182 # Python compatibility
3769
96095d9ff1f8 Add encoding detection
Matt Mackall <mpm@selenic.com>
parents: 3767
diff changeset
183
15656
4f5a78fa4917 util: clean up function ordering
Matt Mackall <mpm@selenic.com>
parents: 15611
diff changeset
184 _notset = object()
4f5a78fa4917 util: clean up function ordering
Matt Mackall <mpm@selenic.com>
parents: 15611
diff changeset
185
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
186
30745
c1b7b2285522 revlog: flag processor
Remi Chaintron <remi@fb.com>
parents: 30736
diff changeset
187 def bitsfrom(container):
c1b7b2285522 revlog: flag processor
Remi Chaintron <remi@fb.com>
parents: 30736
diff changeset
188 bits = 0
c1b7b2285522 revlog: flag processor
Remi Chaintron <remi@fb.com>
parents: 30736
diff changeset
189 for bit in container:
c1b7b2285522 revlog: flag processor
Remi Chaintron <remi@fb.com>
parents: 30736
diff changeset
190 bits |= bit
c1b7b2285522 revlog: flag processor
Remi Chaintron <remi@fb.com>
parents: 30736
diff changeset
191 return bits
c1b7b2285522 revlog: flag processor
Remi Chaintron <remi@fb.com>
parents: 30736
diff changeset
192
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
193
31950
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
194 # python 2.6 still have deprecation warning enabled by default. We do not want
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
195 # to display anything to standard user so detect if we are running test and
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
196 # only use python deprecation warning in this case.
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
197 _dowarn = bool(encoding.environ.get(b'HGEMITWARNINGS'))
31950
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
198 if _dowarn:
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
199 # explicitly unfilter our warning for python 2.7
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
200 #
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
201 # The option of setting PYTHONWARNINGS in the test runner was investigated.
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
202 # However, module name set through PYTHONWARNINGS was exactly matched, so
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
203 # we cannot set 'mercurial' and have it match eg: 'mercurial.scmutil'. This
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
204 # makes the whole PYTHONWARNINGS thing useless for our usecase.
43554
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
205 warnings.filterwarnings('default', '', DeprecationWarning, 'mercurial')
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
206 warnings.filterwarnings('default', '', DeprecationWarning, 'hgext')
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
207 warnings.filterwarnings('default', '', DeprecationWarning, 'hgext3rd')
48988
10839394f39b util: remove superfluous ispy3 test
Gregory Szorc <gregory.szorc@gmail.com>
parents: 48966
diff changeset
208 if _dowarn:
36617
281f66777ff0 py3: silence "bad escape" warning emitted by re.sub()
Yuya Nishihara <yuya@tcha.org>
parents: 36614
diff changeset
209 # silence warning emitted by passing user string to re.sub()
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
210 warnings.filterwarnings(
43554
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
211 'ignore', 'bad escape', DeprecationWarning, 'mercurial'
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
212 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
213 warnings.filterwarnings(
43554
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
214 'ignore', 'invalid escape sequence', DeprecationWarning, 'mercurial'
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
215 )
37455
9ecb7c471cfb py3: silence warning about deprecation of imp module
Yuya Nishihara <yuya@tcha.org>
parents: 37382
diff changeset
216 # TODO: reinvent imp.is_frozen()
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
217 warnings.filterwarnings(
43554
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
218 'ignore',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
219 'the imp module is deprecated',
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
220 DeprecationWarning,
43554
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
221 'mercurial',
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
222 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
223
31950
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
224
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
225 def nouideprecwarn(msg, version, stacklevel=1):
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
226 """Issue an python native deprecation warning
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
227
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
228 This is a noop outside of tests, use 'ui.deprecwarn' when possible.
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
229 """
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
230 if _dowarn:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
231 msg += (
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
232 b"\n(compatibility will be dropped after Mercurial-%s,"
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
233 b" update your code.)"
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
234 ) % version
36166
02ed94dd9fd6 util: call warnings.warn() with a sysstr in nouideprecwarn
Augie Fackler <augie@google.com>
parents: 36076
diff changeset
235 warnings.warn(pycompat.sysstr(msg), DeprecationWarning, stacklevel + 1)
45009
7af5c1f5b3a0 util: flush stderr explicitly after using warnings.warn()
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44927
diff changeset
236 # on python 3 with chg, we will need to explicitly flush the output
7af5c1f5b3a0 util: flush stderr explicitly after using warnings.warn()
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44927
diff changeset
237 sys.stderr.flush()
31950
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
238
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
239
22962
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
240 DIGESTS = {
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
241 b'md5': hashlib.md5,
44060
a61287a95dc3 core: migrate uses of hashlib.sha1 to hashutil.sha1
Augie Fackler <augie@google.com>
parents: 44036
diff changeset
242 b'sha1': hashutil.sha1,
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
243 b'sha512': hashlib.sha512,
22962
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
244 }
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
245 # List of digest types from strongest to weakest
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
246 DIGESTS_BY_STRENGTH = [b'sha512', b'sha1', b'md5']
22962
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
247
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
248 for k in DIGESTS_BY_STRENGTH:
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
249 assert k in DIGESTS
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
250
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
251
49037
642e31cb55f0 py3: use class X: instead of class X(object):
Gregory Szorc <gregory.szorc@gmail.com>
parents: 49031
diff changeset
252 class digester:
22962
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
253 """helper to compute digests.
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
254
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
255 This helper can be used to compute one or more digests given their name.
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
256
34146
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
257 >>> d = digester([b'md5', b'sha1'])
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
258 >>> d.update(b'foo')
22962
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
259 >>> [k for k in sorted(d)]
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
260 ['md5', 'sha1']
34146
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
261 >>> d[b'md5']
22962
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
262 'acbd18db4cc2f85cedef654fccc4a4d8'
34146
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
263 >>> d[b'sha1']
22962
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
264 '0beec7b5ea3f0fdbc95d0dd47f3c5bc275da8a33'
34146
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
265 >>> digester.preferred([b'md5', b'sha1'])
22962
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
266 'sha1'
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
267 """
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
268
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
269 def __init__(self, digests, s=b''):
22962
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
270 self._hashes = {}
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
271 for k in digests:
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
272 if k not in DIGESTS:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
273 raise error.Abort(_(b'unknown digest type: %s') % k)
22962
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
274 self._hashes[k] = DIGESTS[k]()
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
275 if s:
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
276 self.update(s)
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
277
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
278 def update(self, data):
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
279 for h in self._hashes.values():
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
280 h.update(data)
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
281
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
282 def __getitem__(self, key):
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
283 if key not in DIGESTS:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
284 raise error.Abort(_(b'unknown digest type: %s') % k)
46114
59fa3890d40a node: import symbols explicitly
Joerg Sonnenberger <joerg@bec.de>
parents: 45957
diff changeset
285 return hex(self._hashes[key].digest())
22962
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
286
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
287 def __iter__(self):
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
288 return iter(self._hashes)
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
289
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
290 @staticmethod
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
291 def preferred(supported):
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
292 """returns the strongest digest type in both supported and DIGESTS."""
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
293
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
294 for k in DIGESTS_BY_STRENGTH:
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
295 if k in supported:
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
296 return k
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
297 return None
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
298
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
299
49037
642e31cb55f0 py3: use class X: instead of class X(object):
Gregory Szorc <gregory.szorc@gmail.com>
parents: 49031
diff changeset
300 class digestchecker:
22963
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
301 """file handle wrapper that additionally checks content against a given
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
302 size and digests.
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
303
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
304 d = digestchecker(fh, size, {'md5': '...'})
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
305
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
306 When multiple digests are given, all of them are validated.
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
307 """
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
308
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
309 def __init__(self, fh, size, digests):
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
310 self._fh = fh
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
311 self._size = size
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
312 self._got = 0
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
313 self._digests = dict(digests)
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
314 self._digester = digester(self._digests.keys())
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
315
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
316 def read(self, length=-1):
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
317 content = self._fh.read(length)
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
318 self._digester.update(content)
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
319 self._got += len(content)
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
320 return content
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
321
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
322 def validate(self):
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
323 if self._size != self._got:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
324 raise error.Abort(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
325 _(b'size mismatch: expected %d, got %d')
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
326 % (self._size, self._got)
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
327 )
22963
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
328 for k, v in self._digests.items():
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
329 if v != self._digester[k]:
23076
c312ef382033 i18n: add hint to digest mismatch message
Wagner Bruna <wbruna@softwareexpress.com.br>
parents: 23030
diff changeset
330 # i18n: first parameter is a digest name
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
331 raise error.Abort(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
332 _(b'%s mismatch: expected %s, got %s')
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
333 % (k, v, self._digester[k])
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
334 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
335
22963
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
336
11565
7546d4a272c8 util: improved the check for the existence of the 'buffer' builtin
Renato Cunha <renatoc@gmail.com>
parents: 11469
diff changeset
337 try:
46684
7711853110b9 typing: add some type annotations to mercurial/util.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 46114
diff changeset
338 buffer = buffer # pytype: disable=name-error
11565
7546d4a272c8 util: improved the check for the existence of the 'buffer' builtin
Renato Cunha <renatoc@gmail.com>
parents: 11469
diff changeset
339 except NameError:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
340
33549
9a2ee9591acc util: remove dead code which used to be for old python2 versions
Alex Gaynor <agaynor@mozilla.com>
parents: 33446
diff changeset
341 def buffer(sliceable, offset=0, length=None):
9a2ee9591acc util: remove dead code which used to be for old python2 versions
Alex Gaynor <agaynor@mozilla.com>
parents: 33446
diff changeset
342 if length is not None:
51888
f5c46c3518a5 util: make buffer readonly
Rapha?l Gom?s <rgomes@octobus.net>
parents: 51843
diff changeset
343 view = memoryview(sliceable)[offset : offset + length]
f5c46c3518a5 util: make buffer readonly
Rapha?l Gom?s <rgomes@octobus.net>
parents: 51843
diff changeset
344 else:
f5c46c3518a5 util: make buffer readonly
Rapha?l Gom?s <rgomes@octobus.net>
parents: 51843
diff changeset
345 view = memoryview(sliceable)[offset:]
f5c46c3518a5 util: make buffer readonly
Rapha?l Gom?s <rgomes@octobus.net>
parents: 51843
diff changeset
346 return view.toreadonly()
10756
cb681cc59a8d util: fake the builtin buffer if it's missing (jython)
Ronny Pfannschmidt <Ronny.Pfannschmidt@gmx.de>
parents: 10487
diff changeset
347
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
348
25406
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
349 _chunksize = 4096
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
350
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
351
49037
642e31cb55f0 py3: use class X: instead of class X(object):
Gregory Szorc <gregory.szorc@gmail.com>
parents: 49031
diff changeset
352 class bufferedinputpipe:
25406
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
353 """a manually buffered input pipe
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
354
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
355 Python will not let us use buffered IO and lazy reading with 'polling' at
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
356 the same time. We cannot probe the buffer state and select will not detect
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
357 that data are ready to read if they are already buffered.
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
358
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
359 This class let us work around that by implementing its own buffering
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
360 (allowing efficient readline) while offering a way to know if the buffer is
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
361 empty from the output (allowing collaboration of the buffer with polling).
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
362
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
363 This class lives in the 'util' module because it makes use of the 'os'
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
364 module from the python stdlib.
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
365 """
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
366
36554
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36553
diff changeset
367 def __new__(cls, fh):
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36553
diff changeset
368 # If we receive a fileobjectproxy, we need to use a variation of this
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36553
diff changeset
369 # class that notifies observers about activity.
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36553
diff changeset
370 if isinstance(fh, fileobjectproxy):
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36553
diff changeset
371 cls = observedbufferedinputpipe
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36553
diff changeset
372
52668
5cc8deb96b48 pyupgrade: modernize calls to superclass methods
Matt Harbison <matt_harbison@yahoo.com>
parents: 52665
diff changeset
373 return super().__new__(cls)
25406
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
374
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
375 def __init__(self, input):
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
376 self._input = input
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
377 self._buffer = []
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
378 self._eof = False
25672
050dc6eabc92 bufferedinputpipe: remove N^2 computation of buffer length (issue4735)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25671
diff changeset
379 self._lenbuf = 0
25406
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
380
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
381 @property
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
382 def hasbuffer(self):
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
383 """True is any data is currently buffered
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
384
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
385 This will be used externally a pre-step for polling IO. If there is
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
386 already data then no polling should be set in place."""
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
387 return bool(self._buffer)
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
388
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
389 @property
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
390 def closed(self):
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
391 return self._input.closed
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
392
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
393 def fileno(self):
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
394 return self._input.fileno()
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
395
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
396 def close(self):
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
397 return self._input.close()
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
398
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
399 def read(self, size):
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
400 while (not self._eof) and (self._lenbuf < size):
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
401 self._fillbuffer()
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
402 return self._frombuffer(size)
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
403
38713
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38576
diff changeset
404 def unbufferedread(self, size):
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38576
diff changeset
405 if not self._eof and self._lenbuf == 0:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38576
diff changeset
406 self._fillbuffer(max(size, _chunksize))
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38576
diff changeset
407 return self._frombuffer(min(self._lenbuf, size))
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38576
diff changeset
408
25406
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
409 def readline(self, *args, **kwargs):
40030
e2697acd9381 cleanup: some Yoda conditions, this patch removes
Martin von Zweigbergk <martinvonz@google.com>
parents: 39910
diff changeset
410 if len(self._buffer) > 1:
25406
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
411 # this should not happen because both read and readline end with a
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
412 # _frombuffer call that collapse it.
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
413 self._buffer = [b''.join(self._buffer)]
25672
050dc6eabc92 bufferedinputpipe: remove N^2 computation of buffer length (issue4735)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25671
diff changeset
414 self._lenbuf = len(self._buffer[0])
25406
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
415 lfi = -1
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
416 if self._buffer:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
417 lfi = self._buffer[-1].find(b'\n')
25406
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
418 while (not self._eof) and lfi < 0:
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
419 self._fillbuffer()
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
420 if self._buffer:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
421 lfi = self._buffer[-1].find(b'\n')
25406
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
422 size = lfi + 1
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
423 if lfi < 0: # end of file
25406
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
424 size = self._lenbuf
40030
e2697acd9381 cleanup: some Yoda conditions, this patch removes
Martin von Zweigbergk <martinvonz@google.com>
parents: 39910
diff changeset
425 elif len(self._buffer) > 1:
25406
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
426 # we need to take previous chunks into account
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
427 size += self._lenbuf - len(self._buffer[-1])
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
428 return self._frombuffer(size)
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
429
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
430 def _frombuffer(self, size):
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
431 """return at most 'size' data from the buffer
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
432
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
433 The data are removed from the buffer."""
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
434 if size == 0 or not self._buffer:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
435 return b''
25406
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
436 buf = self._buffer[0]
40030
e2697acd9381 cleanup: some Yoda conditions, this patch removes
Martin von Zweigbergk <martinvonz@google.com>
parents: 39910
diff changeset
437 if len(self._buffer) > 1:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
438 buf = b''.join(self._buffer)
25406
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
439
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
440 data = buf[:size]
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
441 buf = buf[len(data) :]
25406
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
442 if buf:
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
443 self._buffer = [buf]
25672
050dc6eabc92 bufferedinputpipe: remove N^2 computation of buffer length (issue4735)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25671
diff changeset
444 self._lenbuf = len(buf)
25406
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
445 else:
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
446 self._buffer = []
25672
050dc6eabc92 bufferedinputpipe: remove N^2 computation of buffer length (issue4735)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25671
diff changeset
447 self._lenbuf = 0
25406
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
448 return data
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
449
38713
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38576
diff changeset
450 def _fillbuffer(self, size=_chunksize):
25406
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
451 """read data to the buffer"""
38713
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38576
diff changeset
452 data = os.read(self._input.fileno(), size)
25406
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
453 if not data:
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
454 self._eof = True
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
455 else:
25672
050dc6eabc92 bufferedinputpipe: remove N^2 computation of buffer length (issue4735)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25671
diff changeset
456 self._lenbuf += len(data)
25406
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
457 self._buffer.append(data)
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
458
36554
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36553
diff changeset
459 return data
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36553
diff changeset
460
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
461
51742
5f37c36f36b9 revlog: use mmap by default is pre-population is available
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51740
diff changeset
462 def has_mmap_populate():
51800
b619ba39d10a mmap: populate mapping in a background thread
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51789
diff changeset
463 return hasattr(osutil, "background_mmap_populate") or hasattr(
b619ba39d10a mmap: populate mapping in a background thread
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51789
diff changeset
464 mmap, 'MAP_POPULATE'
b619ba39d10a mmap: populate mapping in a background thread
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51789
diff changeset
465 )
51742
5f37c36f36b9 revlog: use mmap by default is pre-population is available
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51740
diff changeset
466
5f37c36f36b9 revlog: use mmap by default is pre-population is available
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51740
diff changeset
467
51740
522b4d729e89 mmap: populate the mapping by default
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51649
diff changeset
468 def mmapread(fp, size=None, pre_populate=True):
51649
ba205f944cb4 mmap: add a `is_mmap_safe` method to vfs
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51633
diff changeset
469 """Read a file content using mmap
ba205f944cb4 mmap: add a `is_mmap_safe` method to vfs
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51633
diff changeset
470
51967
2d51b0cf707c util: minor copy editing of the documentation for `mmapread()`
Matt Harbison <matt_harbison@yahoo.com>
parents: 51966
diff changeset
471 The responsibility of checking the file system is mmap safe is the
2d51b0cf707c util: minor copy editing of the documentation for `mmapread()`
Matt Harbison <matt_harbison@yahoo.com>
parents: 51966
diff changeset
472 responsibility of the caller (see `vfs.is_mmap_safe`).
51649
ba205f944cb4 mmap: add a `is_mmap_safe` method to vfs
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51633
diff changeset
473
ba205f944cb4 mmap: add a `is_mmap_safe` method to vfs
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51633
diff changeset
474 In some case, a normal string might be returned.
51740
522b4d729e89 mmap: populate the mapping by default
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51649
diff changeset
475
522b4d729e89 mmap: populate the mapping by default
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51649
diff changeset
476 If `pre_populate` is True (the default), the mmapped data will be
522b4d729e89 mmap: populate the mapping by default
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51649
diff changeset
477 pre-populated in memory if the system support this option, this slow down
51967
2d51b0cf707c util: minor copy editing of the documentation for `mmapread()`
Matt Harbison <matt_harbison@yahoo.com>
parents: 51966
diff changeset
478 the initial mmapping but avoid potentially crippling page fault on later
51740
522b4d729e89 mmap: populate the mapping by default
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51649
diff changeset
479 access. If this is not the desired behavior, set `pre_populate` to False.
51649
ba205f944cb4 mmap: add a `is_mmap_safe` method to vfs
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51633
diff changeset
480 """
44036
8ed8dfbeabb9 mmap: add a size argument to mmapread
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44023
diff changeset
481 if size == 0:
8ed8dfbeabb9 mmap: add a size argument to mmapread
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44023
diff changeset
482 # size of 0 to mmap.mmap() means "all data"
8ed8dfbeabb9 mmap: add a size argument to mmapread
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44023
diff changeset
483 # rather than "zero bytes", so special case that.
8ed8dfbeabb9 mmap: add a size argument to mmapread
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44023
diff changeset
484 return b''
8ed8dfbeabb9 mmap: add a size argument to mmapread
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44023
diff changeset
485 elif size is None:
8ed8dfbeabb9 mmap: add a size argument to mmapread
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44023
diff changeset
486 size = 0
48033
ae79611e3115 util: avoid a name-error warning in the `mmapread` exception handler
Matt Harbison <matt_harbison@yahoo.com>
parents: 47622
diff changeset
487 fd = getattr(fp, 'fileno', lambda: fp)()
51966
bc9ed92d4753 util: make `mmapread()` work on Windows again
Matt Harbison <matt_harbison@yahoo.com>
parents: 51926
diff changeset
488
bc9ed92d4753 util: make `mmapread()` work on Windows again
Matt Harbison <matt_harbison@yahoo.com>
parents: 51926
diff changeset
489 if pycompat.iswindows:
bc9ed92d4753 util: make `mmapread()` work on Windows again
Matt Harbison <matt_harbison@yahoo.com>
parents: 51926
diff changeset
490 _mmap = lambda fd, size: mmap.mmap(fd, size, access=mmap.ACCESS_READ)
bc9ed92d4753 util: make `mmapread()` work on Windows again
Matt Harbison <matt_harbison@yahoo.com>
parents: 51926
diff changeset
491 else:
bc9ed92d4753 util: make `mmapread()` work on Windows again
Matt Harbison <matt_harbison@yahoo.com>
parents: 51926
diff changeset
492 flags = mmap.MAP_PRIVATE
bc9ed92d4753 util: make `mmapread()` work on Windows again
Matt Harbison <matt_harbison@yahoo.com>
parents: 51926
diff changeset
493 bg_populate = hasattr(osutil, "background_mmap_populate")
bc9ed92d4753 util: make `mmapread()` work on Windows again
Matt Harbison <matt_harbison@yahoo.com>
parents: 51926
diff changeset
494
bc9ed92d4753 util: make `mmapread()` work on Windows again
Matt Harbison <matt_harbison@yahoo.com>
parents: 51926
diff changeset
495 if pre_populate and not bg_populate:
bc9ed92d4753 util: make `mmapread()` work on Windows again
Matt Harbison <matt_harbison@yahoo.com>
parents: 51926
diff changeset
496 flags |= getattr(mmap, 'MAP_POPULATE', 0)
bc9ed92d4753 util: make `mmapread()` work on Windows again
Matt Harbison <matt_harbison@yahoo.com>
parents: 51926
diff changeset
497
bc9ed92d4753 util: make `mmapread()` work on Windows again
Matt Harbison <matt_harbison@yahoo.com>
parents: 51926
diff changeset
498 def _mmap(fd, size) -> mmap.mmap:
bc9ed92d4753 util: make `mmapread()` work on Windows again
Matt Harbison <matt_harbison@yahoo.com>
parents: 51926
diff changeset
499 m = mmap.mmap(fd, size, flags=flags, prot=mmap.PROT_READ)
bc9ed92d4753 util: make `mmapread()` work on Windows again
Matt Harbison <matt_harbison@yahoo.com>
parents: 51926
diff changeset
500 if pre_populate and bg_populate:
bc9ed92d4753 util: make `mmapread()` work on Windows again
Matt Harbison <matt_harbison@yahoo.com>
parents: 51926
diff changeset
501 osutil.background_mmap_populate(m)
bc9ed92d4753 util: make `mmapread()` work on Windows again
Matt Harbison <matt_harbison@yahoo.com>
parents: 51926
diff changeset
502 return m
bc9ed92d4753 util: make `mmapread()` work on Windows again
Matt Harbison <matt_harbison@yahoo.com>
parents: 51926
diff changeset
503
34302
3bb2a9f25fe9 util: add an mmapread method
Mark Thomas <mbthomas@fb.com>
parents: 34154
diff changeset
504 try:
51966
bc9ed92d4753 util: make `mmapread()` work on Windows again
Matt Harbison <matt_harbison@yahoo.com>
parents: 51926
diff changeset
505 return _mmap(fd, size)
34302
3bb2a9f25fe9 util: add an mmapread method
Mark Thomas <mbthomas@fb.com>
parents: 34154
diff changeset
506 except ValueError:
3bb2a9f25fe9 util: add an mmapread method
Mark Thomas <mbthomas@fb.com>
parents: 34154
diff changeset
507 # Empty files cannot be mmapped, but mmapread should still work. Check
3bb2a9f25fe9 util: add an mmapread method
Mark Thomas <mbthomas@fb.com>
parents: 34154
diff changeset
508 # if the file is empty, and if so, return an empty buffer.
3bb2a9f25fe9 util: add an mmapread method
Mark Thomas <mbthomas@fb.com>
parents: 34154
diff changeset
509 if os.fstat(fd).st_size == 0:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
510 return b''
34302
3bb2a9f25fe9 util: add an mmapread method
Mark Thomas <mbthomas@fb.com>
parents: 34154
diff changeset
511 raise
3bb2a9f25fe9 util: add an mmapread method
Mark Thomas <mbthomas@fb.com>
parents: 34154
diff changeset
512
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
513
52116
82e2c99c84f3 cachestat: avoid creating cachestat for http path
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51974
diff changeset
514 class uncacheable_cachestat:
82e2c99c84f3 cachestat: avoid creating cachestat for http path
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51974
diff changeset
515 stat: Optional[os.stat_result]
82e2c99c84f3 cachestat: avoid creating cachestat for http path
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51974
diff changeset
516
82e2c99c84f3 cachestat: avoid creating cachestat for http path
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51974
diff changeset
517 def __init__(self) -> None:
82e2c99c84f3 cachestat: avoid creating cachestat for http path
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51974
diff changeset
518 self.stat = None
82e2c99c84f3 cachestat: avoid creating cachestat for http path
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51974
diff changeset
519
82e2c99c84f3 cachestat: avoid creating cachestat for http path
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51974
diff changeset
520 def cacheable(self) -> bool:
82e2c99c84f3 cachestat: avoid creating cachestat for http path
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51974
diff changeset
521 return False
82e2c99c84f3 cachestat: avoid creating cachestat for http path
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51974
diff changeset
522
82e2c99c84f3 cachestat: avoid creating cachestat for http path
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51974
diff changeset
523
49037
642e31cb55f0 py3: use class X: instead of class X(object):
Gregory Szorc <gregory.szorc@gmail.com>
parents: 49031
diff changeset
524 class fileobjectproxy:
36553
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
525 """A proxy around file objects that tells a watcher when events occur.
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
526
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
527 This type is intended to only be used for testing purposes. Think hard
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
528 before using it in important code.
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
529 """
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
530
36553
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
531 __slots__ = (
43554
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
532 '_orig',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
533 '_observer',
36553
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
534 )
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
535
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
536 def __init__(self, fh, observer):
43554
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
537 object.__setattr__(self, '_orig', fh)
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
538 object.__setattr__(self, '_observer', observer)
36553
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
539
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
540 def __getattribute__(self, name):
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
541 ours = {
43554
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
542 '_observer',
36553
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
543 # IOBase
43554
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
544 'close',
36553
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
545 # closed if a property
43554
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
546 'fileno',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
547 'flush',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
548 'isatty',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
549 'readable',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
550 'readline',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
551 'readlines',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
552 'seek',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
553 'seekable',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
554 'tell',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
555 'truncate',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
556 'writable',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
557 'writelines',
36553
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
558 # RawIOBase
43554
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
559 'read',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
560 'readall',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
561 'readinto',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
562 'write',
36553
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
563 # BufferedIOBase
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
564 # raw is a property
43554
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
565 'detach',
36553
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
566 # read defined above
43554
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
567 'read1',
36553
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
568 # readinto defined above
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
569 # write defined above
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
570 }
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
571
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
572 # We only observe some methods.
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
573 if name in ours:
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
574 return object.__getattribute__(self, name)
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
575
43554
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
576 return getattr(object.__getattribute__(self, '_orig'), name)
36553
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
577
36840
6bdea0efdab5 util: forward __bool__()/__nonzero__() on fileobjectproxy
Matt Harbison <matt_harbison@yahoo.com>
parents: 36801
diff changeset
578 def __nonzero__(self):
43554
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
579 return bool(object.__getattribute__(self, '_orig'))
36840
6bdea0efdab5 util: forward __bool__()/__nonzero__() on fileobjectproxy
Matt Harbison <matt_harbison@yahoo.com>
parents: 36801
diff changeset
580
6bdea0efdab5 util: forward __bool__()/__nonzero__() on fileobjectproxy
Matt Harbison <matt_harbison@yahoo.com>
parents: 36801
diff changeset
581 __bool__ = __nonzero__
6bdea0efdab5 util: forward __bool__()/__nonzero__() on fileobjectproxy
Matt Harbison <matt_harbison@yahoo.com>
parents: 36801
diff changeset
582
36553
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
583 def __delattr__(self, name):
43554
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
584 return delattr(object.__getattribute__(self, '_orig'), name)
36553
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
585
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
586 def __setattr__(self, name, value):
43554
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
587 return setattr(object.__getattribute__(self, '_orig'), name, value)
36553
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
588
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
589 def __iter__(self):
43554
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
590 return object.__getattribute__(self, '_orig').__iter__()
36553
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
591
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
592 def _observedcall(self, name, *args, **kwargs):
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
593 # Call the original object.
43554
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
594 orig = object.__getattribute__(self, '_orig')
36553
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
595 res = getattr(orig, name)(*args, **kwargs)
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
596
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
597 # Call a method on the observer of the same name with arguments
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
598 # so it can react, log, etc.
43554
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
599 observer = object.__getattribute__(self, '_observer')
36553
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
600 fn = getattr(observer, name, None)
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
601 if fn:
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
602 fn(res, *args, **kwargs)
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
603
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
604 return res
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
605
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
606 def close(self, *args, **kwargs):
43554
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
607 return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
608 'close', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
609 )
36553
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
610
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
611 def fileno(self, *args, **kwargs):
43554
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
612 return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
613 'fileno', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
614 )
36553
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
615
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
616 def flush(self, *args, **kwargs):
43554
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
617 return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
618 'flush', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
619 )
36553
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
620
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
621 def isatty(self, *args, **kwargs):
43554
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
622 return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
623 'isatty', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
624 )
36553
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
625
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
626 def readable(self, *args, **kwargs):
43554
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
627 return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
628 'readable', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
629 )
36553
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
630
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
631 def readline(self, *args, **kwargs):
43554
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
632 return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
633 'readline', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
634 )
36553
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
635
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
636 def readlines(self, *args, **kwargs):
43554
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
637 return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
638 'readlines', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
639 )
36553
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
640
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
641 def seek(self, *args, **kwargs):
43554
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
642 return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
643 'seek', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
644 )
36553
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
645
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
646 def seekable(self, *args, **kwargs):
43554
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
647 return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
648 'seekable', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
649 )
36553
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
650
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
651 def tell(self, *args, **kwargs):
43554
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
652 return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
653 'tell', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
654 )
36553
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
655
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
656 def truncate(self, *args, **kwargs):
43554
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
657 return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
658 'truncate', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
659 )
36553
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
660
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
661 def writable(self, *args, **kwargs):
43554
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
662 return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
663 'writable', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
664 )
36553
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
665
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
666 def writelines(self, *args, **kwargs):
43554
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
667 return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
668 'writelines', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
669 )
36553
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
670
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
671 def read(self, *args, **kwargs):
43554
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
672 return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
673 'read', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
674 )
36553
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
675
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
676 def readall(self, *args, **kwargs):
43554
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
677 return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
678 'readall', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
679 )
36553
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
680
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
681 def readinto(self, *args, **kwargs):
43554
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
682 return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
683 'readinto', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
684 )
36553
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
685
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
686 def write(self, *args, **kwargs):
43554
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
687 return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
688 'write', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
689 )
36553
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
690
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
691 def detach(self, *args, **kwargs):
43554
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
692 return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
693 'detach', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
694 )
36553
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
695
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
696 def read1(self, *args, **kwargs):
43554
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
697 return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
698 'read1', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
699 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
700
36553
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
701
36554
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36553
diff changeset
702 class observedbufferedinputpipe(bufferedinputpipe):
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36553
diff changeset
703 """A variation of bufferedinputpipe that is aware of fileobjectproxy.
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36553
diff changeset
704
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36553
diff changeset
705 ``bufferedinputpipe`` makes low-level calls to ``os.read()`` that
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36553
diff changeset
706 bypass ``fileobjectproxy``. Because of this, we need to make
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36553
diff changeset
707 ``bufferedinputpipe`` aware of these operations.
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36553
diff changeset
708
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36553
diff changeset
709 This variation of ``bufferedinputpipe`` can notify observers about
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36553
diff changeset
710 ``os.read()`` events. It also re-publishes other events, such as
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36553
diff changeset
711 ``read()`` and ``readline()``.
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36553
diff changeset
712 """
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
713
49909
1d1b244a91b6 util: fix the signature of observedbufferedinputpipe._fillbuffer()
Matt Harbison <matt_harbison@yahoo.com>
parents: 49908
diff changeset
714 def _fillbuffer(self, size=_chunksize):
52668
5cc8deb96b48 pyupgrade: modernize calls to superclass methods
Matt Harbison <matt_harbison@yahoo.com>
parents: 52665
diff changeset
715 res = super()._fillbuffer(size=size)
36554
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36553
diff changeset
716
43103
c95b2f40db7c py3: stop normalizing 2nd argument of *attr() to unicode
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43090
diff changeset
717 fn = getattr(self._input._observer, 'osread', None)
36554
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36553
diff changeset
718 if fn:
49909
1d1b244a91b6 util: fix the signature of observedbufferedinputpipe._fillbuffer()
Matt Harbison <matt_harbison@yahoo.com>
parents: 49908
diff changeset
719 fn(res, size)
36554
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36553
diff changeset
720
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36553
diff changeset
721 return res
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36553
diff changeset
722
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36553
diff changeset
723 # We use different observer methods because the operation isn't
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36553
diff changeset
724 # performed on the actual file object but on us.
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36553
diff changeset
725 def read(self, size):
52668
5cc8deb96b48 pyupgrade: modernize calls to superclass methods
Matt Harbison <matt_harbison@yahoo.com>
parents: 52665
diff changeset
726 res = super().read(size)
36554
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36553
diff changeset
727
43103
c95b2f40db7c py3: stop normalizing 2nd argument of *attr() to unicode
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43090
diff changeset
728 fn = getattr(self._input._observer, 'bufferedread', None)
36554
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36553
diff changeset
729 if fn:
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36553
diff changeset
730 fn(res, size)
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36553
diff changeset
731
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36553
diff changeset
732 return res
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36553
diff changeset
733
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36553
diff changeset
734 def readline(self, *args, **kwargs):
52668
5cc8deb96b48 pyupgrade: modernize calls to superclass methods
Matt Harbison <matt_harbison@yahoo.com>
parents: 52665
diff changeset
735 res = super().readline(*args, **kwargs)
36554
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36553
diff changeset
736
43103
c95b2f40db7c py3: stop normalizing 2nd argument of *attr() to unicode
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43090
diff changeset
737 fn = getattr(self._input._observer, 'bufferedreadline', None)
36554
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36553
diff changeset
738 if fn:
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36553
diff changeset
739 fn(res)
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36553
diff changeset
740
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36553
diff changeset
741 return res
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36553
diff changeset
742
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
743
37013
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
744 PROXIED_SOCKET_METHODS = {
43554
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
745 'makefile',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
746 'recv',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
747 'recvfrom',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
748 'recvfrom_into',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
749 'recv_into',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
750 'send',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
751 'sendall',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
752 'sendto',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
753 'setblocking',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
754 'settimeout',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
755 'gettimeout',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
756 'setsockopt',
37013
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
757 }
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
758
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
759
49037
642e31cb55f0 py3: use class X: instead of class X(object):
Gregory Szorc <gregory.szorc@gmail.com>
parents: 49031
diff changeset
760 class socketproxy:
37013
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
761 """A proxy around a socket that tells a watcher when events occur.
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
762
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
763 This is like ``fileobjectproxy`` except for sockets.
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
764
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
765 This type is intended to only be used for testing purposes. Think hard
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
766 before using it in important code.
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
767 """
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
768
37013
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
769 __slots__ = (
43554
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
770 '_orig',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
771 '_observer',
37013
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
772 )
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
773
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
774 def __init__(self, sock, observer):
43554
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
775 object.__setattr__(self, '_orig', sock)
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
776 object.__setattr__(self, '_observer', observer)
37013
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
777
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
778 def __getattribute__(self, name):
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
779 if name in PROXIED_SOCKET_METHODS:
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
780 return object.__getattribute__(self, name)
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
781
43554
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
782 return getattr(object.__getattribute__(self, '_orig'), name)
37013
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
783
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
784 def __delattr__(self, name):
43554
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
785 return delattr(object.__getattribute__(self, '_orig'), name)
37013
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
786
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
787 def __setattr__(self, name, value):
43554
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
788 return setattr(object.__getattribute__(self, '_orig'), name, value)
37013
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
789
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
790 def __nonzero__(self):
43554
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
791 return bool(object.__getattribute__(self, '_orig'))
37013
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
792
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
793 __bool__ = __nonzero__
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
794
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
795 def _observedcall(self, name, *args, **kwargs):
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
796 # Call the original object.
43554
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
797 orig = object.__getattribute__(self, '_orig')
37013
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
798 res = getattr(orig, name)(*args, **kwargs)
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
799
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
800 # Call a method on the observer of the same name with arguments
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
801 # so it can react, log, etc.
43554
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
802 observer = object.__getattribute__(self, '_observer')
37013
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
803 fn = getattr(observer, name, None)
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
804 if fn:
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
805 fn(res, *args, **kwargs)
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
806
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
807 return res
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
808
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
809 def makefile(self, *args, **kwargs):
43554
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
810 res = object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
811 'makefile', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
812 )
37013
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
813
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
814 # The file object may be used for I/O. So we turn it into a
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
815 # proxy using our observer.
43554
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
816 observer = object.__getattribute__(self, '_observer')
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
817 return makeloggingfileobject(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
818 observer.fh,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
819 res,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
820 observer.name,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
821 reads=observer.reads,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
822 writes=observer.writes,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
823 logdata=observer.logdata,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
824 logdataapis=observer.logdataapis,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
825 )
37013
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
826
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
827 def recv(self, *args, **kwargs):
43554
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
828 return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
829 'recv', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
830 )
37013
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
831
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
832 def recvfrom(self, *args, **kwargs):
43554
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
833 return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
834 'recvfrom', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
835 )
37013
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
836
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
837 def recvfrom_into(self, *args, **kwargs):
43554
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
838 return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
839 'recvfrom_into', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
840 )
37013
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
841
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
842 def recv_into(self, *args, **kwargs):
43554
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
843 return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
844 'recv_info', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
845 )
37013
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
846
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
847 def send(self, *args, **kwargs):
43554
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
848 return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
849 'send', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
850 )
37013
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
851
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
852 def sendall(self, *args, **kwargs):
43554
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
853 return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
854 'sendall', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
855 )
37013
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
856
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
857 def sendto(self, *args, **kwargs):
43554
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
858 return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
859 'sendto', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
860 )
37013
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
861
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
862 def setblocking(self, *args, **kwargs):
43554
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
863 return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
864 'setblocking', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
865 )
37013
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
866
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
867 def settimeout(self, *args, **kwargs):
43554
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
868 return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
869 'settimeout', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
870 )
37013
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
871
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
872 def gettimeout(self, *args, **kwargs):
43554
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
873 return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
874 'gettimeout', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
875 )
37013
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
876
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
877 def setsockopt(self, *args, **kwargs):
43554
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
878 return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
879 'setsockopt', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
880 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
881
37013
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
882
49037
642e31cb55f0 py3: use class X: instead of class X(object):
Gregory Szorc <gregory.szorc@gmail.com>
parents: 49031
diff changeset
883 class baseproxyobserver:
43926
4222b9d5d4fb util: move common proxyobserver attributes to the base class
Matt Harbison <matt_harbison@yahoo.com>
parents: 43913
diff changeset
884 def __init__(self, fh, name, logdata, logdataapis):
4222b9d5d4fb util: move common proxyobserver attributes to the base class
Matt Harbison <matt_harbison@yahoo.com>
parents: 43913
diff changeset
885 self.fh = fh
4222b9d5d4fb util: move common proxyobserver attributes to the base class
Matt Harbison <matt_harbison@yahoo.com>
parents: 43913
diff changeset
886 self.name = name
4222b9d5d4fb util: move common proxyobserver attributes to the base class
Matt Harbison <matt_harbison@yahoo.com>
parents: 43913
diff changeset
887 self.logdata = logdata
4222b9d5d4fb util: move common proxyobserver attributes to the base class
Matt Harbison <matt_harbison@yahoo.com>
parents: 43913
diff changeset
888 self.logdataapis = logdataapis
4222b9d5d4fb util: move common proxyobserver attributes to the base class
Matt Harbison <matt_harbison@yahoo.com>
parents: 43913
diff changeset
889
36553
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
890 def _writedata(self, data):
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
891 if not self.logdata:
37047
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37013
diff changeset
892 if self.logdataapis:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
893 self.fh.write(b'\n')
37047
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37013
diff changeset
894 self.fh.flush()
36553
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
895 return
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
896
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
897 # Simple case writes all data on a single line.
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
898 if b'\n' not in data:
37047
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37013
diff changeset
899 if self.logdataapis:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
900 self.fh.write(b': %s\n' % stringutil.escapestr(data))
37047
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37013
diff changeset
901 else:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
902 self.fh.write(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
903 b'%s> %s\n' % (self.name, stringutil.escapestr(data))
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
904 )
36994
d683c7367989 wireproto: explicitly flush stdio to prevent stalls on Windows
Matt Harbison <matt_harbison@yahoo.com>
parents: 36964
diff changeset
905 self.fh.flush()
36553
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
906 return
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
907
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
908 # Data with newlines is written to multiple lines.
37047
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37013
diff changeset
909 if self.logdataapis:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
910 self.fh.write(b':\n')
37047
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37013
diff changeset
911
36553
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
912 lines = data.splitlines(True)
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
913 for line in lines:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
914 self.fh.write(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
915 b'%s> %s\n' % (self.name, stringutil.escapestr(line))
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
916 )
36994
d683c7367989 wireproto: explicitly flush stdio to prevent stalls on Windows
Matt Harbison <matt_harbison@yahoo.com>
parents: 36964
diff changeset
917 self.fh.flush()
36553
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
918
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
919
37013
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
920 class fileobjectobserver(baseproxyobserver):
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
921 """Logs file object activity."""
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
922
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
923 def __init__(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
924 self, fh, name, reads=True, writes=True, logdata=False, logdataapis=True
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
925 ):
52668
5cc8deb96b48 pyupgrade: modernize calls to superclass methods
Matt Harbison <matt_harbison@yahoo.com>
parents: 52665
diff changeset
926 super().__init__(fh, name, logdata, logdataapis)
37013
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
927 self.reads = reads
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
928 self.writes = writes
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
929
36553
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
930 def read(self, res, size=-1):
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
931 if not self.reads:
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
932 return
36614
26a6b62919e2 util: work around Python 3 returning None at EOF instead of ''
Augie Fackler <augie@google.com>
parents: 36613
diff changeset
933 # Python 3 can return None from reads at EOF instead of empty strings.
26a6b62919e2 util: work around Python 3 returning None at EOF instead of ''
Augie Fackler <augie@google.com>
parents: 36613
diff changeset
934 if res is None:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
935 res = b''
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
936
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
937 if size == -1 and res == b'':
38323
565074cc9ac6 tests: suppress read(-1) -> '' calls in fileobjectobserver
Augie Fackler <augie@google.com>
parents: 38197
diff changeset
938 # Suppress pointless read(-1) calls that return
565074cc9ac6 tests: suppress read(-1) -> '' calls in fileobjectobserver
Augie Fackler <augie@google.com>
parents: 38197
diff changeset
939 # nothing. These happen _a lot_ on Python 3, and there
565074cc9ac6 tests: suppress read(-1) -> '' calls in fileobjectobserver
Augie Fackler <augie@google.com>
parents: 38197
diff changeset
940 # doesn't seem to be a better workaround to have matching
565074cc9ac6 tests: suppress read(-1) -> '' calls in fileobjectobserver
Augie Fackler <augie@google.com>
parents: 38197
diff changeset
941 # Python 2 and 3 behavior. :(
565074cc9ac6 tests: suppress read(-1) -> '' calls in fileobjectobserver
Augie Fackler <augie@google.com>
parents: 38197
diff changeset
942 return
565074cc9ac6 tests: suppress read(-1) -> '' calls in fileobjectobserver
Augie Fackler <augie@google.com>
parents: 38197
diff changeset
943
37047
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37013
diff changeset
944 if self.logdataapis:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
945 self.fh.write(b'%s> read(%d) -> %d' % (self.name, size, len(res)))
37047
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37013
diff changeset
946
36553
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
947 self._writedata(res)
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
948
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
949 def readline(self, res, limit=-1):
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
950 if not self.reads:
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
951 return
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
952
37047
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37013
diff changeset
953 if self.logdataapis:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
954 self.fh.write(b'%s> readline() -> %d' % (self.name, len(res)))
37047
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37013
diff changeset
955
36553
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
956 self._writedata(res)
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
957
36659
29128309c52d util: log readinto() I/O
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36658
diff changeset
958 def readinto(self, res, dest):
29128309c52d util: log readinto() I/O
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36658
diff changeset
959 if not self.reads:
29128309c52d util: log readinto() I/O
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36658
diff changeset
960 return
29128309c52d util: log readinto() I/O
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36658
diff changeset
961
37047
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37013
diff changeset
962 if self.logdataapis:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
963 self.fh.write(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
964 b'%s> readinto(%d) -> %r' % (self.name, len(dest), res)
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
965 )
37047
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37013
diff changeset
966
36659
29128309c52d util: log readinto() I/O
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36658
diff changeset
967 data = dest[0:res] if res is not None else b''
41406
b141b5243b37 util: cast memoryview to bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 41289
diff changeset
968
b141b5243b37 util: cast memoryview to bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 41289
diff changeset
969 # _writedata() uses "in" operator and is confused by memoryview because
b141b5243b37 util: cast memoryview to bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 41289
diff changeset
970 # characters are ints on Python 3.
b141b5243b37 util: cast memoryview to bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 41289
diff changeset
971 if isinstance(data, memoryview):
b141b5243b37 util: cast memoryview to bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 41289
diff changeset
972 data = data.tobytes()
b141b5243b37 util: cast memoryview to bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 41289
diff changeset
973
36659
29128309c52d util: log readinto() I/O
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36658
diff changeset
974 self._writedata(data)
29128309c52d util: log readinto() I/O
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36658
diff changeset
975
36553
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
976 def write(self, res, data):
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
977 if not self.writes:
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
978 return
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
979
36660
8395fddde46c util: report integer result from write()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36659
diff changeset
980 # Python 2 returns None from some write() calls. Python 3 (reasonably)
8395fddde46c util: report integer result from write()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36659
diff changeset
981 # returns the integer bytes written.
8395fddde46c util: report integer result from write()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36659
diff changeset
982 if res is None and data:
8395fddde46c util: report integer result from write()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36659
diff changeset
983 res = len(data)
8395fddde46c util: report integer result from write()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36659
diff changeset
984
37047
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37013
diff changeset
985 if self.logdataapis:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
986 self.fh.write(b'%s> write(%d) -> %r' % (self.name, len(data), res))
37047
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37013
diff changeset
987
36553
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
988 self._writedata(data)
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
989
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
990 def flush(self, res):
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
991 if not self.writes:
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
992 return
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
993
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
994 self.fh.write(b'%s> flush() -> %r\n' % (self.name, res))
36553
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
995
36554
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36553
diff changeset
996 # For observedbufferedinputpipe.
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36553
diff changeset
997 def bufferedread(self, res, size):
37047
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37013
diff changeset
998 if not self.reads:
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37013
diff changeset
999 return
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37013
diff changeset
1000
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37013
diff changeset
1001 if self.logdataapis:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1002 self.fh.write(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1003 b'%s> bufferedread(%d) -> %d' % (self.name, size, len(res))
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1004 )
37047
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37013
diff changeset
1005
36554
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36553
diff changeset
1006 self._writedata(res)
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36553
diff changeset
1007
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36553
diff changeset
1008 def bufferedreadline(self, res):
37047
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37013
diff changeset
1009 if not self.reads:
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37013
diff changeset
1010 return
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37013
diff changeset
1011
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37013
diff changeset
1012 if self.logdataapis:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1013 self.fh.write(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1014 b'%s> bufferedreadline() -> %d' % (self.name, len(res))
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1015 )
37047
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37013
diff changeset
1016
36554
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36553
diff changeset
1017 self._writedata(res)
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36553
diff changeset
1018
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1019
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1020 def makeloggingfileobject(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1021 logh, fh, name, reads=True, writes=True, logdata=False, logdataapis=True
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1022 ):
36553
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
1023 """Turn a file object into a logging file object."""
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
1024
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1025 observer = fileobjectobserver(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1026 logh,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1027 name,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1028 reads=reads,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1029 writes=writes,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1030 logdata=logdata,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1031 logdataapis=logdataapis,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1032 )
36553
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
1033 return fileobjectproxy(fh, observer)
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36462
diff changeset
1034
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1035
37013
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1036 class socketobserver(baseproxyobserver):
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1037 """Logs socket activity."""
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1038
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1039 def __init__(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1040 self,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1041 fh,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1042 name,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1043 reads=True,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1044 writes=True,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1045 states=True,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1046 logdata=False,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1047 logdataapis=True,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1048 ):
52668
5cc8deb96b48 pyupgrade: modernize calls to superclass methods
Matt Harbison <matt_harbison@yahoo.com>
parents: 52665
diff changeset
1049 super().__init__(fh, name, logdata, logdataapis)
37013
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1050 self.reads = reads
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1051 self.writes = writes
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1052 self.states = states
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1053
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1054 def makefile(self, res, mode=None, bufsize=None):
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1055 if not self.states:
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1056 return
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1057
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1058 self.fh.write(b'%s> makefile(%r, %r)\n' % (self.name, mode, bufsize))
37013
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1059
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1060 def recv(self, res, size, flags=0):
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1061 if not self.reads:
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1062 return
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1063
37047
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37013
diff changeset
1064 if self.logdataapis:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1065 self.fh.write(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1066 b'%s> recv(%d, %d) -> %d' % (self.name, size, flags, len(res))
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1067 )
37013
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1068 self._writedata(res)
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1069
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1070 def recvfrom(self, res, size, flags=0):
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1071 if not self.reads:
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1072 return
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1073
37047
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37013
diff changeset
1074 if self.logdataapis:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1075 self.fh.write(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1076 b'%s> recvfrom(%d, %d) -> %d'
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1077 % (self.name, size, flags, len(res[0]))
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1078 )
37047
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37013
diff changeset
1079
37013
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1080 self._writedata(res[0])
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1081
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1082 def recvfrom_into(self, res, buf, size, flags=0):
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1083 if not self.reads:
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1084 return
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1085
37047
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37013
diff changeset
1086 if self.logdataapis:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1087 self.fh.write(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1088 b'%s> recvfrom_into(%d, %d) -> %d'
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1089 % (self.name, size, flags, res[0])
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1090 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1091
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1092 self._writedata(buf[0 : res[0]])
37013
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1093
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1094 def recv_into(self, res, buf, size=0, flags=0):
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1095 if not self.reads:
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1096 return
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1097
37047
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37013
diff changeset
1098 if self.logdataapis:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1099 self.fh.write(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1100 b'%s> recv_into(%d, %d) -> %d' % (self.name, size, flags, res)
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1101 )
37047
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37013
diff changeset
1102
37013
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1103 self._writedata(buf[0:res])
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1104
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1105 def send(self, res, data, flags=0):
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1106 if not self.writes:
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1107 return
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1108
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1109 self.fh.write(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1110 b'%s> send(%d, %d) -> %d' % (self.name, len(data), flags, len(res))
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1111 )
37013
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1112 self._writedata(data)
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1113
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1114 def sendall(self, res, data, flags=0):
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1115 if not self.writes:
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1116 return
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1117
37047
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37013
diff changeset
1118 if self.logdataapis:
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37013
diff changeset
1119 # Returns None on success. So don't bother reporting return value.
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1120 self.fh.write(
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1121 b'%s> sendall(%d, %d)' % (self.name, len(data), flags)
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1122 )
37047
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37013
diff changeset
1123
37013
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1124 self._writedata(data)
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1125
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1126 def sendto(self, res, data, flagsoraddress, address=None):
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1127 if not self.writes:
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1128 return
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1129
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1130 if address:
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1131 flags = flagsoraddress
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1132 else:
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1133 flags = 0
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1134
37047
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37013
diff changeset
1135 if self.logdataapis:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1136 self.fh.write(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1137 b'%s> sendto(%d, %d, %r) -> %d'
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1138 % (self.name, len(data), flags, address, res)
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1139 )
37047
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37013
diff changeset
1140
37013
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1141 self._writedata(data)
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1142
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1143 def setblocking(self, res, flag):
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1144 if not self.states:
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1145 return
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1146
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1147 self.fh.write(b'%s> setblocking(%r)\n' % (self.name, flag))
37013
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1148
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1149 def settimeout(self, res, value):
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1150 if not self.states:
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1151 return
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1152
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1153 self.fh.write(b'%s> settimeout(%r)\n' % (self.name, value))
37013
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1154
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1155 def gettimeout(self, res):
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1156 if not self.states:
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1157 return
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1158
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1159 self.fh.write(b'%s> gettimeout() -> %f\n' % (self.name, res))
37013
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1160
39097
ed8160e4fea0 util: fix signature of setsockopt in socket observer
Augie Fackler <augie@google.com>
parents: 38852
diff changeset
1161 def setsockopt(self, res, level, optname, value):
37013
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1162 if not self.states:
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1163 return
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1164
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1165 self.fh.write(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1166 b'%s> setsockopt(%r, %r, %r) -> %r\n'
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1167 % (self.name, level, optname, value, res)
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1168 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1169
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1170
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1171 def makeloggingsocket(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1172 logh,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1173 fh,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1174 name,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1175 reads=True,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1176 writes=True,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1177 states=True,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1178 logdata=False,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1179 logdataapis=True,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1180 ):
37013
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1181 """Turn a socket into a logging socket."""
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1182
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1183 observer = socketobserver(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1184 logh,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1185 name,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1186 reads=reads,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1187 writes=writes,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1188 states=states,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1189 logdata=logdata,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1190 logdataapis=logdataapis,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1191 )
37013
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1192 return socketproxy(fh, observer)
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36994
diff changeset
1193
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1194
7632
9626819b2e3d refactor version code
Matt Mackall <mpm@selenic.com>
parents: 7559
diff changeset
1195 def version():
9626819b2e3d refactor version code
Matt Mackall <mpm@selenic.com>
parents: 7559
diff changeset
1196 """Return version information if available."""
9626819b2e3d refactor version code
Matt Mackall <mpm@selenic.com>
parents: 7559
diff changeset
1197 try:
51789
92845af308b4 typing: narrow the scope of some recent disabled import warnings
Matt Harbison <matt_harbison@yahoo.com>
parents: 51788
diff changeset
1198 from . import __version__ # pytype: disable=import-error
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1199
7632
9626819b2e3d refactor version code
Matt Mackall <mpm@selenic.com>
parents: 7559
diff changeset
1200 return __version__.version
9626819b2e3d refactor version code
Matt Mackall <mpm@selenic.com>
parents: 7559
diff changeset
1201 except ImportError:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1202 return b'unknown'
7632
9626819b2e3d refactor version code
Matt Mackall <mpm@selenic.com>
parents: 7559
diff changeset
1203
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1204
27112
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1205 def versiontuple(v=None, n=4):
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1206 """Parses a Mercurial version string into an N-tuple.
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1207
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1208 The version string to be parsed is specified with the ``v`` argument.
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1209 If it isn't defined, the current Mercurial version string will be parsed.
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1210
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1211 ``n`` can be 2, 3, or 4. Here is how some version strings map to
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1212 returned values:
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1213
34146
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
1214 >>> v = b'3.6.1+190-df9b73d2d444'
27112
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1215 >>> versiontuple(v, 2)
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1216 (3, 6)
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1217 >>> versiontuple(v, 3)
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1218 (3, 6, 1)
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1219 >>> versiontuple(v, 4)
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1220 (3, 6, 1, '190-df9b73d2d444')
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1221
34146
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
1222 >>> versiontuple(b'3.6.1+190-df9b73d2d444+20151118')
27112
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1223 (3, 6, 1, '190-df9b73d2d444+20151118')
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1224
34146
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
1225 >>> v = b'3.6'
27112
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1226 >>> versiontuple(v, 2)
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1227 (3, 6)
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1228 >>> versiontuple(v, 3)
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1229 (3, 6, None)
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1230 >>> versiontuple(v, 4)
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1231 (3, 6, None, None)
29613
616cbcb59e05 util: better handle '-' in version string (issue5302)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 29592
diff changeset
1232
34146
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
1233 >>> v = b'3.9-rc'
29613
616cbcb59e05 util: better handle '-' in version string (issue5302)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 29592
diff changeset
1234 >>> versiontuple(v, 2)
616cbcb59e05 util: better handle '-' in version string (issue5302)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 29592
diff changeset
1235 (3, 9)
616cbcb59e05 util: better handle '-' in version string (issue5302)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 29592
diff changeset
1236 >>> versiontuple(v, 3)
616cbcb59e05 util: better handle '-' in version string (issue5302)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 29592
diff changeset
1237 (3, 9, None)
616cbcb59e05 util: better handle '-' in version string (issue5302)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 29592
diff changeset
1238 >>> versiontuple(v, 4)
616cbcb59e05 util: better handle '-' in version string (issue5302)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 29592
diff changeset
1239 (3, 9, None, 'rc')
616cbcb59e05 util: better handle '-' in version string (issue5302)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 29592
diff changeset
1240
34146
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
1241 >>> v = b'3.9-rc+2-02a8fea4289b'
29613
616cbcb59e05 util: better handle '-' in version string (issue5302)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 29592
diff changeset
1242 >>> versiontuple(v, 2)
616cbcb59e05 util: better handle '-' in version string (issue5302)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 29592
diff changeset
1243 (3, 9)
616cbcb59e05 util: better handle '-' in version string (issue5302)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 29592
diff changeset
1244 >>> versiontuple(v, 3)
616cbcb59e05 util: better handle '-' in version string (issue5302)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 29592
diff changeset
1245 (3, 9, None)
616cbcb59e05 util: better handle '-' in version string (issue5302)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 29592
diff changeset
1246 >>> versiontuple(v, 4)
616cbcb59e05 util: better handle '-' in version string (issue5302)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 29592
diff changeset
1247 (3, 9, None, 'rc+2-02a8fea4289b')
37801
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings
Yuya Nishihara <yuya@tcha.org>
parents: 37675
diff changeset
1248
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings
Yuya Nishihara <yuya@tcha.org>
parents: 37675
diff changeset
1249 >>> versiontuple(b'4.6rc0')
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings
Yuya Nishihara <yuya@tcha.org>
parents: 37675
diff changeset
1250 (4, 6, None, 'rc0')
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings
Yuya Nishihara <yuya@tcha.org>
parents: 37675
diff changeset
1251 >>> versiontuple(b'4.6rc0+12-425d55e54f98')
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings
Yuya Nishihara <yuya@tcha.org>
parents: 37675
diff changeset
1252 (4, 6, None, 'rc0+12-425d55e54f98')
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings
Yuya Nishihara <yuya@tcha.org>
parents: 37675
diff changeset
1253 >>> versiontuple(b'.1.2.3')
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings
Yuya Nishihara <yuya@tcha.org>
parents: 37675
diff changeset
1254 (None, None, None, '.1.2.3')
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings
Yuya Nishihara <yuya@tcha.org>
parents: 37675
diff changeset
1255 >>> versiontuple(b'12.34..5')
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings
Yuya Nishihara <yuya@tcha.org>
parents: 37675
diff changeset
1256 (12, 34, None, '..5')
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings
Yuya Nishihara <yuya@tcha.org>
parents: 37675
diff changeset
1257 >>> versiontuple(b'1.2.3.4.5.6')
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings
Yuya Nishihara <yuya@tcha.org>
parents: 37675
diff changeset
1258 (1, 2, 3, '.4.5.6')
27112
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1259 """
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1260 if not v:
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1261 v = version()
44023
6d3b67a837a6 cleanup: drop redundant character escapes from `[]` character sets
Matt Harbison <matt_harbison@yahoo.com>
parents: 44020
diff changeset
1262 m = remod.match(br'(\d+(?:\.\d+){,2})[+-]?(.*)', v)
37801
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings
Yuya Nishihara <yuya@tcha.org>
parents: 37675
diff changeset
1263 if not m:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1264 vparts, extra = b'', v
37801
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings
Yuya Nishihara <yuya@tcha.org>
parents: 37675
diff changeset
1265 elif m.group(2):
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings
Yuya Nishihara <yuya@tcha.org>
parents: 37675
diff changeset
1266 vparts, extra = m.groups()
27112
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1267 else:
37801
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings
Yuya Nishihara <yuya@tcha.org>
parents: 37675
diff changeset
1268 vparts, extra = m.group(1), None
27112
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1269
43888
a21a6dad4b38 typing: add an assertion to util.versiontuple
Matt Harbison <matt_harbison@yahoo.com>
parents: 43887
diff changeset
1270 assert vparts is not None # help pytype
a21a6dad4b38 typing: add an assertion to util.versiontuple
Matt Harbison <matt_harbison@yahoo.com>
parents: 43887
diff changeset
1271
27112
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1272 vints = []
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1273 for i in vparts.split(b'.'):
27112
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1274 try:
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1275 vints.append(int(i))
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1276 except ValueError:
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1277 break
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1278 # (3, 6) -> (3, 6, None)
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1279 while len(vints) < 3:
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1280 vints.append(None)
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1281
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1282 if n == 2:
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1283 return (vints[0], vints[1])
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1284 if n == 3:
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1285 return (vints[0], vints[1], vints[2])
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1286 if n == 4:
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1287 return (vints[0], vints[1], vints[2], extra)
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1288
48036
376d08ae904f util: eliminate the possibility of returning None from `versiontuple()`
Matt Harbison <matt_harbison@yahoo.com>
parents: 48033
diff changeset
1289 raise error.ProgrammingError(b"invalid version part request: %d" % n)
376d08ae904f util: eliminate the possibility of returning None from `versiontuple()`
Matt Harbison <matt_harbison@yahoo.com>
parents: 48033
diff changeset
1290
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1291
3145
e4ea47c21480 Add cachefunc to abstract function call cache
Brendan Cully <brendan@kublai.com>
parents: 3131
diff changeset
1292 def cachefunc(func):
e4ea47c21480 Add cachefunc to abstract function call cache
Brendan Cully <brendan@kublai.com>
parents: 3131
diff changeset
1293 '''cache the result of function calls'''
3147
97420a49188d add comments in cachefunc
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 3145
diff changeset
1294 # XXX doesn't handle keywords args
28832
f5ff10f6fa6b util: use __code__ (available since py2.6)
timeless <timeless@mozdev.org>
parents: 28826
diff changeset
1295 if func.__code__.co_argcount == 0:
43886
eff050dbb703 util: rename a variable to avoid confusing pytype
Matt Harbison <matt_harbison@yahoo.com>
parents: 43885
diff changeset
1296 listcache = []
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1297
20835
0e8417131a29 util: add the code path to "cachefunc()" for the function taking no arguments
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 20679
diff changeset
1298 def f():
43886
eff050dbb703 util: rename a variable to avoid confusing pytype
Matt Harbison <matt_harbison@yahoo.com>
parents: 43885
diff changeset
1299 if len(listcache) == 0:
eff050dbb703 util: rename a variable to avoid confusing pytype
Matt Harbison <matt_harbison@yahoo.com>
parents: 43885
diff changeset
1300 listcache.append(func())
eff050dbb703 util: rename a variable to avoid confusing pytype
Matt Harbison <matt_harbison@yahoo.com>
parents: 43885
diff changeset
1301 return listcache[0]
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1302
20835
0e8417131a29 util: add the code path to "cachefunc()" for the function taking no arguments
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 20679
diff changeset
1303 return f
3145
e4ea47c21480 Add cachefunc to abstract function call cache
Brendan Cully <brendan@kublai.com>
parents: 3131
diff changeset
1304 cache = {}
28832
f5ff10f6fa6b util: use __code__ (available since py2.6)
timeless <timeless@mozdev.org>
parents: 28826
diff changeset
1305 if func.__code__.co_argcount == 1:
3147
97420a49188d add comments in cachefunc
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 3145
diff changeset
1306 # we gain a small amount of time because
97420a49188d add comments in cachefunc
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 3145
diff changeset
1307 # we don't need to pack/unpack the list
3145
e4ea47c21480 Add cachefunc to abstract function call cache
Brendan Cully <brendan@kublai.com>
parents: 3131
diff changeset
1308 def f(arg):
e4ea47c21480 Add cachefunc to abstract function call cache
Brendan Cully <brendan@kublai.com>
parents: 3131
diff changeset
1309 if arg not in cache:
e4ea47c21480 Add cachefunc to abstract function call cache
Brendan Cully <brendan@kublai.com>
parents: 3131
diff changeset
1310 cache[arg] = func(arg)
e4ea47c21480 Add cachefunc to abstract function call cache
Brendan Cully <brendan@kublai.com>
parents: 3131
diff changeset
1311 return cache[arg]
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1312
3145
e4ea47c21480 Add cachefunc to abstract function call cache
Brendan Cully <brendan@kublai.com>
parents: 3131
diff changeset
1313 else:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1314
3145
e4ea47c21480 Add cachefunc to abstract function call cache
Brendan Cully <brendan@kublai.com>
parents: 3131
diff changeset
1315 def f(*args):
e4ea47c21480 Add cachefunc to abstract function call cache
Brendan Cully <brendan@kublai.com>
parents: 3131
diff changeset
1316 if args not in cache:
e4ea47c21480 Add cachefunc to abstract function call cache
Brendan Cully <brendan@kublai.com>
parents: 3131
diff changeset
1317 cache[args] = func(*args)
e4ea47c21480 Add cachefunc to abstract function call cache
Brendan Cully <brendan@kublai.com>
parents: 3131
diff changeset
1318 return cache[args]
e4ea47c21480 Add cachefunc to abstract function call cache
Brendan Cully <brendan@kublai.com>
parents: 3131
diff changeset
1319
e4ea47c21480 Add cachefunc to abstract function call cache
Brendan Cully <brendan@kublai.com>
parents: 3131
diff changeset
1320 return f
e4ea47c21480 Add cachefunc to abstract function call cache
Brendan Cully <brendan@kublai.com>
parents: 3131
diff changeset
1321
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1322
49037
642e31cb55f0 py3: use class X: instead of class X(object):
Gregory Szorc <gregory.szorc@gmail.com>
parents: 49031
diff changeset
1323 class cow:
34358
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
1324 """helper class to make copy-on-write easier
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
1325
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
1326 Call preparewrite before doing any writes.
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
1327 """
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
1328
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
1329 def preparewrite(self):
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
1330 """call this before writes, return self or a copied new object"""
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
1331 if getattr(self, '_copied', 0):
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
1332 self._copied -= 1
46901
51841b23670b typing: make minor adjustments to mercurial/util.py to pass pytype checking
Matt Harbison <matt_harbison@yahoo.com>
parents: 46900
diff changeset
1333 # Function cow.__init__ expects 1 arg(s), got 2 [wrong-arg-count]
51841b23670b typing: make minor adjustments to mercurial/util.py to pass pytype checking
Matt Harbison <matt_harbison@yahoo.com>
parents: 46900
diff changeset
1334 return self.__class__(self) # pytype: disable=wrong-arg-count
34358
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
1335 return self
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
1336
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
1337 def copy(self):
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
1338 """always do a cheap copy"""
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
1339 self._copied = getattr(self, '_copied', 0) + 1
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
1340 return self
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
1341
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1342
32340
bd0fd3ff9916 util: rewrite sortdict using Python 2.7's OrderedDict
Martin von Zweigbergk <martinvonz@google.com>
parents: 32331
diff changeset
1343 class sortdict(collections.OrderedDict):
45957
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
1344 """a simple sorted dictionary
32346
2d19664e257d util: drop unneeded override, sortdict.copy()
Yuya Nishihara <yuya@tcha.org>
parents: 32340
diff changeset
1345
34146
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
1346 >>> d1 = sortdict([(b'a', 0), (b'b', 1)])
32346
2d19664e257d util: drop unneeded override, sortdict.copy()
Yuya Nishihara <yuya@tcha.org>
parents: 32340
diff changeset
1347 >>> d2 = d1.copy()
50749
a2df74853f8d tests: fix sortdict doctest with Python 3.12
Mads Kiilerich <mads@kiilerich.com>
parents: 50398
diff changeset
1348 >>> list(d2.items())
a2df74853f8d tests: fix sortdict doctest with Python 3.12
Mads Kiilerich <mads@kiilerich.com>
parents: 50398
diff changeset
1349 [('a', 0), ('b', 1)]
34146
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
1350 >>> d2.update([(b'a', 2)])
34154
be00af4a1ac5 doctest: coerce dict.keys() to list
Yuya Nishihara <yuya@tcha.org>
parents: 34152
diff changeset
1351 >>> list(d2.keys()) # should still be in last-set order
32346
2d19664e257d util: drop unneeded override, sortdict.copy()
Yuya Nishihara <yuya@tcha.org>
parents: 32340
diff changeset
1352 ['b', 'a']
43913
68af0228fedc util: implement sortdict.insert()
Martin von Zweigbergk <martinvonz@google.com>
parents: 43889
diff changeset
1353 >>> d1.insert(1, b'a.5', 0.5)
50749
a2df74853f8d tests: fix sortdict doctest with Python 3.12
Mads Kiilerich <mads@kiilerich.com>
parents: 50398
diff changeset
1354 >>> list(d1.items())
a2df74853f8d tests: fix sortdict doctest with Python 3.12
Mads Kiilerich <mads@kiilerich.com>
parents: 50398
diff changeset
1355 [('a', 0), ('a.5', 0.5), ('b', 1)]
45957
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
1356 """
32346
2d19664e257d util: drop unneeded override, sortdict.copy()
Yuya Nishihara <yuya@tcha.org>
parents: 32340
diff changeset
1357
32340
bd0fd3ff9916 util: rewrite sortdict using Python 2.7's OrderedDict
Martin von Zweigbergk <martinvonz@google.com>
parents: 32331
diff changeset
1358 def __setitem__(self, key, value):
21813
c2262004c2e2 config: move config.sortdict class into util
Angel Ezquerra <angel.ezquerra@gmail.com>
parents: 21046
diff changeset
1359 if key in self:
32340
bd0fd3ff9916 util: rewrite sortdict using Python 2.7's OrderedDict
Martin von Zweigbergk <martinvonz@google.com>
parents: 32331
diff changeset
1360 del self[key]
52668
5cc8deb96b48 pyupgrade: modernize calls to superclass methods
Matt Harbison <matt_harbison@yahoo.com>
parents: 52665
diff changeset
1361 super().__setitem__(key, value)
21813
c2262004c2e2 config: move config.sortdict class into util
Angel Ezquerra <angel.ezquerra@gmail.com>
parents: 21046
diff changeset
1362
33624
524b13fc711f util: fix sortdict.update() to call __setitem__() on PyPy (issue5639)
Yuya Nishihara <yuya@tcha.org>
parents: 33549
diff changeset
1363 if pycompat.ispypy:
524b13fc711f util: fix sortdict.update() to call __setitem__() on PyPy (issue5639)
Yuya Nishihara <yuya@tcha.org>
parents: 33549
diff changeset
1364 # __setitem__() isn't called as of PyPy 5.8.0
46900
64400d05db1e util: fix the signature for the pypy override of sortdict.update()
Matt Harbison <matt_harbison@yahoo.com>
parents: 46827
diff changeset
1365 def update(self, src, **f):
33624
524b13fc711f util: fix sortdict.update() to call __setitem__() on PyPy (issue5639)
Yuya Nishihara <yuya@tcha.org>
parents: 33549
diff changeset
1366 if isinstance(src, dict):
49004
f254fc73d956 global: bulk replace simple pycompat.iteritems(x) with x.items()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 48988
diff changeset
1367 src = src.items()
33624
524b13fc711f util: fix sortdict.update() to call __setitem__() on PyPy (issue5639)
Yuya Nishihara <yuya@tcha.org>
parents: 33549
diff changeset
1368 for k, v in src:
524b13fc711f util: fix sortdict.update() to call __setitem__() on PyPy (issue5639)
Yuya Nishihara <yuya@tcha.org>
parents: 33549
diff changeset
1369 self[k] = v
46900
64400d05db1e util: fix the signature for the pypy override of sortdict.update()
Matt Harbison <matt_harbison@yahoo.com>
parents: 46827
diff changeset
1370 for k in f:
64400d05db1e util: fix the signature for the pypy override of sortdict.update()
Matt Harbison <matt_harbison@yahoo.com>
parents: 46827
diff changeset
1371 self[k] = f[k]
33624
524b13fc711f util: fix sortdict.update() to call __setitem__() on PyPy (issue5639)
Yuya Nishihara <yuya@tcha.org>
parents: 33549
diff changeset
1372
43913
68af0228fedc util: implement sortdict.insert()
Martin von Zweigbergk <martinvonz@google.com>
parents: 43889
diff changeset
1373 def insert(self, position, key, value):
51762
ca7bde5dbafb black: format the codebase with 23.3.0
Rapha?l Gom?s <rgomes@octobus.net>
parents: 51761
diff changeset
1374 for i, (k, v) in enumerate(list(self.items())):
43913
68af0228fedc util: implement sortdict.insert()
Martin von Zweigbergk <martinvonz@google.com>
parents: 43889
diff changeset
1375 if i == position:
68af0228fedc util: implement sortdict.insert()
Martin von Zweigbergk <martinvonz@google.com>
parents: 43889
diff changeset
1376 self[key] = value
68af0228fedc util: implement sortdict.insert()
Martin von Zweigbergk <martinvonz@google.com>
parents: 43889
diff changeset
1377 if i >= position:
68af0228fedc util: implement sortdict.insert()
Martin von Zweigbergk <martinvonz@google.com>
parents: 43889
diff changeset
1378 del self[k]
68af0228fedc util: implement sortdict.insert()
Martin von Zweigbergk <martinvonz@google.com>
parents: 43889
diff changeset
1379 self[k] = v
68af0228fedc util: implement sortdict.insert()
Martin von Zweigbergk <martinvonz@google.com>
parents: 43889
diff changeset
1380
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1381
34358
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
1382 class cowdict(cow, dict):
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
1383 """copy-on-write dict
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
1384
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
1385 Be sure to call d = d.preparewrite() before writing to d.
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
1386
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
1387 >>> a = cowdict()
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
1388 >>> a is a.preparewrite()
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
1389 True
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
1390 >>> b = a.copy()
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
1391 >>> b is a
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
1392 True
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
1393 >>> c = b.copy()
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
1394 >>> c is a
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
1395 True
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
1396 >>> a = a.preparewrite()
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
1397 >>> b is a
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
1398 False
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
1399 >>> a is a.preparewrite()
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
1400 True
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
1401 >>> c = c.preparewrite()
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
1402 >>> b is c
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
1403 False
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
1404 >>> b is b.preparewrite()
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
1405 True
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
1406 """
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
1407
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1408
34358
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
1409 class cowsortdict(cow, sortdict):
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
1410 """copy-on-write sortdict
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
1411
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
1412 Be sure to call d = d.preparewrite() before writing to d.
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
1413 """
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
1414
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1415
49037
642e31cb55f0 py3: use class X: instead of class X(object):
Gregory Szorc <gregory.szorc@gmail.com>
parents: 49031
diff changeset
1416 class transactional: # pytype: disable=ignored-metaclass
33814
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33763
diff changeset
1417 """Base class for making a transactional type into a context manager."""
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1418
33814
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33763
diff changeset
1419 __metaclass__ = abc.ABCMeta
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33763
diff changeset
1420
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33763
diff changeset
1421 @abc.abstractmethod
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33763
diff changeset
1422 def close(self):
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33763
diff changeset
1423 """Successfully closes the transaction."""
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33763
diff changeset
1424
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33763
diff changeset
1425 @abc.abstractmethod
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33763
diff changeset
1426 def release(self):
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33763
diff changeset
1427 """Marks the end of the transaction.
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33763
diff changeset
1428
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33763
diff changeset
1429 If the transaction has not been closed, it will be aborted.
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33763
diff changeset
1430 """
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33763
diff changeset
1431
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33763
diff changeset
1432 def __enter__(self):
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33763
diff changeset
1433 return self
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33763
diff changeset
1434
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33763
diff changeset
1435 def __exit__(self, exc_type, exc_val, exc_tb):
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33763
diff changeset
1436 try:
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33763
diff changeset
1437 if exc_type is None:
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33763
diff changeset
1438 self.close()
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33763
diff changeset
1439 finally:
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33763
diff changeset
1440 self.release()
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33763
diff changeset
1441
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1442
33446
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
1443 @contextlib.contextmanager
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
1444 def acceptintervention(tr=None):
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
1445 """A context manager that closes the transaction on InterventionRequired
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
1446
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
1447 If no transaction was provided, this simply runs the body and returns
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
1448 """
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
1449 if not tr:
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
1450 yield
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
1451 return
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
1452 try:
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
1453 yield
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
1454 tr.close()
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
1455 except error.InterventionRequired:
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
1456 tr.close()
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
1457 raise
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
1458 finally:
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
1459 tr.release()
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
1460
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1461
33691
609606d21765 rebase: use one dirstateguard for when using rebase.singletransaction
Durham Goode <durham@fb.com>
parents: 33549
diff changeset
1462 @contextlib.contextmanager
45146
a0791bfd9cfa util: enhance `nullcontextmanager` to be able to return __enter__ result
Manuel Jacob <me@manueljacob.de>
parents: 45062
diff changeset
1463 def nullcontextmanager(enter_result=None):
a0791bfd9cfa util: enhance `nullcontextmanager` to be able to return __enter__ result
Manuel Jacob <me@manueljacob.de>
parents: 45062
diff changeset
1464 yield enter_result
33691
609606d21765 rebase: use one dirstateguard for when using rebase.singletransaction
Durham Goode <durham@fb.com>
parents: 33549
diff changeset
1465
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1466
49037
642e31cb55f0 py3: use class X: instead of class X(object):
Gregory Szorc <gregory.szorc@gmail.com>
parents: 49031
diff changeset
1467 class _lrucachenode:
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1468 """A node in a doubly linked list.
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1469
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1470 Holds a reference to nodes on either side as well as a key-value
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1471 pair for the dictionary entry.
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1472 """
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1473
43554
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43542
diff changeset
1474 __slots__ = ('next', 'prev', 'key', 'value', 'cost')
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1475
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1476 def __init__(self):
46901
51841b23670b typing: make minor adjustments to mercurial/util.py to pass pytype checking
Matt Harbison <matt_harbison@yahoo.com>
parents: 46900
diff changeset
1477 self.next = self
51841b23670b typing: make minor adjustments to mercurial/util.py to pass pytype checking
Matt Harbison <matt_harbison@yahoo.com>
parents: 46900
diff changeset
1478 self.prev = self
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1479
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1480 self.key = _notset
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1481 self.value = None
39583
ee087f0d7db5 util: allow lrucachedict to track cost of entries
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39582
diff changeset
1482 self.cost = 0
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1483
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1484 def markempty(self):
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1485 """Mark the node as emptied."""
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1486 self.key = _notset
39583
ee087f0d7db5 util: allow lrucachedict to track cost of entries
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39582
diff changeset
1487 self.value = None
ee087f0d7db5 util: allow lrucachedict to track cost of entries
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39582
diff changeset
1488 self.cost = 0
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1489
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1490
49037
642e31cb55f0 py3: use class X: instead of class X(object):
Gregory Szorc <gregory.szorc@gmail.com>
parents: 49031
diff changeset
1491 class lrucachedict:
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1492 """Dict that caches most recent accesses and sets.
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1493
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1494 The dict consists of an actual backing dict - indexed by original
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1495 key - and a doubly linked circular list defining the order of entries in
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1496 the cache.
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1497
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1498 The head node is the newest entry in the cache. If the cache is full,
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1499 we recycle head.prev and make it the new head. Cache accesses result in
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1500 the node being moved to before the existing head and being marked as the
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1501 new head node.
39583
ee087f0d7db5 util: allow lrucachedict to track cost of entries
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39582
diff changeset
1502
ee087f0d7db5 util: allow lrucachedict to track cost of entries
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39582
diff changeset
1503 Items in the cache can be inserted with an optional "cost" value. This is
ee087f0d7db5 util: allow lrucachedict to track cost of entries
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39582
diff changeset
1504 simply an integer that is specified by the caller. The cache can be queried
ee087f0d7db5 util: allow lrucachedict to track cost of entries
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39582
diff changeset
1505 for the total cost of all items presently in the cache.
39584
842cd0bdda75 util: teach lrucachedict to enforce a max total cost
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39583
diff changeset
1506
842cd0bdda75 util: teach lrucachedict to enforce a max total cost
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39583
diff changeset
1507 The cache can also define a maximum cost. If a cache insertion would
842cd0bdda75 util: teach lrucachedict to enforce a max total cost
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39583
diff changeset
1508 cause the total cost of the cache to go beyond the maximum cost limit,
842cd0bdda75 util: teach lrucachedict to enforce a max total cost
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39583
diff changeset
1509 nodes will be evicted to make room for the new code. This can be used
842cd0bdda75 util: teach lrucachedict to enforce a max total cost
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39583
diff changeset
1510 to e.g. set a max memory limit and associate an estimated bytes size
842cd0bdda75 util: teach lrucachedict to enforce a max total cost
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39583
diff changeset
1511 cost to each item in the cache. By default, no maximum cost is enforced.
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1512 """
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1513
39584
842cd0bdda75 util: teach lrucachedict to enforce a max total cost
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39583
diff changeset
1514 def __init__(self, max, maxcost=0):
18603
2251b3184e6e util: add an LRU cache dict
Siddharth Agarwal <sid0@fb.com>
parents: 18537
diff changeset
1515 self._cache = {}
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1516
46901
51841b23670b typing: make minor adjustments to mercurial/util.py to pass pytype checking
Matt Harbison <matt_harbison@yahoo.com>
parents: 46900
diff changeset
1517 self._head = _lrucachenode()
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1518 self._size = 1
39580
5d75a3c16193 util: make capacity a public attribute on lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39579
diff changeset
1519 self.capacity = max
39583
ee087f0d7db5 util: allow lrucachedict to track cost of entries
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39582
diff changeset
1520 self.totalcost = 0
39584
842cd0bdda75 util: teach lrucachedict to enforce a max total cost
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39583
diff changeset
1521 self.maxcost = maxcost
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1522
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1523 def __len__(self):
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1524 return len(self._cache)
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1525
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1526 def __contains__(self, k):
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1527 return k in self._cache
18603
2251b3184e6e util: add an LRU cache dict
Siddharth Agarwal <sid0@fb.com>
parents: 18537
diff changeset
1528
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1529 def __iter__(self):
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1530 # We don't have to iterate in cache order, but why not.
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1531 n = self._head
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1532 for i in range(len(self._cache)):
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1533 yield n.key
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1534 n = n.next
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1535
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1536 def __getitem__(self, k):
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1537 node = self._cache[k]
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1538 self._movetohead(node)
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1539 return node.value
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1540
39583
ee087f0d7db5 util: allow lrucachedict to track cost of entries
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39582
diff changeset
1541 def insert(self, k, v, cost=0):
ee087f0d7db5 util: allow lrucachedict to track cost of entries
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39582
diff changeset
1542 """Insert a new item in the cache with optional cost value."""
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1543 node = self._cache.get(k)
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1544 # Replace existing value and mark as newest.
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1545 if node is not None:
39583
ee087f0d7db5 util: allow lrucachedict to track cost of entries
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39582
diff changeset
1546 self.totalcost -= node.cost
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1547 node.value = v
39583
ee087f0d7db5 util: allow lrucachedict to track cost of entries
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39582
diff changeset
1548 node.cost = cost
ee087f0d7db5 util: allow lrucachedict to track cost of entries
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39582
diff changeset
1549 self.totalcost += cost
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1550 self._movetohead(node)
39584
842cd0bdda75 util: teach lrucachedict to enforce a max total cost
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39583
diff changeset
1551
842cd0bdda75 util: teach lrucachedict to enforce a max total cost
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39583
diff changeset
1552 if self.maxcost:
842cd0bdda75 util: teach lrucachedict to enforce a max total cost
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39583
diff changeset
1553 self._enforcecostlimit()
842cd0bdda75 util: teach lrucachedict to enforce a max total cost
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39583
diff changeset
1554
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1555 return
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1556
39580
5d75a3c16193 util: make capacity a public attribute on lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39579
diff changeset
1557 if self._size < self.capacity:
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1558 node = self._addcapacity()
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1559 else:
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1560 # Grab the last/oldest item.
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1561 node = self._head.prev
18603
2251b3184e6e util: add an LRU cache dict
Siddharth Agarwal <sid0@fb.com>
parents: 18537
diff changeset
1562
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1563 # At capacity. Kill the old entry.
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1564 if node.key is not _notset:
39583
ee087f0d7db5 util: allow lrucachedict to track cost of entries
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39582
diff changeset
1565 self.totalcost -= node.cost
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1566 del self._cache[node.key]
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1567
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1568 node.key = k
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1569 node.value = v
39583
ee087f0d7db5 util: allow lrucachedict to track cost of entries
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39582
diff changeset
1570 node.cost = cost
ee087f0d7db5 util: allow lrucachedict to track cost of entries
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39582
diff changeset
1571 self.totalcost += cost
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1572 self._cache[k] = node
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1573 # And mark it as newest entry. No need to adjust order since it
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1574 # is already self._head.prev.
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1575 self._head = node
18603
2251b3184e6e util: add an LRU cache dict
Siddharth Agarwal <sid0@fb.com>
parents: 18537
diff changeset
1576
39584
842cd0bdda75 util: teach lrucachedict to enforce a max total cost
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39583
diff changeset
1577 if self.maxcost:
842cd0bdda75 util: teach lrucachedict to enforce a max total cost
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39583
diff changeset
1578 self._enforcecostlimit()
842cd0bdda75 util: teach lrucachedict to enforce a max total cost
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39583
diff changeset
1579
39583
ee087f0d7db5 util: allow lrucachedict to track cost of entries
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39582
diff changeset
1580 def __setitem__(self, k, v):
ee087f0d7db5 util: allow lrucachedict to track cost of entries
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39582
diff changeset
1581 self.insert(k, v)
ee087f0d7db5 util: allow lrucachedict to track cost of entries
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39582
diff changeset
1582
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1583 def __delitem__(self, k):
40898
7cda0cacbbf6 util: implement pop() on lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40897
diff changeset
1584 self.pop(k)
7cda0cacbbf6 util: implement pop() on lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40897
diff changeset
1585
7cda0cacbbf6 util: implement pop() on lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40897
diff changeset
1586 def pop(self, k, default=_notset):
7cda0cacbbf6 util: implement pop() on lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40897
diff changeset
1587 try:
7cda0cacbbf6 util: implement pop() on lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40897
diff changeset
1588 node = self._cache.pop(k)
7cda0cacbbf6 util: implement pop() on lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40897
diff changeset
1589 except KeyError:
7cda0cacbbf6 util: implement pop() on lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40897
diff changeset
1590 if default is _notset:
7cda0cacbbf6 util: implement pop() on lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40897
diff changeset
1591 raise
7cda0cacbbf6 util: implement pop() on lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40897
diff changeset
1592 return default
43887
b5655f337bd7 typing: add a couple of assertions to lrucachedict to help pytype
Matt Harbison <matt_harbison@yahoo.com>
parents: 43886
diff changeset
1593
40898
7cda0cacbbf6 util: implement pop() on lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40897
diff changeset
1594 value = node.value
39583
ee087f0d7db5 util: allow lrucachedict to track cost of entries
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39582
diff changeset
1595 self.totalcost -= node.cost
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1596 node.markempty()
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1597
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1598 # Temporarily mark as newest item before re-adjusting head to make
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1599 # this node the oldest item.
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1600 self._movetohead(node)
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1601 self._head = node.next
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1602
40898
7cda0cacbbf6 util: implement pop() on lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40897
diff changeset
1603 return value
7cda0cacbbf6 util: implement pop() on lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40897
diff changeset
1604
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1605 # Additional dict methods.
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1606
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1607 def get(self, k, default=None):
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1608 try:
39587
8f2c0d1b454c util: update lrucachedict order during get()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39586
diff changeset
1609 return self.__getitem__(k)
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1610 except KeyError:
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1611 return default
18603
2251b3184e6e util: add an LRU cache dict
Siddharth Agarwal <sid0@fb.com>
parents: 18537
diff changeset
1612
40897
0c638ff69f5c util: add method to peek item in lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40728
diff changeset
1613 def peek(self, k, default=_notset):
0c638ff69f5c util: add method to peek item in lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40728
diff changeset
1614 """Get the specified item without moving it to the head
0c638ff69f5c util: add method to peek item in lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40728
diff changeset
1615
0c638ff69f5c util: add method to peek item in lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40728
diff changeset
1616 Unlike get(), this doesn't mutate the internal state. But be aware
0c638ff69f5c util: add method to peek item in lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40728
diff changeset
1617 that it doesn't mean peek() is thread safe.
0c638ff69f5c util: add method to peek item in lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40728
diff changeset
1618 """
0c638ff69f5c util: add method to peek item in lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40728
diff changeset
1619 try:
0c638ff69f5c util: add method to peek item in lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40728
diff changeset
1620 node = self._cache[k]
0c638ff69f5c util: add method to peek item in lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40728
diff changeset
1621 return node.value
0c638ff69f5c util: add method to peek item in lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40728
diff changeset
1622 except KeyError:
0c638ff69f5c util: add method to peek item in lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40728
diff changeset
1623 if default is _notset:
0c638ff69f5c util: add method to peek item in lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40728
diff changeset
1624 raise
0c638ff69f5c util: add method to peek item in lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40728
diff changeset
1625 return default
0c638ff69f5c util: add method to peek item in lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40728
diff changeset
1626
19710
887ffa22fd0d lrucachedict: implement clear()
Siddharth Agarwal <sid0@fb.com>
parents: 19461
diff changeset
1627 def clear(self):
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1628 n = self._head
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1629 while n.key is not _notset:
39583
ee087f0d7db5 util: allow lrucachedict to track cost of entries
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39582
diff changeset
1630 self.totalcost -= n.cost
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1631 n.markempty()
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1632 n = n.next
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1633
19710
887ffa22fd0d lrucachedict: implement clear()
Siddharth Agarwal <sid0@fb.com>
parents: 19461
diff changeset
1634 self._cache.clear()
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1635
39584
842cd0bdda75 util: teach lrucachedict to enforce a max total cost
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39583
diff changeset
1636 def copy(self, capacity=None, maxcost=0):
39581
2dcc68c7d25b util: ability to change capacity when copying lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39580
diff changeset
1637 """Create a new cache as a copy of the current one.
2dcc68c7d25b util: ability to change capacity when copying lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39580
diff changeset
1638
2dcc68c7d25b util: ability to change capacity when copying lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39580
diff changeset
1639 By default, the new cache has the same capacity as the existing one.
2dcc68c7d25b util: ability to change capacity when copying lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39580
diff changeset
1640 But, the cache capacity can be changed as part of performing the
2dcc68c7d25b util: ability to change capacity when copying lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39580
diff changeset
1641 copy.
2dcc68c7d25b util: ability to change capacity when copying lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39580
diff changeset
1642
2dcc68c7d25b util: ability to change capacity when copying lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39580
diff changeset
1643 Items in the copy have an insertion/access order matching this
2dcc68c7d25b util: ability to change capacity when copying lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39580
diff changeset
1644 instance.
2dcc68c7d25b util: ability to change capacity when copying lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39580
diff changeset
1645 """
2dcc68c7d25b util: ability to change capacity when copying lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39580
diff changeset
1646
2dcc68c7d25b util: ability to change capacity when copying lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39580
diff changeset
1647 capacity = capacity or self.capacity
39584
842cd0bdda75 util: teach lrucachedict to enforce a max total cost
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39583
diff changeset
1648 maxcost = maxcost or self.maxcost
842cd0bdda75 util: teach lrucachedict to enforce a max total cost
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39583
diff changeset
1649 result = lrucachedict(capacity, maxcost=maxcost)
39579
b31b01f93b11 util: properly copy lrucachedict instances
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39287
diff changeset
1650
b31b01f93b11 util: properly copy lrucachedict instances
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39287
diff changeset
1651 # We copy entries by iterating in oldest-to-newest order so the copy
b31b01f93b11 util: properly copy lrucachedict instances
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39287
diff changeset
1652 # has the correct ordering.
b31b01f93b11 util: properly copy lrucachedict instances
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39287
diff changeset
1653
b31b01f93b11 util: properly copy lrucachedict instances
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39287
diff changeset
1654 # Find the first non-empty entry.
27576
6cd3044985c2 lrucachedict: add copy method
Eric Sumner <ericsumner@fb.com>
parents: 27391
diff changeset
1655 n = self._head.prev
39579
b31b01f93b11 util: properly copy lrucachedict instances
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39287
diff changeset
1656 while n.key is _notset and n is not self._head:
b31b01f93b11 util: properly copy lrucachedict instances
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39287
diff changeset
1657 n = n.prev
b31b01f93b11 util: properly copy lrucachedict instances
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39287
diff changeset
1658
39581
2dcc68c7d25b util: ability to change capacity when copying lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39580
diff changeset
1659 # We could potentially skip the first N items when decreasing capacity.
2dcc68c7d25b util: ability to change capacity when copying lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39580
diff changeset
1660 # But let's keep it simple unless it is a performance problem.
27576
6cd3044985c2 lrucachedict: add copy method
Eric Sumner <ericsumner@fb.com>
parents: 27391
diff changeset
1661 for i in range(len(self._cache)):
39583
ee087f0d7db5 util: allow lrucachedict to track cost of entries
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39582
diff changeset
1662 result.insert(n.key, n.value, cost=n.cost)
27576
6cd3044985c2 lrucachedict: add copy method
Eric Sumner <ericsumner@fb.com>
parents: 27391
diff changeset
1663 n = n.prev
39579
b31b01f93b11 util: properly copy lrucachedict instances
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39287
diff changeset
1664
27576
6cd3044985c2 lrucachedict: add copy method
Eric Sumner <ericsumner@fb.com>
parents: 27391
diff changeset
1665 return result
6cd3044985c2 lrucachedict: add copy method
Eric Sumner <ericsumner@fb.com>
parents: 27391
diff changeset
1666
39582
bd9d3a89f07b util: add a popoldest() method to lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39581
diff changeset
1667 def popoldest(self):
bd9d3a89f07b util: add a popoldest() method to lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39581
diff changeset
1668 """Remove the oldest item from the cache.
bd9d3a89f07b util: add a popoldest() method to lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39581
diff changeset
1669
bd9d3a89f07b util: add a popoldest() method to lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39581
diff changeset
1670 Returns the (key, value) describing the removed cache entry.
bd9d3a89f07b util: add a popoldest() method to lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39581
diff changeset
1671 """
bd9d3a89f07b util: add a popoldest() method to lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39581
diff changeset
1672 if not self._cache:
bd9d3a89f07b util: add a popoldest() method to lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39581
diff changeset
1673 return
bd9d3a89f07b util: add a popoldest() method to lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39581
diff changeset
1674
bd9d3a89f07b util: add a popoldest() method to lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39581
diff changeset
1675 # Walk the linked list backwards starting at tail node until we hit
bd9d3a89f07b util: add a popoldest() method to lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39581
diff changeset
1676 # a non-empty node.
bd9d3a89f07b util: add a popoldest() method to lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39581
diff changeset
1677 n = self._head.prev
46901
51841b23670b typing: make minor adjustments to mercurial/util.py to pass pytype checking
Matt Harbison <matt_harbison@yahoo.com>
parents: 46900
diff changeset
1678
39582
bd9d3a89f07b util: add a popoldest() method to lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39581
diff changeset
1679 while n.key is _notset:
bd9d3a89f07b util: add a popoldest() method to lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39581
diff changeset
1680 n = n.prev
bd9d3a89f07b util: add a popoldest() method to lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39581
diff changeset
1681
bd9d3a89f07b util: add a popoldest() method to lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39581
diff changeset
1682 key, value = n.key, n.value
bd9d3a89f07b util: add a popoldest() method to lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39581
diff changeset
1683
bd9d3a89f07b util: add a popoldest() method to lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39581
diff changeset
1684 # And remove it from the cache and mark it as empty.
bd9d3a89f07b util: add a popoldest() method to lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39581
diff changeset
1685 del self._cache[n.key]
39583
ee087f0d7db5 util: allow lrucachedict to track cost of entries
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39582
diff changeset
1686 self.totalcost -= n.cost
39582
bd9d3a89f07b util: add a popoldest() method to lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39581
diff changeset
1687 n.markempty()
bd9d3a89f07b util: add a popoldest() method to lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39581
diff changeset
1688
bd9d3a89f07b util: add a popoldest() method to lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39581
diff changeset
1689 return key, value
bd9d3a89f07b util: add a popoldest() method to lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39581
diff changeset
1690
51315
c8a2fdf5ca37 pytype: add the couple annotations for pytype to understands the lrunode
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51311
diff changeset
1691 def _movetohead(self, node: _lrucachenode):
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1692 """Mark a node as the newest, making it the new head.
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1693
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1694 When a node is accessed, it becomes the freshest entry in the LRU
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1695 list, which is denoted by self._head.
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1696
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1697 Visually, let's make ``N`` the new head node (* denotes head):
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1698
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1699 previous/oldest <-> head <-> next/next newest
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1700
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1701 ----<->--- A* ---<->-----
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1702 | |
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1703 E <-> D <-> N <-> C <-> B
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1704
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1705 To:
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1706
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1707 ----<->--- N* ---<->-----
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1708 | |
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1709 E <-> D <-> C <-> B <-> A
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1710
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1711 This requires the following moves:
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1712
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1713 C.next = D (node.prev.next = node.next)
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1714 D.prev = C (node.next.prev = node.prev)
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1715 E.next = N (head.prev.next = node)
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1716 N.prev = E (node.prev = head.prev)
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1717 N.next = A (node.next = head)
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1718 A.prev = N (head.prev = node)
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1719 """
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1720 head = self._head
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1721 # C.next = D
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1722 node.prev.next = node.next
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1723 # D.prev = C
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1724 node.next.prev = node.prev
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1725 # N.prev = E
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1726 node.prev = head.prev
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1727 # N.next = A
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1728 # It is tempting to do just "head" here, however if node is
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1729 # adjacent to head, this will do bad things.
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1730 node.next = head.prev.next
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1731 # E.next = N
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1732 node.next.prev = node
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1733 # A.prev = N
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1734 node.prev.next = node
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1735
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1736 self._head = node
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1737
51315
c8a2fdf5ca37 pytype: add the couple annotations for pytype to understands the lrunode
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51311
diff changeset
1738 def _addcapacity(self) -> _lrucachenode:
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1739 """Add a node to the circular linked list.
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1740
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1741 The new node is inserted before the head node.
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1742 """
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1743 head = self._head
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1744 node = _lrucachenode()
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1745 head.prev.next = node
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1746 node.prev = head.prev
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1747 node.next = head
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1748 head.prev = node
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1749 self._size += 1
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1750 return node
19710
887ffa22fd0d lrucachedict: implement clear()
Siddharth Agarwal <sid0@fb.com>
parents: 19461
diff changeset
1751
39584
842cd0bdda75 util: teach lrucachedict to enforce a max total cost
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39583
diff changeset
1752 def _enforcecostlimit(self):
842cd0bdda75 util: teach lrucachedict to enforce a max total cost
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39583
diff changeset
1753 # This should run after an insertion. It should only be called if total
842cd0bdda75 util: teach lrucachedict to enforce a max total cost
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39583
diff changeset
1754 # cost limits are being enforced.
842cd0bdda75 util: teach lrucachedict to enforce a max total cost
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39583
diff changeset
1755 # The most recently inserted node is never evicted.
39585
cc23c09bc562 util: optimize cost auditing on insert
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39584
diff changeset
1756 if len(self) <= 1 or self.totalcost <= self.maxcost:
cc23c09bc562 util: optimize cost auditing on insert
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39584
diff changeset
1757 return
cc23c09bc562 util: optimize cost auditing on insert
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39584
diff changeset
1758
cc23c09bc562 util: optimize cost auditing on insert
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39584
diff changeset
1759 # This is logically equivalent to calling popoldest() until we
cc23c09bc562 util: optimize cost auditing on insert
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39584
diff changeset
1760 # free up enough cost. We don't do that since popoldest() needs
cc23c09bc562 util: optimize cost auditing on insert
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39584
diff changeset
1761 # to walk the linked list and doing this in a loop would be
cc23c09bc562 util: optimize cost auditing on insert
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39584
diff changeset
1762 # quadratic. So we find the first non-empty node and then
cc23c09bc562 util: optimize cost auditing on insert
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39584
diff changeset
1763 # walk nodes until we free up enough capacity.
39586
f296c0b366c8 util: lower water mark when removing nodes after cost limit reached
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39585
diff changeset
1764 #
f296c0b366c8 util: lower water mark when removing nodes after cost limit reached
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39585
diff changeset
1765 # If we only removed the minimum number of nodes to free enough
f296c0b366c8 util: lower water mark when removing nodes after cost limit reached
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39585
diff changeset
1766 # cost at insert time, chances are high that the next insert would
f296c0b366c8 util: lower water mark when removing nodes after cost limit reached
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39585
diff changeset
1767 # also require pruning. This would effectively constitute quadratic
f296c0b366c8 util: lower water mark when removing nodes after cost limit reached
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39585
diff changeset
1768 # behavior for insert-heavy workloads. To mitigate this, we set a
f296c0b366c8 util: lower water mark when removing nodes after cost limit reached
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39585
diff changeset
1769 # target cost that is a percentage of the max cost. This will tend
f296c0b366c8 util: lower water mark when removing nodes after cost limit reached
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39585
diff changeset
1770 # to free more nodes when the high water mark is reached, which
f296c0b366c8 util: lower water mark when removing nodes after cost limit reached
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39585
diff changeset
1771 # lowers the chances of needing to prune on the subsequent insert.
f296c0b366c8 util: lower water mark when removing nodes after cost limit reached
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39585
diff changeset
1772 targetcost = int(self.maxcost * 0.75)
f296c0b366c8 util: lower water mark when removing nodes after cost limit reached
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39585
diff changeset
1773
39585
cc23c09bc562 util: optimize cost auditing on insert
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39584
diff changeset
1774 n = self._head.prev
cc23c09bc562 util: optimize cost auditing on insert
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39584
diff changeset
1775 while n.key is _notset:
cc23c09bc562 util: optimize cost auditing on insert
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39584
diff changeset
1776 n = n.prev
cc23c09bc562 util: optimize cost auditing on insert
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39584
diff changeset
1777
39586
f296c0b366c8 util: lower water mark when removing nodes after cost limit reached
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39585
diff changeset
1778 while len(self) > 1 and self.totalcost > targetcost:
39585
cc23c09bc562 util: optimize cost auditing on insert
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39584
diff changeset
1779 del self._cache[n.key]
cc23c09bc562 util: optimize cost auditing on insert
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39584
diff changeset
1780 self.totalcost -= n.cost
cc23c09bc562 util: optimize cost auditing on insert
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39584
diff changeset
1781 n.markempty()
cc23c09bc562 util: optimize cost auditing on insert
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39584
diff changeset
1782 n = n.prev
39584
842cd0bdda75 util: teach lrucachedict to enforce a max total cost
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39583
diff changeset
1783
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1784
9097
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
1785 def lrucachefunc(func):
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
1786 '''cache most recent results of function calls'''
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
1787 cache = {}
25113
0ca8410ea345 util: drop alias for collections.deque
Martin von Zweigbergk <martinvonz@google.com>
parents: 25112
diff changeset
1788 order = collections.deque()
28832
f5ff10f6fa6b util: use __code__ (available since py2.6)
timeless <timeless@mozdev.org>
parents: 28826
diff changeset
1789 if func.__code__.co_argcount == 1:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1790
9097
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
1791 def f(arg):
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
1792 if arg not in cache:
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
1793 if len(cache) > 20:
16803
107a3270a24a cleanup: use the deque type where appropriate
Bryan O'Sullivan <bryano@fb.com>
parents: 16769
diff changeset
1794 del cache[order.popleft()]
9097
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
1795 cache[arg] = func(arg)
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
1796 else:
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
1797 order.remove(arg)
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
1798 order.append(arg)
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
1799 return cache[arg]
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1800
9097
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
1801 else:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1802
9097
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
1803 def f(*args):
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
1804 if args not in cache:
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
1805 if len(cache) > 20:
16803
107a3270a24a cleanup: use the deque type where appropriate
Bryan O'Sullivan <bryano@fb.com>
parents: 16769
diff changeset
1806 del cache[order.popleft()]
9097
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
1807 cache[args] = func(*args)
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
1808 else:
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
1809 order.remove(args)
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
1810 order.append(args)
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
1811 return cache[args]
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
1812
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
1813 return f
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
1814
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1815
49037
642e31cb55f0 py3: use class X: instead of class X(object):
Gregory Szorc <gregory.szorc@gmail.com>
parents: 49031
diff changeset
1816 class propertycache:
8207
dd8d5be57d65 util: take propertycache from context.py
Matt Mackall <mpm@selenic.com>
parents: 8181
diff changeset
1817 def __init__(self, func):
dd8d5be57d65 util: take propertycache from context.py
Matt Mackall <mpm@selenic.com>
parents: 8181
diff changeset
1818 self.func = func
dd8d5be57d65 util: take propertycache from context.py
Matt Mackall <mpm@selenic.com>
parents: 8181
diff changeset
1819 self.name = func.__name__
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1820
8207
dd8d5be57d65 util: take propertycache from context.py
Matt Mackall <mpm@selenic.com>
parents: 8181
diff changeset
1821 def __get__(self, obj, type=None):
dd8d5be57d65 util: take propertycache from context.py
Matt Mackall <mpm@selenic.com>
parents: 8181
diff changeset
1822 result = self.func(obj)
18013
98c867ac1330 clfilter: add a propertycache that must be unfiltered
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 17962
diff changeset
1823 self.cachevalue(obj, result)
8207
dd8d5be57d65 util: take propertycache from context.py
Matt Mackall <mpm@selenic.com>
parents: 8181
diff changeset
1824 return result
dd8d5be57d65 util: take propertycache from context.py
Matt Mackall <mpm@selenic.com>
parents: 8181
diff changeset
1825
18013
98c867ac1330 clfilter: add a propertycache that must be unfiltered
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 17962
diff changeset
1826 def cachevalue(self, obj, value):
19951
d51c4d85ec23 spelling: random spell checker fixes
Mads Kiilerich <madski@unity3d.com>
parents: 19852
diff changeset
1827 # __dict__ assignment required to bypass __setattr__ (eg: repoview)
19845
a1237a4b437d repoview: make propertycache.setcache compatible with repoview
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 19461
diff changeset
1828 obj.__dict__[self.name] = value
18013
98c867ac1330 clfilter: add a propertycache that must be unfiltered
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 17962
diff changeset
1829
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1830
35053
be6aa0cff8ea util: add util.clearcachedproperty
Mark Thomas <mbthomas@fb.com>
parents: 34647
diff changeset
1831 def clearcachedproperty(obj, prop):
be6aa0cff8ea util: add util.clearcachedproperty
Mark Thomas <mbthomas@fb.com>
parents: 34647
diff changeset
1832 '''clear a cached property value, if one has been set'''
40728
475921a3028c py3: cast attribute name to sysstr in clearcachedproperty()
Yuya Nishihara <yuya@tcha.org>
parents: 40251
diff changeset
1833 prop = pycompat.sysstr(prop)
35053
be6aa0cff8ea util: add util.clearcachedproperty
Mark Thomas <mbthomas@fb.com>
parents: 34647
diff changeset
1834 if prop in obj.__dict__:
be6aa0cff8ea util: add util.clearcachedproperty
Mark Thomas <mbthomas@fb.com>
parents: 34647
diff changeset
1835 del obj.__dict__[prop]
be6aa0cff8ea util: add util.clearcachedproperty
Mark Thomas <mbthomas@fb.com>
parents: 34647
diff changeset
1836
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1837
7396
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1838 def increasingchunks(source, min=1024, max=65536):
45957
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
1839 """return no less than min bytes per chunk while data remains,
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
1840 doubling min after each chunk until it reaches max"""
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1841
7396
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1842 def log2(x):
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1843 if not x:
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1844 return 0
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1845 i = 0
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1846 while x:
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1847 x >>= 1
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1848 i += 1
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1849 return i - 1
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1850
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1851 buf = []
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1852 blen = 0
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1853 for chunk in source:
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1854 buf.append(chunk)
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1855 blen += len(chunk)
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1856 if blen >= min:
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1857 if min < max:
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1858 min = min << 1
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1859 nmin = 1 << log2(blen)
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1860 if nmin > min:
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1861 min = nmin
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1862 if min > max:
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1863 min = max
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1864 yield b''.join(buf)
7396
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1865 blen = 0
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1866 buf = []
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1867 if buf:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1868 yield b''.join(buf)
7396
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1869
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1870
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10264
diff changeset
1871 def always(fn):
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10264
diff changeset
1872 return True
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10264
diff changeset
1873
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1874
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10264
diff changeset
1875 def never(fn):
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10264
diff changeset
1876 return False
724
1c0c413cccdd Get add and locate to use new repo and dirstate walk code.
Bryan O'Sullivan <bos@serpentine.com>
parents: 705
diff changeset
1877
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1878
51587
a452807df09b nocg: make the utility work are both a decorator and context manager
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51428
diff changeset
1879 def nogc(func=None) -> Any:
23495
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
1880 """disable garbage collector
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
1881
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
1882 Python's garbage collector triggers a GC each time a certain number of
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
1883 container objects (the number being defined by gc.get_threshold()) are
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
1884 allocated even when marked not to be tracked by the collector. Tracking has
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
1885 no effect on when GCs are triggered, only on what objects the GC looks
23543
4dd8a6a1240d spelling: fixes from proofreading of spell checker issues
Mads Kiilerich <madski@unity3d.com>
parents: 23495
diff changeset
1886 into. As a workaround, disable GC while building complex (huge)
23495
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
1887 containers.
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
1888
33820
05264fc9d8d6 util: make nogc effective for CPython
Jun Wu <quark@fb.com>
parents: 33814
diff changeset
1889 This garbage collector issue have been fixed in 2.7. But it still affect
05264fc9d8d6 util: make nogc effective for CPython
Jun Wu <quark@fb.com>
parents: 33814
diff changeset
1890 CPython's performance.
23495
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
1891 """
51587
a452807df09b nocg: make the utility work are both a decorator and context manager
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51428
diff changeset
1892 if func is None:
a452807df09b nocg: make the utility work are both a decorator and context manager
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51428
diff changeset
1893 return _nogc_context()
a452807df09b nocg: make the utility work are both a decorator and context manager
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51428
diff changeset
1894 else:
a452807df09b nocg: make the utility work are both a decorator and context manager
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51428
diff changeset
1895 return _nogc_decorator(func)
a452807df09b nocg: make the utility work are both a decorator and context manager
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51428
diff changeset
1896
a452807df09b nocg: make the utility work are both a decorator and context manager
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51428
diff changeset
1897
a452807df09b nocg: make the utility work are both a decorator and context manager
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51428
diff changeset
1898 @contextlib.contextmanager
a452807df09b nocg: make the utility work are both a decorator and context manager
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51428
diff changeset
1899 def _nogc_context():
a452807df09b nocg: make the utility work are both a decorator and context manager
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51428
diff changeset
1900 gcenabled = gc.isenabled()
a452807df09b nocg: make the utility work are both a decorator and context manager
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51428
diff changeset
1901 gc.disable()
a452807df09b nocg: make the utility work are both a decorator and context manager
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51428
diff changeset
1902 try:
a452807df09b nocg: make the utility work are both a decorator and context manager
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51428
diff changeset
1903 yield
a452807df09b nocg: make the utility work are both a decorator and context manager
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51428
diff changeset
1904 finally:
a452807df09b nocg: make the utility work are both a decorator and context manager
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51428
diff changeset
1905 if gcenabled:
a452807df09b nocg: make the utility work are both a decorator and context manager
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51428
diff changeset
1906 gc.enable()
a452807df09b nocg: make the utility work are both a decorator and context manager
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51428
diff changeset
1907
a452807df09b nocg: make the utility work are both a decorator and context manager
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51428
diff changeset
1908
a452807df09b nocg: make the utility work are both a decorator and context manager
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51428
diff changeset
1909 def _nogc_decorator(func):
23495
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
1910 def wrapper(*args, **kwargs):
51587
a452807df09b nocg: make the utility work are both a decorator and context manager
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51428
diff changeset
1911 with _nogc_context():
23495
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
1912 return func(*args, **kwargs)
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1913
23495
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
1914 return wrapper
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
1915
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1916
33820
05264fc9d8d6 util: make nogc effective for CPython
Jun Wu <quark@fb.com>
parents: 33814
diff changeset
1917 if pycompat.ispypy:
05264fc9d8d6 util: make nogc effective for CPython
Jun Wu <quark@fb.com>
parents: 33814
diff changeset
1918 # PyPy runs slower with gc disabled
05264fc9d8d6 util: make nogc effective for CPython
Jun Wu <quark@fb.com>
parents: 33814
diff changeset
1919 nogc = lambda x: x
05264fc9d8d6 util: make nogc effective for CPython
Jun Wu <quark@fb.com>
parents: 33814
diff changeset
1920
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1921
51304
f15cb5111a1e pytype: move some type comment to proper annotation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51302
diff changeset
1922 def pathto(root: bytes, n1: bytes, n2: bytes) -> bytes:
45957
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
1923 """return the relative path from one place to another.
4229
24c22a3f2ef8 pass repo.root to util.pathto() in preparation for the next patch
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4129
diff changeset
1924 root should use os.sep to separate directories
3669
48768b1ab23c fix util.pathto
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 3629
diff changeset
1925 n1 should use os.sep to separate directories
48768b1ab23c fix util.pathto
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 3629
diff changeset
1926 n2 should use "/" to separate directories
48768b1ab23c fix util.pathto
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 3629
diff changeset
1927 returns an os.sep-separated path.
4229
24c22a3f2ef8 pass repo.root to util.pathto() in preparation for the next patch
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4129
diff changeset
1928
24c22a3f2ef8 pass repo.root to util.pathto() in preparation for the next patch
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4129
diff changeset
1929 If n1 is a relative path, it's assumed it's
24c22a3f2ef8 pass repo.root to util.pathto() in preparation for the next patch
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4129
diff changeset
1930 relative to root.
24c22a3f2ef8 pass repo.root to util.pathto() in preparation for the next patch
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4129
diff changeset
1931 n2 should always be relative to root.
45957
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
1932 """
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10264
diff changeset
1933 if not n1:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10264
diff changeset
1934 return localpath(n2)
4230
c93562fb12cc Fix handling of paths when run outside the repo.
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4229
diff changeset
1935 if os.path.isabs(n1):
c93562fb12cc Fix handling of paths when run outside the repo.
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4229
diff changeset
1936 if os.path.splitdrive(root)[0] != os.path.splitdrive(n1)[0]:
c93562fb12cc Fix handling of paths when run outside the repo.
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4229
diff changeset
1937 return os.path.join(root, localpath(n2))
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1938 n2 = b'/'.join((pconvert(root), n2))
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1939 a, b = splitpath(n1), n2.split(b'/')
1541
bf4e7ef08741 fixed some stuff pychecker shows, marked unclear/wrong stuff with XXX
twaldmann@thinkmo.de
parents: 1528
diff changeset
1940 a.reverse()
bf4e7ef08741 fixed some stuff pychecker shows, marked unclear/wrong stuff with XXX
twaldmann@thinkmo.de
parents: 1528
diff changeset
1941 b.reverse()
884
087771ebe2e6 Fix walk code for files that do not exist anywhere, and unhandled types.
Bryan O'Sullivan <bos@serpentine.com>
parents: 878
diff changeset
1942 while a and b and a[-1] == b[-1]:
1541
bf4e7ef08741 fixed some stuff pychecker shows, marked unclear/wrong stuff with XXX
twaldmann@thinkmo.de
parents: 1528
diff changeset
1943 a.pop()
bf4e7ef08741 fixed some stuff pychecker shows, marked unclear/wrong stuff with XXX
twaldmann@thinkmo.de
parents: 1528
diff changeset
1944 b.pop()
884
087771ebe2e6 Fix walk code for files that do not exist anywhere, and unhandled types.
Bryan O'Sullivan <bos@serpentine.com>
parents: 878
diff changeset
1945 b.reverse()
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1946 return pycompat.ossep.join(([b'..'] * len(a)) + b) or b'.'
884
087771ebe2e6 Fix walk code for files that do not exist anywhere, and unhandled types.
Bryan O'Sullivan <bos@serpentine.com>
parents: 878
diff changeset
1947
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1948
44456
09f3e003fc2a phabricator: avoid a stacktrace when command arguments are missing
Matt Harbison <matt_harbison@yahoo.com>
parents: 44060
diff changeset
1949 def checksignature(func, depth=1):
7388
5751631246de dispatch: generalize signature checking for extension command wrapping
Matt Mackall <mpm@selenic.com>
parents: 7301
diff changeset
1950 '''wrap a function with code to check for calling errors'''
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1951
7388
5751631246de dispatch: generalize signature checking for extension command wrapping
Matt Mackall <mpm@selenic.com>
parents: 7301
diff changeset
1952 def check(*args, **kwargs):
5751631246de dispatch: generalize signature checking for extension command wrapping
Matt Mackall <mpm@selenic.com>
parents: 7301
diff changeset
1953 try:
5751631246de dispatch: generalize signature checking for extension command wrapping
Matt Mackall <mpm@selenic.com>
parents: 7301
diff changeset
1954 return func(*args, **kwargs)
5751631246de dispatch: generalize signature checking for extension command wrapping
Matt Mackall <mpm@selenic.com>
parents: 7301
diff changeset
1955 except TypeError:
44456
09f3e003fc2a phabricator: avoid a stacktrace when command arguments are missing
Matt Harbison <matt_harbison@yahoo.com>
parents: 44060
diff changeset
1956 if len(traceback.extract_tb(sys.exc_info()[2])) == depth:
7646
e62a456b8dc5 error: move SignatureError
Matt Mackall <mpm@selenic.com>
parents: 7644
diff changeset
1957 raise error.SignatureError
7388
5751631246de dispatch: generalize signature checking for extension command wrapping
Matt Mackall <mpm@selenic.com>
parents: 7301
diff changeset
1958 raise
5751631246de dispatch: generalize signature checking for extension command wrapping
Matt Mackall <mpm@selenic.com>
parents: 7301
diff changeset
1959
5751631246de dispatch: generalize signature checking for extension command wrapping
Matt Mackall <mpm@selenic.com>
parents: 7301
diff changeset
1960 return check
5751631246de dispatch: generalize signature checking for extension command wrapping
Matt Mackall <mpm@selenic.com>
parents: 7301
diff changeset
1961
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1962
31581
e506e461c7a9 util: disable hardlink for copyfile if fstype is outside a whitelist
Jun Wu <quark@fb.com>
parents: 31579
diff changeset
1963 # a whilelist of known filesystems where hardlink works reliably
32331
bd872f64a8ba cleanup: use set literals
Martin von Zweigbergk <martinvonz@google.com>
parents: 32330
diff changeset
1964 _hardlinkfswhitelist = {
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1965 b'apfs',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1966 b'btrfs',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1967 b'ext2',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1968 b'ext3',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1969 b'ext4',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1970 b'hfs',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1971 b'jfs',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1972 b'NTFS',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1973 b'reiserfs',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1974 b'tmpfs',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1975 b'ufs',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1976 b'xfs',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1977 b'zfs',
32331
bd872f64a8ba cleanup: use set literals
Martin von Zweigbergk <martinvonz@google.com>
parents: 32330
diff changeset
1978 }
31581
e506e461c7a9 util: disable hardlink for copyfile if fstype is outside a whitelist
Jun Wu <quark@fb.com>
parents: 31579
diff changeset
1979
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
1980
47406
9b841267253c util: add `nb_bytes` argument to `copyfile` to partially copy a file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47244
diff changeset
1981 def copyfile(
47451
d756fc11cfb9 copyfile: add a option callback for failed hardlinking
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47423
diff changeset
1982 src,
d756fc11cfb9 copyfile: add a option callback for failed hardlinking
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47423
diff changeset
1983 dest,
d756fc11cfb9 copyfile: add a option callback for failed hardlinking
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47423
diff changeset
1984 hardlink=False,
d756fc11cfb9 copyfile: add a option callback for failed hardlinking
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47423
diff changeset
1985 copystat=False,
d756fc11cfb9 copyfile: add a option callback for failed hardlinking
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47423
diff changeset
1986 checkambig=False,
d756fc11cfb9 copyfile: add a option callback for failed hardlinking
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47423
diff changeset
1987 nb_bytes=None,
d756fc11cfb9 copyfile: add a option callback for failed hardlinking
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47423
diff changeset
1988 no_hardlink_cb=None,
47452
09ff5d532a25 copyfiles: add a way to relax the file system checking for hardlink
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47451
diff changeset
1989 check_fs_hardlink=True,
47406
9b841267253c util: add `nb_bytes` argument to `copyfile` to partially copy a file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47244
diff changeset
1990 ):
45957
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
1991 """copy a file, preserving mode and optionally other stat info like
29367
4e6e280e238f doc: describe detail about checkambig optional argument
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29342
diff changeset
1992 atime/mtime
4e6e280e238f doc: describe detail about checkambig optional argument
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29342
diff changeset
1993
4e6e280e238f doc: describe detail about checkambig optional argument
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29342
diff changeset
1994 checkambig argument is used with filestat, and is useful only if
4e6e280e238f doc: describe detail about checkambig optional argument
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29342
diff changeset
1995 destination file is guarded by any lock (e.g. repo.lock or
4e6e280e238f doc: describe detail about checkambig optional argument
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29342
diff changeset
1996 repo.wlock).
4e6e280e238f doc: describe detail about checkambig optional argument
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29342
diff changeset
1997
4e6e280e238f doc: describe detail about checkambig optional argument
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29342
diff changeset
1998 copystat and checkambig should be exclusive.
47406
9b841267253c util: add `nb_bytes` argument to `copyfile` to partially copy a file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47244
diff changeset
1999
9b841267253c util: add `nb_bytes` argument to `copyfile` to partially copy a file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47244
diff changeset
2000 nb_bytes: if set only copy the first `nb_bytes` of the source file.
45957
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
2001 """
29204
ce2d81aafbae util: make copyfile avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29201
diff changeset
2002 assert not (copystat and checkambig)
ce2d81aafbae util: make copyfile avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29201
diff changeset
2003 oldstat = None
18326
614f769e6aa7 util: copyfile: remove dest before copying
Mads Kiilerich <mads@kiilerich.com>
parents: 18026
diff changeset
2004 if os.path.lexists(dest):
29204
ce2d81aafbae util: make copyfile avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29201
diff changeset
2005 if checkambig:
32792
7ad95626f6a7 filestat: move __init__ to frompath constructor
Siddharth Agarwal <sid0@fb.com>
parents: 32769
diff changeset
2006 oldstat = checkambig and filestat.frompath(dest)
18326
614f769e6aa7 util: copyfile: remove dest before copying
Mads Kiilerich <mads@kiilerich.com>
parents: 18026
diff changeset
2007 unlink(dest)
47452
09ff5d532a25 copyfiles: add a way to relax the file system checking for hardlink
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47451
diff changeset
2008 if hardlink and check_fs_hardlink:
31581
e506e461c7a9 util: disable hardlink for copyfile if fstype is outside a whitelist
Jun Wu <quark@fb.com>
parents: 31579
diff changeset
2009 # Hardlinks are problematic on CIFS (issue4546), do not allow hardlinks
e506e461c7a9 util: disable hardlink for copyfile if fstype is outside a whitelist
Jun Wu <quark@fb.com>
parents: 31579
diff changeset
2010 # unless we are confident that dest is on a whitelisted filesystem.
31683
1ed57a7dd904 statfs: make getfstype() raise OSError
Yuya Nishihara <yuya@tcha.org>
parents: 31667
diff changeset
2011 try:
1ed57a7dd904 statfs: make getfstype() raise OSError
Yuya Nishihara <yuya@tcha.org>
parents: 31667
diff changeset
2012 fstype = getfstype(os.path.dirname(dest))
1ed57a7dd904 statfs: make getfstype() raise OSError
Yuya Nishihara <yuya@tcha.org>
parents: 31667
diff changeset
2013 except OSError:
1ed57a7dd904 statfs: make getfstype() raise OSError
Yuya Nishihara <yuya@tcha.org>
parents: 31667
diff changeset
2014 fstype = None
31581
e506e461c7a9 util: disable hardlink for copyfile if fstype is outside a whitelist
Jun Wu <quark@fb.com>
parents: 31579
diff changeset
2015 if fstype not in _hardlinkfswhitelist:
47451
d756fc11cfb9 copyfile: add a option callback for failed hardlinking
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47423
diff changeset
2016 if no_hardlink_cb is not None:
d756fc11cfb9 copyfile: add a option callback for failed hardlinking
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47423
diff changeset
2017 no_hardlink_cb()
31581
e506e461c7a9 util: disable hardlink for copyfile if fstype is outside a whitelist
Jun Wu <quark@fb.com>
parents: 31579
diff changeset
2018 hardlink = False
31583
e7a02e9ad162 util: enable hardlink for copyfile
Jun Wu <quark@fb.com>
parents: 31581
diff changeset
2019 if hardlink:
23899
4e451d1359de copyfile: allow optional hardlinking
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23864
diff changeset
2020 try:
4e451d1359de copyfile: allow optional hardlinking
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23864
diff changeset
2021 oslink(src, dest)
47406
9b841267253c util: add `nb_bytes` argument to `copyfile` to partially copy a file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47244
diff changeset
2022 if nb_bytes is not None:
9b841267253c util: add `nb_bytes` argument to `copyfile` to partially copy a file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47244
diff changeset
2023 m = "the `nb_bytes` argument is incompatible with `hardlink`"
9b841267253c util: add `nb_bytes` argument to `copyfile` to partially copy a file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47244
diff changeset
2024 raise error.ProgrammingError(m)
23899
4e451d1359de copyfile: allow optional hardlinking
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23864
diff changeset
2025 return
52665
24ee91ba9aa8 pyupgrade: drop usage of py3 aliases for `OSError`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52392
diff changeset
2026 except OSError as exc:
47451
d756fc11cfb9 copyfile: add a option callback for failed hardlinking
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47423
diff changeset
2027 if exc.errno != errno.EEXIST and no_hardlink_cb is not None:
d756fc11cfb9 copyfile: add a option callback for failed hardlinking
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47423
diff changeset
2028 no_hardlink_cb()
d756fc11cfb9 copyfile: add a option callback for failed hardlinking
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47423
diff changeset
2029 # fall back to normal copy
4271
1eaa8d90c689 fix util.copyfile to deal with symlinks
Eric St-Jean <esj@wwd.ca>
parents: 4256
diff changeset
2030 if os.path.islink(src):
1eaa8d90c689 fix util.copyfile to deal with symlinks
Eric St-Jean <esj@wwd.ca>
parents: 4256
diff changeset
2031 os.symlink(os.readlink(src), dest)
27369
c48ecc0b5bc9 copyfile: add an optional parameter to copy other stat data
Siddharth Agarwal <sid0@fb.com>
parents: 26665
diff changeset
2032 # copytime is ignored for symlinks, but in general copytime isn't needed
c48ecc0b5bc9 copyfile: add an optional parameter to copy other stat data
Siddharth Agarwal <sid0@fb.com>
parents: 26665
diff changeset
2033 # for them anyway
47406
9b841267253c util: add `nb_bytes` argument to `copyfile` to partially copy a file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47244
diff changeset
2034 if nb_bytes is not None:
9b841267253c util: add `nb_bytes` argument to `copyfile` to partially copy a file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47244
diff changeset
2035 m = "cannot use `nb_bytes` on a symlink"
9b841267253c util: add `nb_bytes` argument to `copyfile` to partially copy a file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47244
diff changeset
2036 raise error.ProgrammingError(m)
4271
1eaa8d90c689 fix util.copyfile to deal with symlinks
Eric St-Jean <esj@wwd.ca>
parents: 4256
diff changeset
2037 else:
1eaa8d90c689 fix util.copyfile to deal with symlinks
Eric St-Jean <esj@wwd.ca>
parents: 4256
diff changeset
2038 try:
1eaa8d90c689 fix util.copyfile to deal with symlinks
Eric St-Jean <esj@wwd.ca>
parents: 4256
diff changeset
2039 shutil.copyfile(src, dest)
27369
c48ecc0b5bc9 copyfile: add an optional parameter to copy other stat data
Siddharth Agarwal <sid0@fb.com>
parents: 26665
diff changeset
2040 if copystat:
c48ecc0b5bc9 copyfile: add an optional parameter to copy other stat data
Siddharth Agarwal <sid0@fb.com>
parents: 26665
diff changeset
2041 # copystat also copies mode
c48ecc0b5bc9 copyfile: add an optional parameter to copy other stat data
Siddharth Agarwal <sid0@fb.com>
parents: 26665
diff changeset
2042 shutil.copystat(src, dest)
c48ecc0b5bc9 copyfile: add an optional parameter to copy other stat data
Siddharth Agarwal <sid0@fb.com>
parents: 26665
diff changeset
2043 else:
c48ecc0b5bc9 copyfile: add an optional parameter to copy other stat data
Siddharth Agarwal <sid0@fb.com>
parents: 26665
diff changeset
2044 shutil.copymode(src, dest)
29204
ce2d81aafbae util: make copyfile avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29201
diff changeset
2045 if oldstat and oldstat.stat:
32792
7ad95626f6a7 filestat: move __init__ to frompath constructor
Siddharth Agarwal <sid0@fb.com>
parents: 32769
diff changeset
2046 newstat = filestat.frompath(dest)
29204
ce2d81aafbae util: make copyfile avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29201
diff changeset
2047 if newstat.isambig(oldstat):
ce2d81aafbae util: make copyfile avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29201
diff changeset
2048 # stat of copied file is ambiguous to original one
36789
ffa3026d4196 cleanup: use stat_result[stat.ST_MTIME] instead of stat_result.st_mtime
Augie Fackler <augie@google.com>
parents: 36747
diff changeset
2049 advanced = (
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2050 oldstat.stat[stat.ST_MTIME] + 1
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2051 ) & 0x7FFFFFFF
29204
ce2d81aafbae util: make copyfile avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29201
diff changeset
2052 os.utime(dest, (advanced, advanced))
47406
9b841267253c util: add `nb_bytes` argument to `copyfile` to partially copy a file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47244
diff changeset
2053 # We could do something smarter using `copy_file_range` call or similar
9b841267253c util: add `nb_bytes` argument to `copyfile` to partially copy a file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47244
diff changeset
2054 if nb_bytes is not None:
9b841267253c util: add `nb_bytes` argument to `copyfile` to partially copy a file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47244
diff changeset
2055 with open(dest, mode='r+') as f:
9b841267253c util: add `nb_bytes` argument to `copyfile` to partially copy a file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47244
diff changeset
2056 f.truncate(nb_bytes)
25660
328739ea70c3 global: mass rewrite to use modern exception syntax
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25420
diff changeset
2057 except shutil.Error as inst:
43440
5be128f669d4 util: convert an exception to bytes when passing to Abort()
Matt Harbison <matt_harbison@yahoo.com>
parents: 43117
diff changeset
2058 raise error.Abort(stringutil.forcebytestr(inst))
3629
4cfb72bcb978 util: add copyfile function
Matt Mackall <mpm@selenic.com>
parents: 3568
diff changeset
2059
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2060
38386
63e6f5ae84bc copystore: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 38323
diff changeset
2061 def copyfiles(src, dst, hardlink=None, progress=None):
24439
2ddfac2f163e util: add progress callback support to copyfiles
Augie Fackler <augie@google.com>
parents: 24236
diff changeset
2062 """Copy a directory tree using hardlinks if possible."""
2ddfac2f163e util: add progress callback support to copyfiles
Augie Fackler <augie@google.com>
parents: 24236
diff changeset
2063 num = 0
1241
3b4f05ff3130 Add support for cloning with hardlinks on windows.
Stephen Darnell
parents: 1207
diff changeset
2064
38386
63e6f5ae84bc copystore: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 38323
diff changeset
2065 def settopic():
63e6f5ae84bc copystore: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 38323
diff changeset
2066 if progress:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2067 progress.topic = _(b'linking') if hardlink else _(b'copying')
698
df78d8ccac4c Use python function instead of external 'cp' command when cloning repos.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 667
diff changeset
2068
1207
a7b8812973d9 Rewrite copytree as copyfiles
mpm@selenic.com
parents: 1200
diff changeset
2069 if os.path.isdir(src):
31724
456efd1b51fd hardlink: duplicate hardlink detection for copying files and directories
Jun Wu <quark@fb.com>
parents: 31723
diff changeset
2070 if hardlink is None:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2071 hardlink = (
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2072 os.stat(src).st_dev == os.stat(os.path.dirname(dst)).st_dev
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2073 )
38386
63e6f5ae84bc copystore: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 38323
diff changeset
2074 settopic()
1207
a7b8812973d9 Rewrite copytree as copyfiles
mpm@selenic.com
parents: 1200
diff changeset
2075 os.mkdir(dst)
32248
d74b0cff94a9 osutil: proxy through util (and platform) modules (API)
Yuya Nishihara <yuya@tcha.org>
parents: 32245
diff changeset
2076 for name, kind in listdir(src):
1207
a7b8812973d9 Rewrite copytree as copyfiles
mpm@selenic.com
parents: 1200
diff changeset
2077 srcname = os.path.join(src, name)
a7b8812973d9 Rewrite copytree as copyfiles
mpm@selenic.com
parents: 1200
diff changeset
2078 dstname = os.path.join(dst, name)
38386
63e6f5ae84bc copystore: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 38323
diff changeset
2079 hardlink, n = copyfiles(srcname, dstname, hardlink, progress)
11251
c61442f6d106 clone: print number of linked/copied files on --debug
Adrian Buehlmann <adrian@cadifra.com>
parents: 11232
diff changeset
2080 num += n
1207
a7b8812973d9 Rewrite copytree as copyfiles
mpm@selenic.com
parents: 1200
diff changeset
2081 else:
31724
456efd1b51fd hardlink: duplicate hardlink detection for copying files and directories
Jun Wu <quark@fb.com>
parents: 31723
diff changeset
2082 if hardlink is None:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2083 hardlink = (
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2084 os.stat(os.path.dirname(src)).st_dev
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2085 == os.stat(os.path.dirname(dst)).st_dev
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2086 )
38386
63e6f5ae84bc copystore: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 38323
diff changeset
2087 settopic()
31724
456efd1b51fd hardlink: duplicate hardlink detection for copying files and directories
Jun Wu <quark@fb.com>
parents: 31723
diff changeset
2088
1241
3b4f05ff3130 Add support for cloning with hardlinks on windows.
Stephen Darnell
parents: 1207
diff changeset
2089 if hardlink:
3b4f05ff3130 Add support for cloning with hardlinks on windows.
Stephen Darnell
parents: 1207
diff changeset
2090 try:
14235
b9e1b041744f rename util.os_link to oslink
Adrian Buehlmann <adrian@cadifra.com>
parents: 14234
diff changeset
2091 oslink(src, dst)
52665
24ee91ba9aa8 pyupgrade: drop usage of py3 aliases for `OSError`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52392
diff changeset
2092 except OSError as exc:
47423
9ea525216edb copyfiles: deal with existing file when hardlinking
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47406
diff changeset
2093 if exc.errno != errno.EEXIST:
9ea525216edb copyfiles: deal with existing file when hardlinking
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47406
diff changeset
2094 hardlink = False
9ea525216edb copyfiles: deal with existing file when hardlinking
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47406
diff changeset
2095 # XXX maybe try to relink if the file exist ?
1591
5a3229cf1492 do not copy atime and mtime in util.copyfiles
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 1585
diff changeset
2096 shutil.copy(src, dst)
1241
3b4f05ff3130 Add support for cloning with hardlinks on windows.
Stephen Darnell
parents: 1207
diff changeset
2097 else:
1591
5a3229cf1492 do not copy atime and mtime in util.copyfiles
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 1585
diff changeset
2098 shutil.copy(src, dst)
11251
c61442f6d106 clone: print number of linked/copied files on --debug
Adrian Buehlmann <adrian@cadifra.com>
parents: 11232
diff changeset
2099 num += 1
38386
63e6f5ae84bc copystore: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 38323
diff changeset
2100 if progress:
63e6f5ae84bc copystore: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 38323
diff changeset
2101 progress.increment()
698
df78d8ccac4c Use python function instead of external 'cp' command when cloning repos.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 667
diff changeset
2102
11251
c61442f6d106 clone: print number of linked/copied files on --debug
Adrian Buehlmann <adrian@cadifra.com>
parents: 11232
diff changeset
2103 return hardlink, num
11254
640d419725d0 util.copyfiles: don't try os_link() again if it failed before
Adrian Buehlmann <adrian@cadifra.com>
parents: 11010
diff changeset
2104
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2105
34069
ca6a3852daf0 util: use set for reserved Windows filenames
Gregory Szorc <gregory.szorc@gmail.com>
parents: 34020
diff changeset
2106 _winreservednames = {
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2107 b'con',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2108 b'prn',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2109 b'aux',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2110 b'nul',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2111 b'com1',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2112 b'com2',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2113 b'com3',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2114 b'com4',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2115 b'com5',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2116 b'com6',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2117 b'com7',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2118 b'com8',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2119 b'com9',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2120 b'lpt1',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2121 b'lpt2',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2122 b'lpt3',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2123 b'lpt4',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2124 b'lpt5',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2125 b'lpt6',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2126 b'lpt7',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2127 b'lpt8',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2128 b'lpt9',
34069
ca6a3852daf0 util: use set for reserved Windows filenames
Gregory Szorc <gregory.szorc@gmail.com>
parents: 34020
diff changeset
2129 }
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2130 _winreservedchars = b':*?"<>|'
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2131
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2132
51304
f15cb5111a1e pytype: move some type comment to proper annotation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51302
diff changeset
2133 def checkwinfilename(path: bytes) -> Optional[bytes]:
45957
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
2134 r"""Check that the base-relative path is a valid filename on Windows.
13916
98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13910
diff changeset
2135 Returns None if the path is ok, or a UI string describing the problem.
98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13910
diff changeset
2136
34146
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
2137 >>> checkwinfilename(b"just/a/normal/path")
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
2138 >>> checkwinfilename(b"foo/bar/con.xml")
13916
98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13910
diff changeset
2139 "filename contains 'con', which is reserved on Windows"
34146
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
2140 >>> checkwinfilename(b"foo/con.xml/bar")
13916
98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13910
diff changeset
2141 "filename contains 'con', which is reserved on Windows"
34146
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
2142 >>> checkwinfilename(b"foo/bar/xml.con")
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
2143 >>> checkwinfilename(b"foo/bar/AUX/bla.txt")
13916
98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13910
diff changeset
2144 "filename contains 'AUX', which is reserved on Windows"
34146
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
2145 >>> checkwinfilename(b"foo/bar/bla:.txt")
13916
98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13910
diff changeset
2146 "filename contains ':', which is reserved on Windows"
34146
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
2147 >>> checkwinfilename(b"foo/bar/b\07la.txt")
20000
0849d280663e util: warn when adding paths ending with \
Mads Kiilerich <madski@unity3d.com>
parents: 19951
diff changeset
2148 "filename contains '\\x07', which is invalid on Windows"
34146
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
2149 >>> checkwinfilename(b"foo/bar/bla ")
13916
98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13910
diff changeset
2150 "filename ends with ' ', which is not allowed on Windows"
34146
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
2151 >>> checkwinfilename(b"../bar")
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
2152 >>> checkwinfilename(b"foo\\")
20000
0849d280663e util: warn when adding paths ending with \
Mads Kiilerich <madski@unity3d.com>
parents: 19951
diff changeset
2153 "filename ends with '\\', which is invalid on Windows"
34146
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
2154 >>> checkwinfilename(b"foo\\/bar")
20000
0849d280663e util: warn when adding paths ending with \
Mads Kiilerich <madski@unity3d.com>
parents: 19951
diff changeset
2155 "directory name ends with '\\', which is invalid on Windows"
45957
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
2156 """
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2157 if path.endswith(b'\\'):
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2158 return _(b"filename ends with '\\', which is invalid on Windows")
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2159 if b'\\/' in path:
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2160 return _(b"directory name ends with '\\', which is invalid on Windows")
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2161 for n in path.replace(b'\\', b'/').split(b'/'):
13916
98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13910
diff changeset
2162 if not n:
98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13910
diff changeset
2163 continue
32131
377c74ef008d win32mbcs: avoid unintentional failure at colorization
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 31952
diff changeset
2164 for c in _filenamebytestr(n):
14262
23cd7eeff678 util: rename _windows_reserved_filenames and _windows_reserved_chars
Adrian Buehlmann <adrian@cadifra.com>
parents: 14250
diff changeset
2165 if c in _winreservedchars:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2166 return (
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2167 _(
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2168 b"filename contains '%s', which is reserved "
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2169 b"on Windows"
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2170 )
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2171 % c
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2172 )
13916
98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13910
diff changeset
2173 if ord(c) <= 31:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2174 return _(
43117
8ff1ecfadcd1 cleanup: join string literals that are already on one line
Martin von Zweigbergk <martinvonz@google.com>
parents: 43106
diff changeset
2175 b"filename contains '%s', which is invalid on Windows"
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2176 ) % stringutil.escapestr(c)
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2177 base = n.split(b'.')[0]
14262
23cd7eeff678 util: rename _windows_reserved_filenames and _windows_reserved_chars
Adrian Buehlmann <adrian@cadifra.com>
parents: 14250
diff changeset
2178 if base and base.lower() in _winreservednames:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2179 return (
43117
8ff1ecfadcd1 cleanup: join string literals that are already on one line
Martin von Zweigbergk <martinvonz@google.com>
parents: 43106
diff changeset
2180 _(b"filename contains '%s', which is reserved on Windows")
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2181 % base
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2182 )
34362
7508a7dc95c1 py3: replace bytes[n] with slicing in checkwinfilename()
Yuya Nishihara <yuya@tcha.org>
parents: 34361
diff changeset
2183 t = n[-1:]
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2184 if t in b'. ' and n not in b'..':
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2185 return (
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2186 _(
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2187 b"filename ends with '%s', which is not allowed "
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2188 b"on Windows"
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2189 )
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2190 % t
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2191 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2192
13916
98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13910
diff changeset
2193
44020
f9d29e1d3354 util: avoid referencing `time.clock()` on Windows when missing (issue6238)
Matt Harbison <matt_harbison@yahoo.com>
parents: 43926
diff changeset
2194 timer = getattr(time, "perf_counter", None)
f9d29e1d3354 util: avoid referencing `time.clock()` on Windows when missing (issue6238)
Matt Harbison <matt_harbison@yahoo.com>
parents: 43926
diff changeset
2195
34645
75979c8d4572 codemod: use pycompat.iswindows
Jun Wu <quark@fb.com>
parents: 34554
diff changeset
2196 if pycompat.iswindows:
13916
98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13910
diff changeset
2197 checkosfilename = checkwinfilename
44020
f9d29e1d3354 util: avoid referencing `time.clock()` on Windows when missing (issue6238)
Matt Harbison <matt_harbison@yahoo.com>
parents: 43926
diff changeset
2198 if not timer:
51311
7d3b92e8df13 pytype: ignore attribute error for time.clock
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51304
diff changeset
2199 timer = time.clock # pytype: disable=module-attr
7890
e710f0f592b2 util: split out posix, windows, and win32 modules
Matt Mackall <mpm@selenic.com>
parents: 7879
diff changeset
2200 else:
43889
09bcbeacedc7 typing: suppress a warning that mercurial.windows.checkosfilename is missing
Matt Harbison <matt_harbison@yahoo.com>
parents: 43888
diff changeset
2201 # mercurial.windows doesn't have platform.checkosfilename
09bcbeacedc7 typing: suppress a warning that mercurial.windows.checkosfilename is missing
Matt Harbison <matt_harbison@yahoo.com>
parents: 43888
diff changeset
2202 checkosfilename = platform.checkosfilename # pytype: disable=module-attr
44020
f9d29e1d3354 util: avoid referencing `time.clock()` on Windows when missing (issue6238)
Matt Harbison <matt_harbison@yahoo.com>
parents: 43926
diff changeset
2203 if not timer:
f9d29e1d3354 util: avoid referencing `time.clock()` on Windows when missing (issue6238)
Matt Harbison <matt_harbison@yahoo.com>
parents: 43926
diff changeset
2204 timer = time.time
7890
e710f0f592b2 util: split out posix, windows, and win32 modules
Matt Mackall <mpm@selenic.com>
parents: 7879
diff changeset
2205
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2206
51926
cbd01bf33802 typing: add type annotations to `mercurial.util.makelock()`
Matt Harbison <matt_harbison@yahoo.com>
parents: 51925
diff changeset
2207 def makelock(info: bytes, pathname: bytes) -> None:
36724
d77c3b023393 lock: block signal interrupt while making a lock file
Yuya Nishihara <yuya@tcha.org>
parents: 36722
diff changeset
2208 """Create a lock file atomically if possible
d77c3b023393 lock: block signal interrupt while making a lock file
Yuya Nishihara <yuya@tcha.org>
parents: 36722
diff changeset
2209
d77c3b023393 lock: block signal interrupt while making a lock file
Yuya Nishihara <yuya@tcha.org>
parents: 36722
diff changeset
2210 This may leave a stale lock file if symlink isn't supported and signal
d77c3b023393 lock: block signal interrupt while making a lock file
Yuya Nishihara <yuya@tcha.org>
parents: 36722
diff changeset
2211 interrupt is enabled.
d77c3b023393 lock: block signal interrupt while making a lock file
Yuya Nishihara <yuya@tcha.org>
parents: 36722
diff changeset
2212 """
7890
e710f0f592b2 util: split out posix, windows, and win32 modules
Matt Mackall <mpm@selenic.com>
parents: 7879
diff changeset
2213 try:
e710f0f592b2 util: split out posix, windows, and win32 modules
Matt Mackall <mpm@selenic.com>
parents: 7879
diff changeset
2214 return os.symlink(info, pathname)
25660
328739ea70c3 global: mass rewrite to use modern exception syntax
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25420
diff changeset
2215 except OSError as why:
7890
e710f0f592b2 util: split out posix, windows, and win32 modules
Matt Mackall <mpm@selenic.com>
parents: 7879
diff changeset
2216 if why.errno == errno.EEXIST:
e710f0f592b2 util: split out posix, windows, and win32 modules
Matt Mackall <mpm@selenic.com>
parents: 7879
diff changeset
2217 raise
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2218 except AttributeError: # no symlink in os
7890
e710f0f592b2 util: split out posix, windows, and win32 modules
Matt Mackall <mpm@selenic.com>
parents: 7879
diff changeset
2219 pass
e710f0f592b2 util: split out posix, windows, and win32 modules
Matt Mackall <mpm@selenic.com>
parents: 7879
diff changeset
2220
36791
1fbbb8e83392 py3: read/write plain lock file in binary mode
Yuya Nishihara <yuya@tcha.org>
parents: 36790
diff changeset
2221 flags = os.O_CREAT | os.O_WRONLY | os.O_EXCL | getattr(os, 'O_BINARY', 0)
1fbbb8e83392 py3: read/write plain lock file in binary mode
Yuya Nishihara <yuya@tcha.org>
parents: 36790
diff changeset
2222 ld = os.open(pathname, flags)
51925
f833ad92ee44 util: avoid a leaked file descriptor in `util.makelock()` exceptional case
Matt Harbison <matt_harbison@yahoo.com>
parents: 51924
diff changeset
2223 try:
f833ad92ee44 util: avoid a leaked file descriptor in `util.makelock()` exceptional case
Matt Harbison <matt_harbison@yahoo.com>
parents: 51924
diff changeset
2224 os.write(ld, info)
f833ad92ee44 util: avoid a leaked file descriptor in `util.makelock()` exceptional case
Matt Harbison <matt_harbison@yahoo.com>
parents: 51924
diff changeset
2225 finally:
f833ad92ee44 util: avoid a leaked file descriptor in `util.makelock()` exceptional case
Matt Harbison <matt_harbison@yahoo.com>
parents: 51924
diff changeset
2226 os.close(ld)
704
5ca319a641e1 Make makelock and readlock work on filesystems without symlink support.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 698
diff changeset
2227
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2228
51304
f15cb5111a1e pytype: move some type comment to proper annotation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51302
diff changeset
2229 def readlock(pathname: bytes) -> bytes:
7890
e710f0f592b2 util: split out posix, windows, and win32 modules
Matt Mackall <mpm@selenic.com>
parents: 7879
diff changeset
2230 try:
39909
5fe0b880200e py3: convert os.readlink() path to native strings on Windows
Matt Harbison <matt_harbison@yahoo.com>
parents: 39793
diff changeset
2231 return readlink(pathname)
25660
328739ea70c3 global: mass rewrite to use modern exception syntax
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25420
diff changeset
2232 except OSError as why:
7890
e710f0f592b2 util: split out posix, windows, and win32 modules
Matt Mackall <mpm@selenic.com>
parents: 7879
diff changeset
2233 if why.errno not in (errno.EINVAL, errno.ENOSYS):
e710f0f592b2 util: split out posix, windows, and win32 modules
Matt Mackall <mpm@selenic.com>
parents: 7879
diff changeset
2234 raise
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2235 except AttributeError: # no symlink in os
7890
e710f0f592b2 util: split out posix, windows, and win32 modules
Matt Mackall <mpm@selenic.com>
parents: 7879
diff changeset
2236 pass
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2237 with posixfile(pathname, b'rb') as fp:
39910
4017968f0a1d util: use a context manager in readlock()
Matt Harbison <matt_harbison@yahoo.com>
parents: 39909
diff changeset
2238 return fp.read()
704
5ca319a641e1 Make makelock and readlock work on filesystems without symlink support.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 698
diff changeset
2239
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2240
2176
9b42304d9896 fix file handling bugs on windows.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2117
diff changeset
2241 def fstat(fp):
9b42304d9896 fix file handling bugs on windows.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2117
diff changeset
2242 '''stat file object that may not have fileno method.'''
9b42304d9896 fix file handling bugs on windows.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2117
diff changeset
2243 try:
9b42304d9896 fix file handling bugs on windows.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2117
diff changeset
2244 return os.fstat(fp.fileno())
9b42304d9896 fix file handling bugs on windows.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2117
diff changeset
2245 except AttributeError:
9b42304d9896 fix file handling bugs on windows.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2117
diff changeset
2246 return os.stat(fp.name)
9b42304d9896 fix file handling bugs on windows.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2117
diff changeset
2247
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2248
3784
1427949b8f80 imported patch folding
Matt Mackall <mpm@selenic.com>
parents: 3770
diff changeset
2249 # File system features
1427949b8f80 imported patch folding
Matt Mackall <mpm@selenic.com>
parents: 3770
diff changeset
2250
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2251
51304
f15cb5111a1e pytype: move some type comment to proper annotation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51302
diff changeset
2252 def fscasesensitive(path: bytes) -> bool:
3784
1427949b8f80 imported patch folding
Matt Mackall <mpm@selenic.com>
parents: 3770
diff changeset
2253 """
18911
451eb1c27c1b util: improve doc for checkcase
Mads Kiilerich <mads@kiilerich.com>
parents: 18868
diff changeset
2254 Return true if the given path is on a case-sensitive filesystem
3784
1427949b8f80 imported patch folding
Matt Mackall <mpm@selenic.com>
parents: 3770
diff changeset
2255
1427949b8f80 imported patch folding
Matt Mackall <mpm@selenic.com>
parents: 3770
diff changeset
2256 Requires a path (like /foo/.hg) ending with a foldable final
1427949b8f80 imported patch folding
Matt Mackall <mpm@selenic.com>
parents: 3770
diff changeset
2257 directory component.
1427949b8f80 imported patch folding
Matt Mackall <mpm@selenic.com>
parents: 3770
diff changeset
2258 """
24902
986a5c23b1c1 util.checkcase: don't abort on broken symlinks
Siddharth Agarwal <sid0@fb.com>
parents: 24692
diff changeset
2259 s1 = os.lstat(path)
3784
1427949b8f80 imported patch folding
Matt Mackall <mpm@selenic.com>
parents: 3770
diff changeset
2260 d, b = os.path.split(path)
15667
eacfd851cb9e icasefs: consider as case sensitive if there is no counterevidence, for safety
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15609
diff changeset
2261 b2 = b.upper()
eacfd851cb9e icasefs: consider as case sensitive if there is no counterevidence, for safety
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15609
diff changeset
2262 if b == b2:
eacfd851cb9e icasefs: consider as case sensitive if there is no counterevidence, for safety
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15609
diff changeset
2263 b2 = b.lower()
eacfd851cb9e icasefs: consider as case sensitive if there is no counterevidence, for safety
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15609
diff changeset
2264 if b == b2:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2265 return True # no evidence against case sensitivity
15667
eacfd851cb9e icasefs: consider as case sensitive if there is no counterevidence, for safety
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15609
diff changeset
2266 p2 = os.path.join(d, b2)
3784
1427949b8f80 imported patch folding
Matt Mackall <mpm@selenic.com>
parents: 3770
diff changeset
2267 try:
24902
986a5c23b1c1 util.checkcase: don't abort on broken symlinks
Siddharth Agarwal <sid0@fb.com>
parents: 24692
diff changeset
2268 s2 = os.lstat(p2)
3784
1427949b8f80 imported patch folding
Matt Mackall <mpm@selenic.com>
parents: 3770
diff changeset
2269 if s2 == s1:
1427949b8f80 imported patch folding
Matt Mackall <mpm@selenic.com>
parents: 3770
diff changeset
2270 return False
1427949b8f80 imported patch folding
Matt Mackall <mpm@selenic.com>
parents: 3770
diff changeset
2271 return True
14004
97ed99d1f419 eliminate various naked except clauses
Idan Kamara <idankk86@gmail.com>
parents: 13985
diff changeset
2272 except OSError:
3784
1427949b8f80 imported patch folding
Matt Mackall <mpm@selenic.com>
parents: 3770
diff changeset
2273 return True
1427949b8f80 imported patch folding
Matt Mackall <mpm@selenic.com>
parents: 3770
diff changeset
2274
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2275
46422
112826b59476 re2: feed unicode string to re2 module when necessary
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46372
diff changeset
2276 _re2_input = lambda x: x
51486
6c39edd1d348 re2: make errors quiet
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51428
diff changeset
2277 # google-re2 will need to be tell to not output error on its own
6c39edd1d348 re2: make errors quiet
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51428
diff changeset
2278 _re2_options = None
16943
8d08a28aa63e matcher: use re2 bindings if available
Bryan O'Sullivan <bryano@fb.com>
parents: 16873
diff changeset
2279 try:
43885
40bd667491a7 pytype: suppress the import-error in util.py when importing re2
Matt Harbison <matt_harbison@yahoo.com>
parents: 43774
diff changeset
2280 import re2 # pytype: disable=import-error
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2281
16943
8d08a28aa63e matcher: use re2 bindings if available
Bryan O'Sullivan <bryano@fb.com>
parents: 16873
diff changeset
2282 _re2 = None
8d08a28aa63e matcher: use re2 bindings if available
Bryan O'Sullivan <bryano@fb.com>
parents: 16873
diff changeset
2283 except ImportError:
8d08a28aa63e matcher: use re2 bindings if available
Bryan O'Sullivan <bryano@fb.com>
parents: 16873
diff changeset
2284 _re2 = False
8d08a28aa63e matcher: use re2 bindings if available
Bryan O'Sullivan <bryano@fb.com>
parents: 16873
diff changeset
2285
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2286
50398
a45460e235a2 re2: fix reporting of availability in `hg debuginstall`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50397
diff changeset
2287 def has_re2():
a45460e235a2 re2: fix reporting of availability in `hg debuginstall`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50397
diff changeset
2288 """return True is re2 is available, False otherwise"""
a45460e235a2 re2: fix reporting of availability in `hg debuginstall`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50397
diff changeset
2289 if _re2 is None:
a45460e235a2 re2: fix reporting of availability in `hg debuginstall`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50397
diff changeset
2290 _re._checkre2()
a45460e235a2 re2: fix reporting of availability in `hg debuginstall`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50397
diff changeset
2291 return _re2
a45460e235a2 re2: fix reporting of availability in `hg debuginstall`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50397
diff changeset
2292
a45460e235a2 re2: fix reporting of availability in `hg debuginstall`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50397
diff changeset
2293
49037
642e31cb55f0 py3: use class X: instead of class X(object):
Gregory Szorc <gregory.szorc@gmail.com>
parents: 49031
diff changeset
2294 class _re:
50397
293e1763982e re: make _checkre2 a static method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50396
diff changeset
2295 @staticmethod
293e1763982e re: make _checkre2 a static method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50396
diff changeset
2296 def _checkre2():
21913
50aad4609224 util.re: move check for re2 into a separate method
Siddharth Agarwal <sid0@fb.com>
parents: 21912
diff changeset
2297 global _re2
46422
112826b59476 re2: feed unicode string to re2 module when necessary
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46372
diff changeset
2298 global _re2_input
51486
6c39edd1d348 re2: make errors quiet
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51428
diff changeset
2299 global _re2_options
50396
82cf392c99f6 re2: exit `_checkre2` early if calling it is useless
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 49943
diff changeset
2300 if _re2 is not None:
82cf392c99f6 re2: exit `_checkre2` early if calling it is useless
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 49943
diff changeset
2301 # we already have the answer
82cf392c99f6 re2: exit `_checkre2` early if calling it is useless
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 49943
diff changeset
2302 return
46926
9c3e84569071 re2: adjust local variable assignment scope
Matt Harbison <matt_harbison@yahoo.com>
parents: 46907
diff changeset
2303
9c3e84569071 re2: adjust local variable assignment scope
Matt Harbison <matt_harbison@yahoo.com>
parents: 46907
diff changeset
2304 check_pattern = br'\[([^\[]+)\]'
9c3e84569071 re2: adjust local variable assignment scope
Matt Harbison <matt_harbison@yahoo.com>
parents: 46907
diff changeset
2305 check_input = b'[ui]'
21913
50aad4609224 util.re: move check for re2 into a separate method
Siddharth Agarwal <sid0@fb.com>
parents: 21912
diff changeset
2306 try:
50aad4609224 util.re: move check for re2 into a separate method
Siddharth Agarwal <sid0@fb.com>
parents: 21912
diff changeset
2307 # check if match works, see issue3964
46422
112826b59476 re2: feed unicode string to re2 module when necessary
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46372
diff changeset
2308 _re2 = bool(re2.match(check_pattern, check_input))
21913
50aad4609224 util.re: move check for re2 into a separate method
Siddharth Agarwal <sid0@fb.com>
parents: 21912
diff changeset
2309 except ImportError:
50aad4609224 util.re: move check for re2 into a separate method
Siddharth Agarwal <sid0@fb.com>
parents: 21912
diff changeset
2310 _re2 = False
46422
112826b59476 re2: feed unicode string to re2 module when necessary
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46372
diff changeset
2311 except TypeError:
112826b59476 re2: feed unicode string to re2 module when necessary
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46372
diff changeset
2312 # the `pyre-2` project provides a re2 module that accept bytes
112826b59476 re2: feed unicode string to re2 module when necessary
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46372
diff changeset
2313 # the `fb-re2` project provides a re2 module that acccept sysstr
112826b59476 re2: feed unicode string to re2 module when necessary
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46372
diff changeset
2314 check_pattern = pycompat.sysstr(check_pattern)
112826b59476 re2: feed unicode string to re2 module when necessary
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46372
diff changeset
2315 check_input = pycompat.sysstr(check_input)
112826b59476 re2: feed unicode string to re2 module when necessary
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46372
diff changeset
2316 _re2 = bool(re2.match(check_pattern, check_input))
112826b59476 re2: feed unicode string to re2 module when necessary
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46372
diff changeset
2317 _re2_input = pycompat.sysstr
51486
6c39edd1d348 re2: make errors quiet
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51428
diff changeset
2318 try:
6c39edd1d348 re2: make errors quiet
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51428
diff changeset
2319 quiet = re2.Options()
6c39edd1d348 re2: make errors quiet
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51428
diff changeset
2320 quiet.log_errors = False
6c39edd1d348 re2: make errors quiet
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51428
diff changeset
2321 _re2_options = quiet
6c39edd1d348 re2: make errors quiet
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51428
diff changeset
2322 except AttributeError:
6c39edd1d348 re2: make errors quiet
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51428
diff changeset
2323 pass
21913
50aad4609224 util.re: move check for re2 into a separate method
Siddharth Agarwal <sid0@fb.com>
parents: 21912
diff changeset
2324
21908
cad9dadc9d26 util: move compilere to a class
Siddharth Agarwal <sid0@fb.com>
parents: 21907
diff changeset
2325 def compile(self, pat, flags=0):
45957
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
2326 """Compile a regular expression, using re2 if possible
16943
8d08a28aa63e matcher: use re2 bindings if available
Bryan O'Sullivan <bryano@fb.com>
parents: 16873
diff changeset
2327
21908
cad9dadc9d26 util: move compilere to a class
Siddharth Agarwal <sid0@fb.com>
parents: 21907
diff changeset
2328 For best performance, use only re2-compatible regexp features. The
cad9dadc9d26 util: move compilere to a class
Siddharth Agarwal <sid0@fb.com>
parents: 21907
diff changeset
2329 only flags from the re module that are re2-compatible are
45957
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
2330 IGNORECASE and MULTILINE."""
21908
cad9dadc9d26 util: move compilere to a class
Siddharth Agarwal <sid0@fb.com>
parents: 21907
diff changeset
2331 if _re2 is None:
21913
50aad4609224 util.re: move check for re2 into a separate method
Siddharth Agarwal <sid0@fb.com>
parents: 21912
diff changeset
2332 self._checkre2()
21908
cad9dadc9d26 util: move compilere to a class
Siddharth Agarwal <sid0@fb.com>
parents: 21907
diff changeset
2333 if _re2 and (flags & ~(remod.IGNORECASE | remod.MULTILINE)) == 0:
cad9dadc9d26 util: move compilere to a class
Siddharth Agarwal <sid0@fb.com>
parents: 21907
diff changeset
2334 if flags & remod.IGNORECASE:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2335 pat = b'(?i)' + pat
21908
cad9dadc9d26 util: move compilere to a class
Siddharth Agarwal <sid0@fb.com>
parents: 21907
diff changeset
2336 if flags & remod.MULTILINE:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2337 pat = b'(?m)' + pat
21908
cad9dadc9d26 util: move compilere to a class
Siddharth Agarwal <sid0@fb.com>
parents: 21907
diff changeset
2338 try:
51486
6c39edd1d348 re2: make errors quiet
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51428
diff changeset
2339 input_regex = _re2_input(pat)
6c39edd1d348 re2: make errors quiet
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51428
diff changeset
2340 if _re2_options is not None:
6c39edd1d348 re2: make errors quiet
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51428
diff changeset
2341 compiled = re2.compile(input_regex, options=_re2_options)
6c39edd1d348 re2: make errors quiet
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51428
diff changeset
2342 else:
6c39edd1d348 re2: make errors quiet
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51428
diff changeset
2343 compiled = re2.compile(input_regex)
6c39edd1d348 re2: make errors quiet
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51428
diff changeset
2344 return compiled
21908
cad9dadc9d26 util: move compilere to a class
Siddharth Agarwal <sid0@fb.com>
parents: 21907
diff changeset
2345 except re2.error:
cad9dadc9d26 util: move compilere to a class
Siddharth Agarwal <sid0@fb.com>
parents: 21907
diff changeset
2346 pass
cad9dadc9d26 util: move compilere to a class
Siddharth Agarwal <sid0@fb.com>
parents: 21907
diff changeset
2347 return remod.compile(pat, flags)
cad9dadc9d26 util: move compilere to a class
Siddharth Agarwal <sid0@fb.com>
parents: 21907
diff changeset
2348
21914
10e99839a7a4 util.re: add an escape method
Siddharth Agarwal <sid0@fb.com>
parents: 21913
diff changeset
2349 @propertycache
10e99839a7a4 util.re: add an escape method
Siddharth Agarwal <sid0@fb.com>
parents: 21913
diff changeset
2350 def escape(self):
45957
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
2351 """Return the version of escape corresponding to self.compile.
21914
10e99839a7a4 util.re: add an escape method
Siddharth Agarwal <sid0@fb.com>
parents: 21913
diff changeset
2352
10e99839a7a4 util.re: add an escape method
Siddharth Agarwal <sid0@fb.com>
parents: 21913
diff changeset
2353 This is imperfect because whether re2 or re is used for a particular
10e99839a7a4 util.re: add an escape method
Siddharth Agarwal <sid0@fb.com>
parents: 21913
diff changeset
2354 function depends on the flags, etc, but it's the best we can do.
45957
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
2355 """
21914
10e99839a7a4 util.re: add an escape method
Siddharth Agarwal <sid0@fb.com>
parents: 21913
diff changeset
2356 global _re2
10e99839a7a4 util.re: add an escape method
Siddharth Agarwal <sid0@fb.com>
parents: 21913
diff changeset
2357 if _re2 is None:
10e99839a7a4 util.re: add an escape method
Siddharth Agarwal <sid0@fb.com>
parents: 21913
diff changeset
2358 self._checkre2()
10e99839a7a4 util.re: add an escape method
Siddharth Agarwal <sid0@fb.com>
parents: 21913
diff changeset
2359 if _re2:
10e99839a7a4 util.re: add an escape method
Siddharth Agarwal <sid0@fb.com>
parents: 21913
diff changeset
2360 return re2.escape
10e99839a7a4 util.re: add an escape method
Siddharth Agarwal <sid0@fb.com>
parents: 21913
diff changeset
2361 else:
10e99839a7a4 util.re: add an escape method
Siddharth Agarwal <sid0@fb.com>
parents: 21913
diff changeset
2362 return remod.escape
10e99839a7a4 util.re: add an escape method
Siddharth Agarwal <sid0@fb.com>
parents: 21913
diff changeset
2363
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2364
21908
cad9dadc9d26 util: move compilere to a class
Siddharth Agarwal <sid0@fb.com>
parents: 21907
diff changeset
2365 re = _re()
16943
8d08a28aa63e matcher: use re2 bindings if available
Bryan O'Sullivan <bryano@fb.com>
parents: 16873
diff changeset
2366
6676
33045179d079 Add a new function, fspath
Paul Moore <p.f.moore@gmail.com>
parents: 6595
diff changeset
2367 _fspathcache = {}
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2368
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2369
51304
f15cb5111a1e pytype: move some type comment to proper annotation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51302
diff changeset
2370 def fspath(name: bytes, root: bytes) -> bytes:
45957
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
2371 """Get name in the case stored in the filesystem
6676
33045179d079 Add a new function, fspath
Paul Moore <p.f.moore@gmail.com>
parents: 6595
diff changeset
2372
15710
f63e40047372 icasefs: avoid path-absoluteness/existance check in util.fspath() for efficiency
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15709
diff changeset
2373 The name should be relative to root, and be normcase-ed for efficiency.
f63e40047372 icasefs: avoid path-absoluteness/existance check in util.fspath() for efficiency
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15709
diff changeset
2374
f63e40047372 icasefs: avoid path-absoluteness/existance check in util.fspath() for efficiency
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15709
diff changeset
2375 Note that this function is unnecessary, and should not be
6676
33045179d079 Add a new function, fspath
Paul Moore <p.f.moore@gmail.com>
parents: 6595
diff changeset
2376 called, for case-sensitive filesystems (simply because it's expensive).
15670
d6c19cfa03ce icasefs: avoid normcase()-ing in util.fspath() for efficiency
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15669
diff changeset
2377
15710
f63e40047372 icasefs: avoid path-absoluteness/existance check in util.fspath() for efficiency
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15709
diff changeset
2378 The root should be normcase-ed, too.
45957
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
2379 """
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2380
23097
30124c40d11f util.fspath: use a dict rather than a linear scan for lookups
Siddharth Agarwal <sid0@fb.com>
parents: 23076
diff changeset
2381 def _makefspathcacheentry(dir):
44470
9d2b2df2c2ba cleanup: run pyupgrade on our source tree to clean up varying things
Augie Fackler <augie@google.com>
parents: 44456
diff changeset
2382 return {normcase(n): n for n in os.listdir(dir)}
15709
a1f4bd47d18e icasefs: retry directory scan once for already invalidated cache
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15674
diff changeset
2383
30618
1112ff99d965 py3: replace os.sep with pycompat.ossep (part 1 of 4)
Pulkit Goyal <7895pulkit@gmail.com>
parents: 30545
diff changeset
2384 seps = pycompat.ossep
30630
bcf4a975f93d py3: replace os.altsep with pycompat.altsep
Pulkit Goyal <7895pulkit@gmail.com>
parents: 30618
diff changeset
2385 if pycompat.osaltsep:
bcf4a975f93d py3: replace os.altsep with pycompat.altsep
Pulkit Goyal <7895pulkit@gmail.com>
parents: 30618
diff changeset
2386 seps = seps + pycompat.osaltsep
6676
33045179d079 Add a new function, fspath
Paul Moore <p.f.moore@gmail.com>
parents: 6595
diff changeset
2387 # Protect backslashes. This gets silly very quickly.
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2388 seps.replace(b'\\', b'\\\\')
31503
670166e0fcaa util: use bytes re on bytes input in fspath
Augie Fackler <augie@google.com>
parents: 31502
diff changeset
2389 pattern = remod.compile(br'([^%s]+)|([%s]+)' % (seps, seps))
15669
390bcd01775a icasefs: use util.normcase() instead of lower() or os.path.normcase in fspath
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15667
diff changeset
2390 dir = os.path.normpath(root)
6676
33045179d079 Add a new function, fspath
Paul Moore <p.f.moore@gmail.com>
parents: 6595
diff changeset
2391 result = []
33045179d079 Add a new function, fspath
Paul Moore <p.f.moore@gmail.com>
parents: 6595
diff changeset
2392 for part, sep in pattern.findall(name):
33045179d079 Add a new function, fspath
Paul Moore <p.f.moore@gmail.com>
parents: 6595
diff changeset
2393 if sep:
33045179d079 Add a new function, fspath
Paul Moore <p.f.moore@gmail.com>
parents: 6595
diff changeset
2394 result.append(sep)
33045179d079 Add a new function, fspath
Paul Moore <p.f.moore@gmail.com>
parents: 6595
diff changeset
2395 continue
33045179d079 Add a new function, fspath
Paul Moore <p.f.moore@gmail.com>
parents: 6595
diff changeset
2396
15719
1dd60426b061 icasefs: follow standard cache look up pattern
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15718
diff changeset
2397 if dir not in _fspathcache:
23097
30124c40d11f util.fspath: use a dict rather than a linear scan for lookups
Siddharth Agarwal <sid0@fb.com>
parents: 23076
diff changeset
2398 _fspathcache[dir] = _makefspathcacheentry(dir)
15719
1dd60426b061 icasefs: follow standard cache look up pattern
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15718
diff changeset
2399 contents = _fspathcache[dir]
6676
33045179d079 Add a new function, fspath
Paul Moore <p.f.moore@gmail.com>
parents: 6595
diff changeset
2400
23097
30124c40d11f util.fspath: use a dict rather than a linear scan for lookups
Siddharth Agarwal <sid0@fb.com>
parents: 23076
diff changeset
2401 found = contents.get(part)
15709
a1f4bd47d18e icasefs: retry directory scan once for already invalidated cache
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15674
diff changeset
2402 if not found:
15720
3bcfea777efc icasefs: rewrite comment to explain situtation precisely
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15719
diff changeset
2403 # retry "once per directory" per "dirstate.walk" which
3bcfea777efc icasefs: rewrite comment to explain situtation precisely
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15719
diff changeset
2404 # may take place for each patches of "hg qpush", for example
23097
30124c40d11f util.fspath: use a dict rather than a linear scan for lookups
Siddharth Agarwal <sid0@fb.com>
parents: 23076
diff changeset
2405 _fspathcache[dir] = contents = _makefspathcacheentry(dir)
30124c40d11f util.fspath: use a dict rather than a linear scan for lookups
Siddharth Agarwal <sid0@fb.com>
parents: 23076
diff changeset
2406 found = contents.get(part)
15709
a1f4bd47d18e icasefs: retry directory scan once for already invalidated cache
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15674
diff changeset
2407
a1f4bd47d18e icasefs: retry directory scan once for already invalidated cache
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15674
diff changeset
2408 result.append(found or part)
15669
390bcd01775a icasefs: use util.normcase() instead of lower() or os.path.normcase in fspath
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15667
diff changeset
2409 dir = os.path.join(dir, part)
6676
33045179d079 Add a new function, fspath
Paul Moore <p.f.moore@gmail.com>
parents: 6595
diff changeset
2410
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2411 return b''.join(result)
6676
33045179d079 Add a new function, fspath
Paul Moore <p.f.moore@gmail.com>
parents: 6595
diff changeset
2412
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2413
51304
f15cb5111a1e pytype: move some type comment to proper annotation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51302
diff changeset
2414 def checknlink(testfile: bytes) -> bool:
12938
bf826c0b9537 opener: check hardlink count reporting (issue1866)
Adrian Buehlmann <adrian@cadifra.com>
parents: 12927
diff changeset
2415 '''check whether hardlink count reporting works properly'''
bf826c0b9537 opener: check hardlink count reporting (issue1866)
Adrian Buehlmann <adrian@cadifra.com>
parents: 12927
diff changeset
2416
13204
5b83ab614dab checknlink: use two testfiles (issue2543)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13188
diff changeset
2417 # testfile may be open, so we need a separate file for checking to
5b83ab614dab checknlink: use two testfiles (issue2543)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13188
diff changeset
2418 # work around issue2543 (or testfile may get lost on Samba shares)
34101
6c5cdb02f2f9 checknlink: rename file object from 'fd' to 'fp'
Jun Wu <quark@fb.com>
parents: 34096
diff changeset
2419 f1, f2, fp = None, None, None
12938
bf826c0b9537 opener: check hardlink count reporting (issue1866)
Adrian Buehlmann <adrian@cadifra.com>
parents: 12927
diff changeset
2420 try:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2421 fd, f1 = pycompat.mkstemp(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2422 prefix=b'.%s-' % os.path.basename(testfile),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2423 suffix=b'1~',
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2424 dir=os.path.dirname(testfile),
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2425 )
34096
1104718fb090 checknlink: use a random temp file name for checking
Jun Wu <quark@fb.com>
parents: 34089
diff changeset
2426 os.close(fd)
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2427 f2 = b'%s2~' % f1[:-2]
34096
1104718fb090 checknlink: use a random temp file name for checking
Jun Wu <quark@fb.com>
parents: 34089
diff changeset
2428
25088
754df8e932d3 util: use try/except/finally
Matt Mackall <mpm@selenic.com>
parents: 24902
diff changeset
2429 oslink(f1, f2)
12938
bf826c0b9537 opener: check hardlink count reporting (issue1866)
Adrian Buehlmann <adrian@cadifra.com>
parents: 12927
diff changeset
2430 # nlinks() may behave differently for files on Windows shares if
bf826c0b9537 opener: check hardlink count reporting (issue1866)
Adrian Buehlmann <adrian@cadifra.com>
parents: 12927
diff changeset
2431 # the file is open.
34101
6c5cdb02f2f9 checknlink: rename file object from 'fd' to 'fp'
Jun Wu <quark@fb.com>
parents: 34096
diff changeset
2432 fp = posixfile(f2)
13204
5b83ab614dab checknlink: use two testfiles (issue2543)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13188
diff changeset
2433 return nlinks(f2) > 1
25088
754df8e932d3 util: use try/except/finally
Matt Mackall <mpm@selenic.com>
parents: 24902
diff changeset
2434 except OSError:
754df8e932d3 util: use try/except/finally
Matt Mackall <mpm@selenic.com>
parents: 24902
diff changeset
2435 return False
12938
bf826c0b9537 opener: check hardlink count reporting (issue1866)
Adrian Buehlmann <adrian@cadifra.com>
parents: 12927
diff changeset
2436 finally:
34101
6c5cdb02f2f9 checknlink: rename file object from 'fd' to 'fp'
Jun Wu <quark@fb.com>
parents: 34096
diff changeset
2437 if fp is not None:
6c5cdb02f2f9 checknlink: rename file object from 'fd' to 'fp'
Jun Wu <quark@fb.com>
parents: 34096
diff changeset
2438 fp.close()
13204
5b83ab614dab checknlink: use two testfiles (issue2543)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13188
diff changeset
2439 for f in (f1, f2):
5b83ab614dab checknlink: use two testfiles (issue2543)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13188
diff changeset
2440 try:
34096
1104718fb090 checknlink: use a random temp file name for checking
Jun Wu <quark@fb.com>
parents: 34089
diff changeset
2441 if f is not None:
1104718fb090 checknlink: use a random temp file name for checking
Jun Wu <quark@fb.com>
parents: 34089
diff changeset
2442 os.unlink(f)
13204
5b83ab614dab checknlink: use two testfiles (issue2543)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13188
diff changeset
2443 except OSError:
5b83ab614dab checknlink: use two testfiles (issue2543)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13188
diff changeset
2444 pass
12938
bf826c0b9537 opener: check hardlink count reporting (issue1866)
Adrian Buehlmann <adrian@cadifra.com>
parents: 12927
diff changeset
2445
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2446
51304
f15cb5111a1e pytype: move some type comment to proper annotation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51302
diff changeset
2447 def endswithsep(path: bytes) -> bool:
5843
83c354c4d529 Add endswithsep() and use it instead of using os.sep and os.altsep directly.
Shun-ichi GOTO <shunichi.goto@gmail.com>
parents: 5802
diff changeset
2448 '''Check path ends with os.sep or os.altsep.'''
46684
7711853110b9 typing: add some type annotations to mercurial/util.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 46114
diff changeset
2449 return bool( # help pytype
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2450 path.endswith(pycompat.ossep)
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2451 or pycompat.osaltsep
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2452 and path.endswith(pycompat.osaltsep)
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2453 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2454
5843
83c354c4d529 Add endswithsep() and use it instead of using os.sep and os.altsep directly.
Shun-ichi GOTO <shunichi.goto@gmail.com>
parents: 5802
diff changeset
2455
51304
f15cb5111a1e pytype: move some type comment to proper annotation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51302
diff changeset
2456 def splitpath(path: bytes) -> List[bytes]:
45957
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
2457 """Split path by os.sep.
5844
07d8eb78dd68 Add util.splitpath() and use it instead of using os.sep directly.
Shun-ichi GOTO <shunichi.goto@gmail.com>
parents: 5843
diff changeset
2458 Note that this function does not use os.altsep because this is
07d8eb78dd68 Add util.splitpath() and use it instead of using os.sep directly.
Shun-ichi GOTO <shunichi.goto@gmail.com>
parents: 5843
diff changeset
2459 an alternative of simple "xxx.split(os.sep)".
07d8eb78dd68 Add util.splitpath() and use it instead of using os.sep directly.
Shun-ichi GOTO <shunichi.goto@gmail.com>
parents: 5843
diff changeset
2460 It is recommended to use os.path.normpath() before using this
45957
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
2461 function if need."""
30618
1112ff99d965 py3: replace os.sep with pycompat.ossep (part 1 of 4)
Pulkit Goyal <7895pulkit@gmail.com>
parents: 30545
diff changeset
2462 return path.split(pycompat.ossep)
5844
07d8eb78dd68 Add util.splitpath() and use it instead of using os.sep directly.
Shun-ichi GOTO <shunichi.goto@gmail.com>
parents: 5843
diff changeset
2463
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2464
51843
62806be5cbda typing: add hints to `mercurial.util.mktempcopy()`
Matt Harbison <matt_harbison@yahoo.com>
parents: 51800
diff changeset
2465 def mktempcopy(
62806be5cbda typing: add hints to `mercurial.util.mktempcopy()`
Matt Harbison <matt_harbison@yahoo.com>
parents: 51800
diff changeset
2466 name: bytes,
62806be5cbda typing: add hints to `mercurial.util.mktempcopy()`
Matt Harbison <matt_harbison@yahoo.com>
parents: 51800
diff changeset
2467 emptyok: bool = False,
62806be5cbda typing: add hints to `mercurial.util.mktempcopy()`
Matt Harbison <matt_harbison@yahoo.com>
parents: 51800
diff changeset
2468 createmode: Optional[int] = None,
62806be5cbda typing: add hints to `mercurial.util.mktempcopy()`
Matt Harbison <matt_harbison@yahoo.com>
parents: 51800
diff changeset
2469 enforcewritable: bool = False,
62806be5cbda typing: add hints to `mercurial.util.mktempcopy()`
Matt Harbison <matt_harbison@yahoo.com>
parents: 51800
diff changeset
2470 ) -> bytes:
4827
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
2471 """Create a temporary file with the same contents from name
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
2472
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
2473 The permission bits are copied from the original file.
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
2474
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
2475 If the temporary file is going to be truncated immediately, you
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
2476 can use emptyok=True as an optimization.
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
2477
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
2478 Returns the name of the temporary file.
2176
9b42304d9896 fix file handling bugs on windows.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2117
diff changeset
2479 """
4827
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
2480 d, fn = os.path.split(name)
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2481 fd, temp = pycompat.mkstemp(prefix=b'.%s-' % fn, suffix=b'~', dir=d)
4827
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
2482 os.close(fd)
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
2483 # Temporary files are created with mode 0600, which is usually not
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
2484 # what we want. If the original file already exists, just copy
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
2485 # its mode. Otherwise, manually obey umask.
41289
593f6359681d update: fix edge-case with update.atomic-file and read-only files
Boris Feld <boris.feld@octobus.net>
parents: 40898
diff changeset
2486 copymode(name, temp, createmode, enforcewritable)
593f6359681d update: fix edge-case with update.atomic-file and read-only files
Boris Feld <boris.feld@octobus.net>
parents: 40898
diff changeset
2487
4827
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
2488 if emptyok:
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
2489 return temp
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
2490 try:
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
2491 try:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2492 ifp = posixfile(name, b"rb")
52665
24ee91ba9aa8 pyupgrade: drop usage of py3 aliases for `OSError`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52392
diff changeset
2493 except OSError as inst:
4827
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
2494 if inst.errno == errno.ENOENT:
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
2495 return temp
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
2496 if not getattr(inst, 'filename', None):
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
2497 inst.filename = name
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
2498 raise
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2499 ofp = posixfile(temp, b"wb")
4827
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
2500 for chunk in filechunkiter(ifp):
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
2501 ofp.write(chunk)
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
2502 ifp.close()
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
2503 ofp.close()
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2504 except: # re-raises
34436
5326e4ef1dab style: never put multiple statements on one line
Alex Gaynor <agaynor@mozilla.com>
parents: 34363
diff changeset
2505 try:
5326e4ef1dab style: never put multiple statements on one line
Alex Gaynor <agaynor@mozilla.com>
parents: 34363
diff changeset
2506 os.unlink(temp)
5326e4ef1dab style: never put multiple statements on one line
Alex Gaynor <agaynor@mozilla.com>
parents: 34363
diff changeset
2507 except OSError:
5326e4ef1dab style: never put multiple statements on one line
Alex Gaynor <agaynor@mozilla.com>
parents: 34363
diff changeset
2508 pass
4827
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
2509 raise
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
2510 return temp
2176
9b42304d9896 fix file handling bugs on windows.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2117
diff changeset
2511
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2512
49037
642e31cb55f0 py3: use class X: instead of class X(object):
Gregory Szorc <gregory.szorc@gmail.com>
parents: 49031
diff changeset
2513 class filestat:
29200
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2514 """help to exactly detect change of a file
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2515
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2516 'stat' attribute is result of 'os.stat()' if specified 'path'
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2517 exists. Otherwise, it is None. This can avoid preparative
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2518 'exists()' examination on client side of this class.
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2519 """
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2520
51924
1d95a87813ad typing: add type annotations to the `mercurial.util.filestat` class
Matt Harbison <matt_harbison@yahoo.com>
parents: 51917
diff changeset
2521 def __init__(self, stat: Optional[os.stat_result]) -> None:
32792
7ad95626f6a7 filestat: move __init__ to frompath constructor
Siddharth Agarwal <sid0@fb.com>
parents: 32769
diff changeset
2522 self.stat = stat
7ad95626f6a7 filestat: move __init__ to frompath constructor
Siddharth Agarwal <sid0@fb.com>
parents: 32769
diff changeset
2523
7ad95626f6a7 filestat: move __init__ to frompath constructor
Siddharth Agarwal <sid0@fb.com>
parents: 32769
diff changeset
2524 @classmethod
51924
1d95a87813ad typing: add type annotations to the `mercurial.util.filestat` class
Matt Harbison <matt_harbison@yahoo.com>
parents: 51917
diff changeset
2525 def frompath(cls: Type[_Tfilestat], path: bytes) -> _Tfilestat:
29200
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2526 try:
32792
7ad95626f6a7 filestat: move __init__ to frompath constructor
Siddharth Agarwal <sid0@fb.com>
parents: 32769
diff changeset
2527 stat = os.stat(path)
49314
2e726c934fcd py3: catch FileNotFoundError instead of checking errno == ENOENT
Manuel Jacob <me@manueljacob.de>
parents: 49286
diff changeset
2528 except FileNotFoundError:
32792
7ad95626f6a7 filestat: move __init__ to frompath constructor
Siddharth Agarwal <sid0@fb.com>
parents: 32769
diff changeset
2529 stat = None
7ad95626f6a7 filestat: move __init__ to frompath constructor
Siddharth Agarwal <sid0@fb.com>
parents: 32769
diff changeset
2530 return cls(stat)
29200
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2531
32835
1b25c648d5b7 fsmonitor: don't write out state if identity has changed (issue5581)
Siddharth Agarwal <sid0@fb.com>
parents: 32792
diff changeset
2532 @classmethod
51924
1d95a87813ad typing: add type annotations to the `mercurial.util.filestat` class
Matt Harbison <matt_harbison@yahoo.com>
parents: 51917
diff changeset
2533 def fromfp(cls: Type[_Tfilestat], fp: BinaryIO) -> _Tfilestat:
32835
1b25c648d5b7 fsmonitor: don't write out state if identity has changed (issue5581)
Siddharth Agarwal <sid0@fb.com>
parents: 32792
diff changeset
2534 stat = os.fstat(fp.fileno())
1b25c648d5b7 fsmonitor: don't write out state if identity has changed (issue5581)
Siddharth Agarwal <sid0@fb.com>
parents: 32792
diff changeset
2535 return cls(stat)
1b25c648d5b7 fsmonitor: don't write out state if identity has changed (issue5581)
Siddharth Agarwal <sid0@fb.com>
parents: 32792
diff changeset
2536
29200
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2537 __hash__ = object.__hash__
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2538
51924
1d95a87813ad typing: add type annotations to the `mercurial.util.filestat` class
Matt Harbison <matt_harbison@yahoo.com>
parents: 51917
diff changeset
2539 def __eq__(self, old) -> bool:
29200
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2540 try:
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2541 # if ambiguity between stat of new and old file is
30342
318a24b52eeb spelling: fixes of non-dictionary words
Mads Kiilerich <madski@unity3d.com>
parents: 30339
diff changeset
2542 # avoided, comparison of size, ctime and mtime is enough
29200
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2543 # to exactly detect change of a file regardless of platform
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2544 return (
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2545 self.stat.st_size == old.stat.st_size
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2546 and self.stat[stat.ST_CTIME] == old.stat[stat.ST_CTIME]
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2547 and self.stat[stat.ST_MTIME] == old.stat[stat.ST_MTIME]
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2548 )
29200
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2549 except AttributeError:
32769
b5524fd9a4e3 util: make filestat.__eq__ return True if both of self and old have None stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 32766
diff changeset
2550 pass
b5524fd9a4e3 util: make filestat.__eq__ return True if both of self and old have None stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 32766
diff changeset
2551 try:
b5524fd9a4e3 util: make filestat.__eq__ return True if both of self and old have None stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 32766
diff changeset
2552 return self.stat is None and old.stat is None
b5524fd9a4e3 util: make filestat.__eq__ return True if both of self and old have None stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 32766
diff changeset
2553 except AttributeError:
29200
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2554 return False
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2555
51924
1d95a87813ad typing: add type annotations to the `mercurial.util.filestat` class
Matt Harbison <matt_harbison@yahoo.com>
parents: 51917
diff changeset
2556 def isambig(self, old: _Tfilestat) -> bool:
29200
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2557 """Examine whether new (= self) stat is ambiguous against old one
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2558
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2559 "S[N]" below means stat of a file at N-th change:
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2560
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2561 - S[n-1].ctime < S[n].ctime: can detect change of a file
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2562 - S[n-1].ctime == S[n].ctime
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2563 - S[n-1].ctime < S[n].mtime: means natural advancing (*1)
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2564 - S[n-1].ctime == S[n].mtime: is ambiguous (*2)
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2565 - S[n-1].ctime > S[n].mtime: never occurs naturally (don't care)
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2566 - S[n-1].ctime > S[n].ctime: never occurs naturally (don't care)
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2567
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2568 Case (*2) above means that a file was changed twice or more at
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2569 same time in sec (= S[n-1].ctime), and comparison of timestamp
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2570 is ambiguous.
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2571
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2572 Base idea to avoid such ambiguity is "advance mtime 1 sec, if
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2573 timestamp is ambiguous".
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2574
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2575 But advancing mtime only in case (*2) doesn't work as
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2576 expected, because naturally advanced S[n].mtime in case (*1)
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2577 might be equal to manually advanced S[n-1 or earlier].mtime.
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2578
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2579 Therefore, all "S[n-1].ctime == S[n].ctime" cases should be
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2580 treated as ambiguous regardless of mtime, to avoid overlooking
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2581 by confliction between such mtime.
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2582
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2583 Advancing mtime "if isambig(oldstat)" ensures "S[n-1].mtime !=
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2584 S[n].mtime", even if size of a file isn't changed.
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2585 """
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2586 try:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2587 return self.stat[stat.ST_CTIME] == old.stat[stat.ST_CTIME]
29200
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2588 except AttributeError:
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2589 return False
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2590
51924
1d95a87813ad typing: add type annotations to the `mercurial.util.filestat` class
Matt Harbison <matt_harbison@yahoo.com>
parents: 51917
diff changeset
2591 def avoidambig(self, path: bytes, old: _Tfilestat) -> bool:
30253
b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 30181
diff changeset
2592 """Change file stat of specified path to avoid ambiguity
b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 30181
diff changeset
2593
b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 30181
diff changeset
2594 'old' should be previous filestat of 'path'.
b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 30181
diff changeset
2595
b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 30181
diff changeset
2596 This skips avoiding ambiguity, if a process doesn't have
32766
77f354ae1123 util: make filestat.avoidambig() return whether ambiguity is avoided or not
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 32674
diff changeset
2597 appropriate privileges for 'path'. This returns False in this
77f354ae1123 util: make filestat.avoidambig() return whether ambiguity is avoided or not
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 32674
diff changeset
2598 case.
77f354ae1123 util: make filestat.avoidambig() return whether ambiguity is avoided or not
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 32674
diff changeset
2599
77f354ae1123 util: make filestat.avoidambig() return whether ambiguity is avoided or not
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 32674
diff changeset
2600 Otherwise, this returns True, as "ambiguity is avoided".
30253
b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 30181
diff changeset
2601 """
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2602 advanced = (old.stat[stat.ST_MTIME] + 1) & 0x7FFFFFFF
30253
b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 30181
diff changeset
2603 try:
b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 30181
diff changeset
2604 os.utime(path, (advanced, advanced))
49316
d2adebe35635 py3: catch PermissionError instead of checking errno == EPERM
Manuel Jacob <me@manueljacob.de>
parents: 49314
diff changeset
2605 except PermissionError:
d2adebe35635 py3: catch PermissionError instead of checking errno == EPERM
Manuel Jacob <me@manueljacob.de>
parents: 49314
diff changeset
2606 # utime() on the file created by another user causes EPERM,
d2adebe35635 py3: catch PermissionError instead of checking errno == EPERM
Manuel Jacob <me@manueljacob.de>
parents: 49314
diff changeset
2607 # if a process doesn't have appropriate privileges
d2adebe35635 py3: catch PermissionError instead of checking errno == EPERM
Manuel Jacob <me@manueljacob.de>
parents: 49314
diff changeset
2608 return False
32766
77f354ae1123 util: make filestat.avoidambig() return whether ambiguity is avoided or not
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 32674
diff changeset
2609 return True
30253
b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 30181
diff changeset
2610
51924
1d95a87813ad typing: add type annotations to the `mercurial.util.filestat` class
Matt Harbison <matt_harbison@yahoo.com>
parents: 51917
diff changeset
2611 def __ne__(self, other) -> bool:
29298
82f6193ff2bc util: add __ne__ to filestat class for consistency
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29261
diff changeset
2612 return not self == other
82f6193ff2bc util: add __ne__ to filestat class for consistency
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29261
diff changeset
2613
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2614
49037
642e31cb55f0 py3: use class X: instead of class X(object):
Gregory Szorc <gregory.szorc@gmail.com>
parents: 49031
diff changeset
2615 class atomictempfile:
45957
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
2616 """writable file object that atomically updates a file
4827
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
2617
14008
da65edcac72a atomictempfile: rewrite docstring to clarify rename() vs. close().
Greg Ward <greg@gerg.ca>
parents: 14007
diff changeset
2618 All writes will go to a temporary copy of the original file. Call
15057
774da7121fc9 atomictempfile: make close() consistent with other file-like objects.
Greg Ward <greg@gerg.ca>
parents: 15050
diff changeset
2619 close() when you are done writing, and atomictempfile will rename
774da7121fc9 atomictempfile: make close() consistent with other file-like objects.
Greg Ward <greg@gerg.ca>
parents: 15050
diff changeset
2620 the temporary copy to the original name, making the changes
774da7121fc9 atomictempfile: make close() consistent with other file-like objects.
Greg Ward <greg@gerg.ca>
parents: 15050
diff changeset
2621 visible. If the object is destroyed without being closed, all your
774da7121fc9 atomictempfile: make close() consistent with other file-like objects.
Greg Ward <greg@gerg.ca>
parents: 15050
diff changeset
2622 writes are discarded.
29367
4e6e280e238f doc: describe detail about checkambig optional argument
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29342
diff changeset
2623
4e6e280e238f doc: describe detail about checkambig optional argument
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29342
diff changeset
2624 checkambig argument of constructor is used with filestat, and is
4e6e280e238f doc: describe detail about checkambig optional argument
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29342
diff changeset
2625 useful only if target file is guarded by any lock (e.g. repo.lock
4e6e280e238f doc: describe detail about checkambig optional argument
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29342
diff changeset
2626 or repo.wlock).
45957
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
2627 """
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2628
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2629 def __init__(self, name, mode=b'w+b', createmode=None, checkambig=False):
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2630 self.__name = name # permanent name
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2631 self._tempname = mktempcopy(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2632 name,
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2633 emptyok=(b'w' in mode),
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2634 createmode=createmode,
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2635 enforcewritable=(b'w' in mode),
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2636 )
41289
593f6359681d update: fix edge-case with update.atomic-file and read-only files
Boris Feld <boris.feld@octobus.net>
parents: 40898
diff changeset
2637
14007
d764463b433e atomictempfile: avoid infinite recursion in __del__().
Greg Ward <greg@gerg.ca>
parents: 14004
diff changeset
2638 self._fp = posixfile(self._tempname, mode)
29201
a109bf7e0dc2 util: make atomictempfile avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29200
diff changeset
2639 self._checkambig = checkambig
8327
aa25be1c2889 atomictempfile: delegate to posixfile instead of inheriting from it
Bryan O'Sullivan <bos@serpentine.com>
parents: 8312
diff changeset
2640
14007
d764463b433e atomictempfile: avoid infinite recursion in __del__().
Greg Ward <greg@gerg.ca>
parents: 14004
diff changeset
2641 # delegated methods
29393
50269a4dce61 atomictempfile: add read to the supported file operations
Martijn Pieters <mjpieters@fb.com>
parents: 29367
diff changeset
2642 self.read = self._fp.read
14007
d764463b433e atomictempfile: avoid infinite recursion in __del__().
Greg Ward <greg@gerg.ca>
parents: 14004
diff changeset
2643 self.write = self._fp.write
49721
c4f07a011714 util: implement `writelines()` on atomictempfile
Matt Harbison <matt_harbison@yahoo.com>
parents: 49316
diff changeset
2644 self.writelines = self._fp.writelines
17237
e73128535105 util: delegate seek and tell methods of atomictempfile
Bryan O'Sullivan <bryano@fb.com>
parents: 17203
diff changeset
2645 self.seek = self._fp.seek
e73128535105 util: delegate seek and tell methods of atomictempfile
Bryan O'Sullivan <bryano@fb.com>
parents: 17203
diff changeset
2646 self.tell = self._fp.tell
14007
d764463b433e atomictempfile: avoid infinite recursion in __del__().
Greg Ward <greg@gerg.ca>
parents: 14004
diff changeset
2647 self.fileno = self._fp.fileno
4827
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
2648
15057
774da7121fc9 atomictempfile: make close() consistent with other file-like objects.
Greg Ward <greg@gerg.ca>
parents: 15050
diff changeset
2649 def close(self):
8785
7a9151bc5b37 atomictempfile: fix exception in __del__ if mktempcopy fails (self._fp is None)
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 8778
diff changeset
2650 if not self._fp.closed:
8327
aa25be1c2889 atomictempfile: delegate to posixfile instead of inheriting from it
Bryan O'Sullivan <bos@serpentine.com>
parents: 8312
diff changeset
2651 self._fp.close()
29201
a109bf7e0dc2 util: make atomictempfile avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29200
diff changeset
2652 filename = localpath(self.__name)
32792
7ad95626f6a7 filestat: move __init__ to frompath constructor
Siddharth Agarwal <sid0@fb.com>
parents: 32769
diff changeset
2653 oldstat = self._checkambig and filestat.frompath(filename)
29201
a109bf7e0dc2 util: make atomictempfile avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29200
diff changeset
2654 if oldstat and oldstat.stat:
a109bf7e0dc2 util: make atomictempfile avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29200
diff changeset
2655 rename(self._tempname, filename)
32792
7ad95626f6a7 filestat: move __init__ to frompath constructor
Siddharth Agarwal <sid0@fb.com>
parents: 32769
diff changeset
2656 newstat = filestat.frompath(filename)
29201
a109bf7e0dc2 util: make atomictempfile avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29200
diff changeset
2657 if newstat.isambig(oldstat):
a109bf7e0dc2 util: make atomictempfile avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29200
diff changeset
2658 # stat of changed file is ambiguous to original one
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2659 advanced = (oldstat.stat[stat.ST_MTIME] + 1) & 0x7FFFFFFF
29201
a109bf7e0dc2 util: make atomictempfile avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29200
diff changeset
2660 os.utime(filename, (advanced, advanced))
a109bf7e0dc2 util: make atomictempfile avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29200
diff changeset
2661 else:
a109bf7e0dc2 util: make atomictempfile avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29200
diff changeset
2662 rename(self._tempname, filename)
4827
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
2663
15057
774da7121fc9 atomictempfile: make close() consistent with other file-like objects.
Greg Ward <greg@gerg.ca>
parents: 15050
diff changeset
2664 def discard(self):
8785
7a9151bc5b37 atomictempfile: fix exception in __del__ if mktempcopy fails (self._fp is None)
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 8778
diff changeset
2665 if not self._fp.closed:
4827
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
2666 try:
14007
d764463b433e atomictempfile: avoid infinite recursion in __del__().
Greg Ward <greg@gerg.ca>
parents: 14004
diff changeset
2667 os.unlink(self._tempname)
d764463b433e atomictempfile: avoid infinite recursion in __del__().
Greg Ward <greg@gerg.ca>
parents: 14004
diff changeset
2668 except OSError:
d764463b433e atomictempfile: avoid infinite recursion in __del__().
Greg Ward <greg@gerg.ca>
parents: 14004
diff changeset
2669 pass
8327
aa25be1c2889 atomictempfile: delegate to posixfile instead of inheriting from it
Bryan O'Sullivan <bos@serpentine.com>
parents: 8312
diff changeset
2670 self._fp.close()
4827
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
2671
13098
f7d6750dcd01 util: make atomicfiles closable
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 13072
diff changeset
2672 def __del__(self):
50951
d718eddf01d9 safehasattr: drop usage in favor of hasattr
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50749
diff changeset
2673 if hasattr(self, '_fp'): # constructor actually did something
15057
774da7121fc9 atomictempfile: make close() consistent with other file-like objects.
Greg Ward <greg@gerg.ca>
parents: 15050
diff changeset
2674 self.discard()
13098
f7d6750dcd01 util: make atomicfiles closable
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 13072
diff changeset
2675
29394
6d96658a22b0 atomictempfile: add context manager support
Martijn Pieters <mjpieters@fb.com>
parents: 29393
diff changeset
2676 def __enter__(self):
6d96658a22b0 atomictempfile: add context manager support
Martijn Pieters <mjpieters@fb.com>
parents: 29393
diff changeset
2677 return self
6d96658a22b0 atomictempfile: add context manager support
Martijn Pieters <mjpieters@fb.com>
parents: 29393
diff changeset
2678
6d96658a22b0 atomictempfile: add context manager support
Martijn Pieters <mjpieters@fb.com>
parents: 29393
diff changeset
2679 def __exit__(self, exctype, excvalue, traceback):
6d96658a22b0 atomictempfile: add context manager support
Martijn Pieters <mjpieters@fb.com>
parents: 29393
diff changeset
2680 if exctype is not None:
6d96658a22b0 atomictempfile: add context manager support
Martijn Pieters <mjpieters@fb.com>
parents: 29393
diff changeset
2681 self.discard()
6d96658a22b0 atomictempfile: add context manager support
Martijn Pieters <mjpieters@fb.com>
parents: 29393
diff changeset
2682 else:
6d96658a22b0 atomictempfile: add context manager support
Martijn Pieters <mjpieters@fb.com>
parents: 29393
diff changeset
2683 self.close()
6d96658a22b0 atomictempfile: add context manager support
Martijn Pieters <mjpieters@fb.com>
parents: 29393
diff changeset
2684
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2685
49222
4ff4e23de7df clone: use better names for temp files
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 49070
diff changeset
2686 def tryrmdir(f):
4ff4e23de7df clone: use better names for temp files
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 49070
diff changeset
2687 try:
4ff4e23de7df clone: use better names for temp files
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 49070
diff changeset
2688 removedirs(f)
4ff4e23de7df clone: use better names for temp files
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 49070
diff changeset
2689 except OSError as e:
4ff4e23de7df clone: use better names for temp files
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 49070
diff changeset
2690 if e.errno != errno.ENOENT and e.errno != errno.ENOTEMPTY:
4ff4e23de7df clone: use better names for temp files
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 49070
diff changeset
2691 raise
4ff4e23de7df clone: use better names for temp files
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 49070
diff changeset
2692
4ff4e23de7df clone: use better names for temp files
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 49070
diff changeset
2693
51304
f15cb5111a1e pytype: move some type comment to proper annotation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51302
diff changeset
2694 def unlinkpath(
f15cb5111a1e pytype: move some type comment to proper annotation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51302
diff changeset
2695 f: bytes, ignoremissing: bool = False, rmdir: bool = True
f15cb5111a1e pytype: move some type comment to proper annotation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51302
diff changeset
2696 ) -> None:
31545
52361c4f4dac util: unify unlinkpath
Ryan McElroy <rmcelroy@fb.com>
parents: 31539
diff changeset
2697 """unlink and remove the directory if it is empty"""
31547
bd9daafbf87c util: use tryunlink in unlinkpath
Ryan McElroy <rmcelroy@fb.com>
parents: 31546
diff changeset
2698 if ignoremissing:
bd9daafbf87c util: use tryunlink in unlinkpath
Ryan McElroy <rmcelroy@fb.com>
parents: 31546
diff changeset
2699 tryunlink(f)
bd9daafbf87c util: use tryunlink in unlinkpath
Ryan McElroy <rmcelroy@fb.com>
parents: 31546
diff changeset
2700 else:
31545
52361c4f4dac util: unify unlinkpath
Ryan McElroy <rmcelroy@fb.com>
parents: 31539
diff changeset
2701 unlink(f)
38497
da2a7d8354b2 unlinkpath: make empty directory removal optional (issue5901) (issue5826)
Kyle Lippincott <spectral@google.com>
parents: 38386
diff changeset
2702 if rmdir:
da2a7d8354b2 unlinkpath: make empty directory removal optional (issue5901) (issue5826)
Kyle Lippincott <spectral@google.com>
parents: 38386
diff changeset
2703 # try removing directories that might now be empty
da2a7d8354b2 unlinkpath: make empty directory removal optional (issue5901) (issue5826)
Kyle Lippincott <spectral@google.com>
parents: 38386
diff changeset
2704 try:
da2a7d8354b2 unlinkpath: make empty directory removal optional (issue5901) (issue5826)
Kyle Lippincott <spectral@google.com>
parents: 38386
diff changeset
2705 removedirs(os.path.dirname(f))
da2a7d8354b2 unlinkpath: make empty directory removal optional (issue5901) (issue5826)
Kyle Lippincott <spectral@google.com>
parents: 38386
diff changeset
2706 except OSError:
da2a7d8354b2 unlinkpath: make empty directory removal optional (issue5901) (issue5826)
Kyle Lippincott <spectral@google.com>
parents: 38386
diff changeset
2707 pass
31545
52361c4f4dac util: unify unlinkpath
Ryan McElroy <rmcelroy@fb.com>
parents: 31539
diff changeset
2708
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2709
51428
187c5769a629 vfs: have tryunlink tell what it did
Georges Racinet <georges.racinet@octobus.net>
parents: 51315
diff changeset
2710 def tryunlink(f: bytes) -> bool:
187c5769a629 vfs: have tryunlink tell what it did
Georges Racinet <georges.racinet@octobus.net>
parents: 51315
diff changeset
2711 """Attempt to remove a file, ignoring FileNotFoundError.
187c5769a629 vfs: have tryunlink tell what it did
Georges Racinet <georges.racinet@octobus.net>
parents: 51315
diff changeset
2712
187c5769a629 vfs: have tryunlink tell what it did
Georges Racinet <georges.racinet@octobus.net>
parents: 51315
diff changeset
2713 Returns False in case the file did not exit, True otherwise
187c5769a629 vfs: have tryunlink tell what it did
Georges Racinet <georges.racinet@octobus.net>
parents: 51315
diff changeset
2714 """
31546
6d5b77abf306 util: add tryunlink function
Ryan McElroy <rmcelroy@fb.com>
parents: 31545
diff changeset
2715 try:
6d5b77abf306 util: add tryunlink function
Ryan McElroy <rmcelroy@fb.com>
parents: 31545
diff changeset
2716 unlink(f)
51428
187c5769a629 vfs: have tryunlink tell what it did
Georges Racinet <georges.racinet@octobus.net>
parents: 51315
diff changeset
2717 return True
49314
2e726c934fcd py3: catch FileNotFoundError instead of checking errno == ENOENT
Manuel Jacob <me@manueljacob.de>
parents: 49286
diff changeset
2718 except FileNotFoundError:
51428
187c5769a629 vfs: have tryunlink tell what it did
Georges Racinet <georges.racinet@octobus.net>
parents: 51315
diff changeset
2719 return False
31546
6d5b77abf306 util: add tryunlink function
Ryan McElroy <rmcelroy@fb.com>
parents: 31545
diff changeset
2720
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2721
51304
f15cb5111a1e pytype: move some type comment to proper annotation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51302
diff changeset
2722 def makedirs(
f15cb5111a1e pytype: move some type comment to proper annotation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51302
diff changeset
2723 name: bytes, mode: Optional[int] = None, notindexed: bool = False
f15cb5111a1e pytype: move some type comment to proper annotation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51302
diff changeset
2724 ) -> None:
29017
07be86828e79 util: fix race in makedirs()
Adam Simpkins <simpkins@fb.com>
parents: 28883
diff changeset
2725 """recursive directory creation with parent mode inheritance
07be86828e79 util: fix race in makedirs()
Adam Simpkins <simpkins@fb.com>
parents: 28883
diff changeset
2726
07be86828e79 util: fix race in makedirs()
Adam Simpkins <simpkins@fb.com>
parents: 28883
diff changeset
2727 Newly created directories are marked as "not to be indexed by
07be86828e79 util: fix race in makedirs()
Adam Simpkins <simpkins@fb.com>
parents: 28883
diff changeset
2728 the content indexing service", if ``notindexed`` is specified
07be86828e79 util: fix race in makedirs()
Adam Simpkins <simpkins@fb.com>
parents: 28883
diff changeset
2729 for "write" mode access.
07be86828e79 util: fix race in makedirs()
Adam Simpkins <simpkins@fb.com>
parents: 28883
diff changeset
2730 """
6062
3c3b126e5619 Make files in .hg inherit the permissions from .hg/store
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 6007
diff changeset
2731 try:
18938
e22107cff6bf util: add notindexed optional parameter to makedirs function
Angel Ezquerra <angel.ezquerra@gmail.com>
parents: 18930
diff changeset
2732 makedir(name, notindexed)
25660
328739ea70c3 global: mass rewrite to use modern exception syntax
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25420
diff changeset
2733 except OSError as err:
6062
3c3b126e5619 Make files in .hg inherit the permissions from .hg/store
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 6007
diff changeset
2734 if err.errno == errno.EEXIST:
3c3b126e5619 Make files in .hg inherit the permissions from .hg/store
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 6007
diff changeset
2735 return
15058
81f33be0ea79 util: postpone and reorder parent calculation in makedirs
Adrian Buehlmann <adrian@cadifra.com>
parents: 15057
diff changeset
2736 if err.errno != errno.ENOENT or not name:
81f33be0ea79 util: postpone and reorder parent calculation in makedirs
Adrian Buehlmann <adrian@cadifra.com>
parents: 15057
diff changeset
2737 raise
47622
bb917eea1605 windows: introduce a `util.abspath` to replace os.path.abspath
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47452
diff changeset
2738 parent = os.path.dirname(abspath(name))
15058
81f33be0ea79 util: postpone and reorder parent calculation in makedirs
Adrian Buehlmann <adrian@cadifra.com>
parents: 15057
diff changeset
2739 if parent == name:
6062
3c3b126e5619 Make files in .hg inherit the permissions from .hg/store
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 6007
diff changeset
2740 raise
18938
e22107cff6bf util: add notindexed optional parameter to makedirs function
Angel Ezquerra <angel.ezquerra@gmail.com>
parents: 18930
diff changeset
2741 makedirs(parent, mode, notindexed)
29017
07be86828e79 util: fix race in makedirs()
Adam Simpkins <simpkins@fb.com>
parents: 28883
diff changeset
2742 try:
07be86828e79 util: fix race in makedirs()
Adam Simpkins <simpkins@fb.com>
parents: 28883
diff changeset
2743 makedir(name, notindexed)
07be86828e79 util: fix race in makedirs()
Adam Simpkins <simpkins@fb.com>
parents: 28883
diff changeset
2744 except OSError as err:
07be86828e79 util: fix race in makedirs()
Adam Simpkins <simpkins@fb.com>
parents: 28883
diff changeset
2745 # Catch EEXIST to handle races
07be86828e79 util: fix race in makedirs()
Adam Simpkins <simpkins@fb.com>
parents: 28883
diff changeset
2746 if err.errno == errno.EEXIST:
07be86828e79 util: fix race in makedirs()
Adam Simpkins <simpkins@fb.com>
parents: 28883
diff changeset
2747 return
07be86828e79 util: fix race in makedirs()
Adam Simpkins <simpkins@fb.com>
parents: 28883
diff changeset
2748 raise
18678
423eee0b0b14 util: make ensuredirs safer against races
Bryan O'Sullivan <bryano@fb.com>
parents: 18668
diff changeset
2749 if mode is not None:
423eee0b0b14 util: make ensuredirs safer against races
Bryan O'Sullivan <bryano@fb.com>
parents: 18668
diff changeset
2750 os.chmod(name, mode)
18668
4034b8d551b1 scmutil: create directories in a race-safe way during update
Bryan O'Sullivan <bryano@fb.com>
parents: 18614
diff changeset
2751
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2752
51304
f15cb5111a1e pytype: move some type comment to proper annotation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51302
diff changeset
2753 def readfile(path: bytes) -> bytes:
52392
fca7d38e040b util: stop using the `pycompat.open()` shim
Matt Harbison <matt_harbison@yahoo.com>
parents: 52116
diff changeset
2754 with open(path, 'rb') as fp:
14100
3e9e02a41dfb util: really drop size from readfile
Matt Mackall <mpm@selenic.com>
parents: 14099
diff changeset
2755 return fp.read()
14099
0824a0a3cefc util: add readfile() & writefile() helper functions
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 14077
diff changeset
2756
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2757
51304
f15cb5111a1e pytype: move some type comment to proper annotation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51302
diff changeset
2758 def writefile(path: bytes, text: bytes) -> None:
52392
fca7d38e040b util: stop using the `pycompat.open()` shim
Matt Harbison <matt_harbison@yahoo.com>
parents: 52116
diff changeset
2759 with open(path, 'wb') as fp:
14167
0e4753807c93 util & scmutil: adapt read/write helpers as request by mpm
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 14147
diff changeset
2760 fp.write(text)
0e4753807c93 util & scmutil: adapt read/write helpers as request by mpm
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 14147
diff changeset
2761
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2762
51304
f15cb5111a1e pytype: move some type comment to proper annotation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51302
diff changeset
2763 def appendfile(path: bytes, text: bytes) -> None:
52392
fca7d38e040b util: stop using the `pycompat.open()` shim
Matt Harbison <matt_harbison@yahoo.com>
parents: 52116
diff changeset
2764 with open(path, 'ab') as fp:
14099
0824a0a3cefc util: add readfile() & writefile() helper functions
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 14077
diff changeset
2765 fp.write(text)
0824a0a3cefc util: add readfile() & writefile() helper functions
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 14077
diff changeset
2766
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2767
49037
642e31cb55f0 py3: use class X: instead of class X(object):
Gregory Szorc <gregory.szorc@gmail.com>
parents: 49031
diff changeset
2768 class chunkbuffer:
1199
78ceaf83f28f Created a class in util called chunkbuffer that buffers reads from an
Eric Hopper <hopper@omnifarious.org>
parents: 1169
diff changeset
2769 """Allow arbitrary sized chunks of data to be efficiently read from an
78ceaf83f28f Created a class in util called chunkbuffer that buffers reads from an
Eric Hopper <hopper@omnifarious.org>
parents: 1169
diff changeset
2770 iterator over chunks of arbitrary size."""
1200
333de1d53846 Minor cleanups.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1199
diff changeset
2771
5446
fa836e050c50 chunkbuffer: removed unused method and arg
Matt Mackall <mpm@selenic.com>
parents: 5420
diff changeset
2772 def __init__(self, in_iter):
32176
5f53c267e362 util: remove doc of long gone 'targetsize' argument
Martin von Zweigbergk <martinvonz@google.com>
parents: 31952
diff changeset
2773 """in_iter is the iterator that's iterating over the input chunks."""
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2774
11670
1b3b843e1100 chunkbuffer: split big strings directly in chunkbuffer
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 11668
diff changeset
2775 def splitbig(chunks):
1b3b843e1100 chunkbuffer: split big strings directly in chunkbuffer
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 11668
diff changeset
2776 for chunk in chunks:
51762
ca7bde5dbafb black: format the codebase with 23.3.0
Rapha?l Gom?s <rgomes@octobus.net>
parents: 51761
diff changeset
2777 if len(chunk) > 2**20:
11670
1b3b843e1100 chunkbuffer: split big strings directly in chunkbuffer
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 11668
diff changeset
2778 pos = 0
1b3b843e1100 chunkbuffer: split big strings directly in chunkbuffer
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 11668
diff changeset
2779 while pos < len(chunk):
51762
ca7bde5dbafb black: format the codebase with 23.3.0
Rapha?l Gom?s <rgomes@octobus.net>
parents: 51761
diff changeset
2780 end = pos + 2**18
11670
1b3b843e1100 chunkbuffer: split big strings directly in chunkbuffer
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 11668
diff changeset
2781 yield chunk[pos:end]
1b3b843e1100 chunkbuffer: split big strings directly in chunkbuffer
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 11668
diff changeset
2782 pos = end
1b3b843e1100 chunkbuffer: split big strings directly in chunkbuffer
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 11668
diff changeset
2783 else:
1b3b843e1100 chunkbuffer: split big strings directly in chunkbuffer
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 11668
diff changeset
2784 yield chunk
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2785
11670
1b3b843e1100 chunkbuffer: split big strings directly in chunkbuffer
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 11668
diff changeset
2786 self.iter = splitbig(in_iter)
25113
0ca8410ea345 util: drop alias for collections.deque
Martin von Zweigbergk <martinvonz@google.com>
parents: 25112
diff changeset
2787 self._queue = collections.deque()
26480
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
2788 self._chunkoffset = 0
1200
333de1d53846 Minor cleanups.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1199
diff changeset
2789
21018
c848bfd02366 util: support None size in chunkbuffer.read()
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20835
diff changeset
2790 def read(self, l=None):
1200
333de1d53846 Minor cleanups.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1199
diff changeset
2791 """Read L bytes of data from the iterator of chunks of data.
21018
c848bfd02366 util: support None size in chunkbuffer.read()
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20835
diff changeset
2792 Returns less than L bytes if the iterator runs dry.
c848bfd02366 util: support None size in chunkbuffer.read()
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20835
diff changeset
2793
23139
e53f6b72a0e4 spelling: fixes from proofreading of spell checker issues
Mads Kiilerich <madski@unity3d.com>
parents: 23097
diff changeset
2794 If size parameter is omitted, read everything"""
26478
a3f7e5461dbd util.chunkbuffer: special case reading everything
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26450
diff changeset
2795 if l is None:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2796 return b''.join(self.iter)
26478
a3f7e5461dbd util.chunkbuffer: special case reading everything
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26450
diff changeset
2797
11758
a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint
Matt Mackall <mpm@selenic.com>
parents: 11469
diff changeset
2798 left = l
17962
4c29668ca316 util: make chunkbuffer non-quadratic on Windows
Matt Mackall <mpm@selenic.com>
parents: 17560
diff changeset
2799 buf = []
16873
37e081609828 util: simplify queue management in chunkbuffer
Bryan O'Sullivan <bryano@fb.com>
parents: 16834
diff changeset
2800 queue = self._queue
26478
a3f7e5461dbd util.chunkbuffer: special case reading everything
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26450
diff changeset
2801 while left > 0:
11758
a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint
Matt Mackall <mpm@selenic.com>
parents: 11469
diff changeset
2802 # refill the queue
a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint
Matt Mackall <mpm@selenic.com>
parents: 11469
diff changeset
2803 if not queue:
51762
ca7bde5dbafb black: format the codebase with 23.3.0
Rapha?l Gom?s <rgomes@octobus.net>
parents: 51761
diff changeset
2804 target = 2**18
11758
a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint
Matt Mackall <mpm@selenic.com>
parents: 11469
diff changeset
2805 for chunk in self.iter:
a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint
Matt Mackall <mpm@selenic.com>
parents: 11469
diff changeset
2806 queue.append(chunk)
a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint
Matt Mackall <mpm@selenic.com>
parents: 11469
diff changeset
2807 target -= len(chunk)
a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint
Matt Mackall <mpm@selenic.com>
parents: 11469
diff changeset
2808 if target <= 0:
a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint
Matt Mackall <mpm@selenic.com>
parents: 11469
diff changeset
2809 break
a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint
Matt Mackall <mpm@selenic.com>
parents: 11469
diff changeset
2810 if not queue:
1199
78ceaf83f28f Created a class in util called chunkbuffer that buffers reads from an
Eric Hopper <hopper@omnifarious.org>
parents: 1169
diff changeset
2811 break
11758
a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint
Matt Mackall <mpm@selenic.com>
parents: 11469
diff changeset
2812
26480
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
2813 # The easy way to do this would be to queue.popleft(), modify the
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
2814 # chunk (if necessary), then queue.appendleft(). However, for cases
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
2815 # where we read partial chunk content, this incurs 2 dequeue
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
2816 # mutations and creates a new str for the remaining chunk in the
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
2817 # queue. Our code below avoids this overhead.
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
2818
26479
46143f31290e util.chunkbuffer: refactor chunk handling logic
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26478
diff changeset
2819 chunk = queue[0]
46143f31290e util.chunkbuffer: refactor chunk handling logic
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26478
diff changeset
2820 chunkl = len(chunk)
26480
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
2821 offset = self._chunkoffset
26479
46143f31290e util.chunkbuffer: refactor chunk handling logic
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26478
diff changeset
2822
46143f31290e util.chunkbuffer: refactor chunk handling logic
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26478
diff changeset
2823 # Use full chunk.
26480
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
2824 if offset == 0 and left >= chunkl:
26479
46143f31290e util.chunkbuffer: refactor chunk handling logic
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26478
diff changeset
2825 left -= chunkl
46143f31290e util.chunkbuffer: refactor chunk handling logic
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26478
diff changeset
2826 queue.popleft()
46143f31290e util.chunkbuffer: refactor chunk handling logic
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26478
diff changeset
2827 buf.append(chunk)
26480
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
2828 # self._chunkoffset remains at 0.
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
2829 continue
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
2830
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
2831 chunkremaining = chunkl - offset
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
2832
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
2833 # Use all of unconsumed part of chunk.
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
2834 if left >= chunkremaining:
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
2835 left -= chunkremaining
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
2836 queue.popleft()
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
2837 # offset == 0 is enabled by block above, so this won't merely
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
2838 # copy via ``chunk[0:]``.
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
2839 buf.append(chunk[offset:])
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
2840 self._chunkoffset = 0
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
2841
26479
46143f31290e util.chunkbuffer: refactor chunk handling logic
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26478
diff changeset
2842 # Partial chunk needed.
46143f31290e util.chunkbuffer: refactor chunk handling logic
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26478
diff changeset
2843 else:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2844 buf.append(chunk[offset : offset + left])
26480
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
2845 self._chunkoffset += left
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
2846 left -= chunkremaining
11758
a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint
Matt Mackall <mpm@selenic.com>
parents: 11469
diff changeset
2847
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2848 return b''.join(buf)
11758
a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint
Matt Mackall <mpm@selenic.com>
parents: 11469
diff changeset
2849
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2850
30181
7356e6b1f5b8 util: increase filechunkiter size to 128k
Mads Kiilerich <madski@unity3d.com>
parents: 30087
diff changeset
2851 def filechunkiter(f, size=131072, limit=None):
2462
d610bcfd66a8 util: add limit to amount filechunkiter will read
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2448
diff changeset
2852 """Create a generator that produces the data in the file size
30181
7356e6b1f5b8 util: increase filechunkiter size to 128k
Mads Kiilerich <madski@unity3d.com>
parents: 30087
diff changeset
2853 (default 131072) bytes at a time, up to optional limit (default is
2462
d610bcfd66a8 util: add limit to amount filechunkiter will read
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2448
diff changeset
2854 to read all data). Chunks may be less than size bytes if the
d610bcfd66a8 util: add limit to amount filechunkiter will read
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2448
diff changeset
2855 chunk is the last chunk in the file, or the file is a socket or
d610bcfd66a8 util: add limit to amount filechunkiter will read
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2448
diff changeset
2856 some other type of file that sometimes reads less data than is
d610bcfd66a8 util: add limit to amount filechunkiter will read
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2448
diff changeset
2857 requested."""
d610bcfd66a8 util: add limit to amount filechunkiter will read
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2448
diff changeset
2858 assert size >= 0
d610bcfd66a8 util: add limit to amount filechunkiter will read
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2448
diff changeset
2859 assert limit is None or limit >= 0
d610bcfd66a8 util: add limit to amount filechunkiter will read
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2448
diff changeset
2860 while True:
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10264
diff changeset
2861 if limit is None:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10264
diff changeset
2862 nbytes = size
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10264
diff changeset
2863 else:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10264
diff changeset
2864 nbytes = min(limit, size)
2462
d610bcfd66a8 util: add limit to amount filechunkiter will read
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2448
diff changeset
2865 s = nbytes and f.read(nbytes)
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10264
diff changeset
2866 if not s:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10264
diff changeset
2867 break
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10264
diff changeset
2868 if limit:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10264
diff changeset
2869 limit -= len(s)
1199
78ceaf83f28f Created a class in util called chunkbuffer that buffers reads from an
Eric Hopper <hopper@omnifarious.org>
parents: 1169
diff changeset
2870 yield s
1320
5f277e73778f Fix up representation of dates in hgweb.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1312
diff changeset
2871
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2872
49037
642e31cb55f0 py3: use class X: instead of class X(object):
Gregory Szorc <gregory.szorc@gmail.com>
parents: 49031
diff changeset
2873 class cappedreader:
36395
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36299
diff changeset
2874 """A file object proxy that allows reading up to N bytes.
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36299
diff changeset
2875
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36299
diff changeset
2876 Given a source file object, instances of this type allow reading up to
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36299
diff changeset
2877 N bytes from that source file object. Attempts to read past the allowed
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36299
diff changeset
2878 limit are treated as EOF.
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36299
diff changeset
2879
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36299
diff changeset
2880 It is assumed that I/O is not performed on the original file object
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36299
diff changeset
2881 in addition to I/O that is performed by this instance. If there is,
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36299
diff changeset
2882 state tracking will get out of sync and unexpected results will ensue.
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36299
diff changeset
2883 """
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2884
36395
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36299
diff changeset
2885 def __init__(self, fh, limit):
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36299
diff changeset
2886 """Allow reading up to <limit> bytes from <fh>."""
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36299
diff changeset
2887 self._fh = fh
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36299
diff changeset
2888 self._left = limit
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36299
diff changeset
2889
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36299
diff changeset
2890 def read(self, n=-1):
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36299
diff changeset
2891 if not self._left:
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36299
diff changeset
2892 return b''
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36299
diff changeset
2893
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36299
diff changeset
2894 if n < 0:
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36299
diff changeset
2895 n = self._left
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36299
diff changeset
2896
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36299
diff changeset
2897 data = self._fh.read(min(n, self._left))
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36299
diff changeset
2898 self._left -= len(data)
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36299
diff changeset
2899 assert self._left >= 0
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36299
diff changeset
2900
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36299
diff changeset
2901 return data
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36299
diff changeset
2902
37055
8c3c47362934 wireproto: implement basic frame reading and processing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37047
diff changeset
2903 def readinto(self, b):
8c3c47362934 wireproto: implement basic frame reading and processing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37047
diff changeset
2904 res = self.read(len(b))
8c3c47362934 wireproto: implement basic frame reading and processing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37047
diff changeset
2905 if res is None:
8c3c47362934 wireproto: implement basic frame reading and processing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37047
diff changeset
2906 return None
8c3c47362934 wireproto: implement basic frame reading and processing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37047
diff changeset
2907
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2908 b[0 : len(res)] = res
37055
8c3c47362934 wireproto: implement basic frame reading and processing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37047
diff changeset
2909 return len(res)
8c3c47362934 wireproto: implement basic frame reading and processing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37047
diff changeset
2910
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2911
18735
716cad930691 util: generalize bytecount to unitcountfn
Bryan O'Sullivan <bryano@fb.com>
parents: 18678
diff changeset
2912 def unitcountfn(*unittable):
716cad930691 util: generalize bytecount to unitcountfn
Bryan O'Sullivan <bryano@fb.com>
parents: 18678
diff changeset
2913 '''return a function that renders a readable count of some quantity'''
716cad930691 util: generalize bytecount to unitcountfn
Bryan O'Sullivan <bryano@fb.com>
parents: 18678
diff changeset
2914
716cad930691 util: generalize bytecount to unitcountfn
Bryan O'Sullivan <bryano@fb.com>
parents: 18678
diff changeset
2915 def go(count):
716cad930691 util: generalize bytecount to unitcountfn
Bryan O'Sullivan <bryano@fb.com>
parents: 18678
diff changeset
2916 for multiplier, divisor, format in unittable:
31946
f3b80537a70d util: fix human-readable printing of negative byte counts
G?bor Stefanik <gabor.stefanik@nng.com>
parents: 31934
diff changeset
2917 if abs(count) >= divisor * multiplier:
18735
716cad930691 util: generalize bytecount to unitcountfn
Bryan O'Sullivan <bryano@fb.com>
parents: 18678
diff changeset
2918 return format % (count / float(divisor))
716cad930691 util: generalize bytecount to unitcountfn
Bryan O'Sullivan <bryano@fb.com>
parents: 18678
diff changeset
2919 return unittable[-1][2] % count
716cad930691 util: generalize bytecount to unitcountfn
Bryan O'Sullivan <bryano@fb.com>
parents: 18678
diff changeset
2920
716cad930691 util: generalize bytecount to unitcountfn
Bryan O'Sullivan <bryano@fb.com>
parents: 18678
diff changeset
2921 return go
716cad930691 util: generalize bytecount to unitcountfn
Bryan O'Sullivan <bryano@fb.com>
parents: 18678
diff changeset
2922
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2923
51304
f15cb5111a1e pytype: move some type comment to proper annotation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51302
diff changeset
2924 def processlinerange(fromline: int, toline: int) -> Tuple[int, int]:
31667
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31603
diff changeset
2925 """Check that linerange <fromline>:<toline> makes sense and return a
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31603
diff changeset
2926 0-based range.
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31603
diff changeset
2927
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31603
diff changeset
2928 >>> processlinerange(10, 20)
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31603
diff changeset
2929 (9, 20)
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31603
diff changeset
2930 >>> processlinerange(2, 1)
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31603
diff changeset
2931 Traceback (most recent call last):
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31603
diff changeset
2932 ...
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31603
diff changeset
2933 ParseError: line range must be positive
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31603
diff changeset
2934 >>> processlinerange(0, 5)
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31603
diff changeset
2935 Traceback (most recent call last):
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31603
diff changeset
2936 ...
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31603
diff changeset
2937 ParseError: fromline must be strictly positive
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31603
diff changeset
2938 """
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31603
diff changeset
2939 if toline - fromline < 0:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2940 raise error.ParseError(_(b"line range must be positive"))
31667
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31603
diff changeset
2941 if fromline < 1:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2942 raise error.ParseError(_(b"fromline must be strictly positive"))
31667
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31603
diff changeset
2943 return fromline - 1, toline
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31603
diff changeset
2944
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2945
18735
716cad930691 util: generalize bytecount to unitcountfn
Bryan O'Sullivan <bryano@fb.com>
parents: 18678
diff changeset
2946 bytecount = unitcountfn(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2947 (100, 1 << 30, _(b'%.0f GB')),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2948 (10, 1 << 30, _(b'%.1f GB')),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2949 (1, 1 << 30, _(b'%.2f GB')),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2950 (100, 1 << 20, _(b'%.0f MB')),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2951 (10, 1 << 20, _(b'%.1f MB')),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2952 (1, 1 << 20, _(b'%.2f MB')),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2953 (100, 1 << 10, _(b'%.0f KB')),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2954 (10, 1 << 10, _(b'%.1f KB')),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2955 (1, 1 << 10, _(b'%.2f KB')),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2956 (1, 1, _(b'%.0f bytes')),
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2957 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2958
16397
f0f7f3fab315 util: create bytecount array just once
Matt Mackall <mpm@selenic.com>
parents: 16383
diff changeset
2959
51780
ed28085827ec typing: explicitly type some `mercurial.util` eol code to avoid @overload
Matt Harbison <matt_harbison@yahoo.com>
parents: 51762
diff changeset
2960 class transformingwriter(typelib.BinaryIO_Proxy):
36845
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness
Yuya Nishihara <yuya@tcha.org>
parents: 36843
diff changeset
2961 """Writable file wrapper to transform data by function"""
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness
Yuya Nishihara <yuya@tcha.org>
parents: 36843
diff changeset
2962
51780
ed28085827ec typing: explicitly type some `mercurial.util` eol code to avoid @overload
Matt Harbison <matt_harbison@yahoo.com>
parents: 51762
diff changeset
2963 def __init__(self, fp: BinaryIO, encode: Callable[[bytes], bytes]) -> None:
36845
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness
Yuya Nishihara <yuya@tcha.org>
parents: 36843
diff changeset
2964 self._fp = fp
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness
Yuya Nishihara <yuya@tcha.org>
parents: 36843
diff changeset
2965 self._encode = encode
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness
Yuya Nishihara <yuya@tcha.org>
parents: 36843
diff changeset
2966
51780
ed28085827ec typing: explicitly type some `mercurial.util` eol code to avoid @overload
Matt Harbison <matt_harbison@yahoo.com>
parents: 51762
diff changeset
2967 def close(self) -> None:
36845
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness
Yuya Nishihara <yuya@tcha.org>
parents: 36843
diff changeset
2968 self._fp.close()
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness
Yuya Nishihara <yuya@tcha.org>
parents: 36843
diff changeset
2969
51780
ed28085827ec typing: explicitly type some `mercurial.util` eol code to avoid @overload
Matt Harbison <matt_harbison@yahoo.com>
parents: 51762
diff changeset
2970 def flush(self) -> None:
36845
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness
Yuya Nishihara <yuya@tcha.org>
parents: 36843
diff changeset
2971 self._fp.flush()
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness
Yuya Nishihara <yuya@tcha.org>
parents: 36843
diff changeset
2972
51780
ed28085827ec typing: explicitly type some `mercurial.util` eol code to avoid @overload
Matt Harbison <matt_harbison@yahoo.com>
parents: 51762
diff changeset
2973 def write(self, data: bytes) -> int:
36845
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness
Yuya Nishihara <yuya@tcha.org>
parents: 36843
diff changeset
2974 return self._fp.write(self._encode(data))
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness
Yuya Nishihara <yuya@tcha.org>
parents: 36843
diff changeset
2975
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2976
31779
fe9b33bcec6a util: extract pure tolf/tocrlf() functions from eol extension
Yuya Nishihara <yuya@tcha.org>
parents: 31725
diff changeset
2977 # Matches a single EOL which can either be a CRLF where repeated CR
fe9b33bcec6a util: extract pure tolf/tocrlf() functions from eol extension
Yuya Nishihara <yuya@tcha.org>
parents: 31725
diff changeset
2978 # are removed or a LF. We do not care about old Macintosh files, so a
fe9b33bcec6a util: extract pure tolf/tocrlf() functions from eol extension
Yuya Nishihara <yuya@tcha.org>
parents: 31725
diff changeset
2979 # stray CR is an error.
fe9b33bcec6a util: extract pure tolf/tocrlf() functions from eol extension
Yuya Nishihara <yuya@tcha.org>
parents: 31725
diff changeset
2980 _eolre = remod.compile(br'\r*\n')
fe9b33bcec6a util: extract pure tolf/tocrlf() functions from eol extension
Yuya Nishihara <yuya@tcha.org>
parents: 31725
diff changeset
2981
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2982
51304
f15cb5111a1e pytype: move some type comment to proper annotation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51302
diff changeset
2983 def tolf(s: bytes) -> bytes:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2984 return _eolre.sub(b'\n', s)
31779
fe9b33bcec6a util: extract pure tolf/tocrlf() functions from eol extension
Yuya Nishihara <yuya@tcha.org>
parents: 31725
diff changeset
2985
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2986
51304
f15cb5111a1e pytype: move some type comment to proper annotation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51302
diff changeset
2987 def tocrlf(s: bytes) -> bytes:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2988 return _eolre.sub(b'\r\n', s)
31779
fe9b33bcec6a util: extract pure tolf/tocrlf() functions from eol extension
Yuya Nishihara <yuya@tcha.org>
parents: 31725
diff changeset
2989
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2990
51780
ed28085827ec typing: explicitly type some `mercurial.util` eol code to avoid @overload
Matt Harbison <matt_harbison@yahoo.com>
parents: 51762
diff changeset
2991 def _crlfwriter(fp: typelib.BinaryIO_Proxy) -> typelib.BinaryIO_Proxy:
36845
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness
Yuya Nishihara <yuya@tcha.org>
parents: 36843
diff changeset
2992 return transformingwriter(fp, tocrlf)
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness
Yuya Nishihara <yuya@tcha.org>
parents: 36843
diff changeset
2993
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
2994
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2995 if pycompat.oslinesep == b'\r\n':
31780
6a5b69b0abec util: add helper to convert between LF and native EOL
Yuya Nishihara <yuya@tcha.org>
parents: 31779
diff changeset
2996 tonativeeol = tocrlf
6a5b69b0abec util: add helper to convert between LF and native EOL
Yuya Nishihara <yuya@tcha.org>
parents: 31779
diff changeset
2997 fromnativeeol = tolf
36845
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness
Yuya Nishihara <yuya@tcha.org>
parents: 36843
diff changeset
2998 nativeeolwriter = _crlfwriter
31780
6a5b69b0abec util: add helper to convert between LF and native EOL
Yuya Nishihara <yuya@tcha.org>
parents: 31779
diff changeset
2999 else:
6a5b69b0abec util: add helper to convert between LF and native EOL
Yuya Nishihara <yuya@tcha.org>
parents: 31779
diff changeset
3000 tonativeeol = pycompat.identity
6a5b69b0abec util: add helper to convert between LF and native EOL
Yuya Nishihara <yuya@tcha.org>
parents: 31779
diff changeset
3001 fromnativeeol = pycompat.identity
36845
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness
Yuya Nishihara <yuya@tcha.org>
parents: 36843
diff changeset
3002 nativeeolwriter = pycompat.identity
31780
6a5b69b0abec util: add helper to convert between LF and native EOL
Yuya Nishihara <yuya@tcha.org>
parents: 31779
diff changeset
3003
51780
ed28085827ec typing: explicitly type some `mercurial.util` eol code to avoid @overload
Matt Harbison <matt_harbison@yahoo.com>
parents: 51762
diff changeset
3004 if typing.TYPE_CHECKING:
ed28085827ec typing: explicitly type some `mercurial.util` eol code to avoid @overload
Matt Harbison <matt_harbison@yahoo.com>
parents: 51762
diff changeset
3005 # Replace the various overloads that come along with aliasing other methods
ed28085827ec typing: explicitly type some `mercurial.util` eol code to avoid @overload
Matt Harbison <matt_harbison@yahoo.com>
parents: 51762
diff changeset
3006 # with the narrow definition that we care about in the type checking phase
ed28085827ec typing: explicitly type some `mercurial.util` eol code to avoid @overload
Matt Harbison <matt_harbison@yahoo.com>
parents: 51762
diff changeset
3007 # only. This ensures that both Windows and POSIX see only the definition
ed28085827ec typing: explicitly type some `mercurial.util` eol code to avoid @overload
Matt Harbison <matt_harbison@yahoo.com>
parents: 51762
diff changeset
3008 # that is actually available.
ed28085827ec typing: explicitly type some `mercurial.util` eol code to avoid @overload
Matt Harbison <matt_harbison@yahoo.com>
parents: 51762
diff changeset
3009
ed28085827ec typing: explicitly type some `mercurial.util` eol code to avoid @overload
Matt Harbison <matt_harbison@yahoo.com>
parents: 51762
diff changeset
3010 def tonativeeol(s: bytes) -> bytes:
ed28085827ec typing: explicitly type some `mercurial.util` eol code to avoid @overload
Matt Harbison <matt_harbison@yahoo.com>
parents: 51762
diff changeset
3011 raise NotImplementedError
ed28085827ec typing: explicitly type some `mercurial.util` eol code to avoid @overload
Matt Harbison <matt_harbison@yahoo.com>
parents: 51762
diff changeset
3012
ed28085827ec typing: explicitly type some `mercurial.util` eol code to avoid @overload
Matt Harbison <matt_harbison@yahoo.com>
parents: 51762
diff changeset
3013 def fromnativeeol(s: bytes) -> bytes:
ed28085827ec typing: explicitly type some `mercurial.util` eol code to avoid @overload
Matt Harbison <matt_harbison@yahoo.com>
parents: 51762
diff changeset
3014 raise NotImplementedError
ed28085827ec typing: explicitly type some `mercurial.util` eol code to avoid @overload
Matt Harbison <matt_harbison@yahoo.com>
parents: 51762
diff changeset
3015
ed28085827ec typing: explicitly type some `mercurial.util` eol code to avoid @overload
Matt Harbison <matt_harbison@yahoo.com>
parents: 51762
diff changeset
3016 def nativeeolwriter(fp: typelib.BinaryIO_Proxy) -> typelib.BinaryIO_Proxy:
ed28085827ec typing: explicitly type some `mercurial.util` eol code to avoid @overload
Matt Harbison <matt_harbison@yahoo.com>
parents: 51762
diff changeset
3017 raise NotImplementedError
ed28085827ec typing: explicitly type some `mercurial.util` eol code to avoid @overload
Matt Harbison <matt_harbison@yahoo.com>
parents: 51762
diff changeset
3018
49031
2974cdda819b util: remove iterfile() variant for buggy EINTR handling
Gregory Szorc <gregory.szorc@gmail.com>
parents: 49004
diff changeset
3019
2974cdda819b util: remove iterfile() variant for buggy EINTR handling
Gregory Szorc <gregory.szorc@gmail.com>
parents: 49004
diff changeset
3020 # TODO delete since workaround variant for Python 2 no longer needed.
2974cdda819b util: remove iterfile() variant for buggy EINTR handling
Gregory Szorc <gregory.szorc@gmail.com>
parents: 49004
diff changeset
3021 def iterfile(fp):
2974cdda819b util: remove iterfile() variant for buggy EINTR handling
Gregory Szorc <gregory.szorc@gmail.com>
parents: 49004
diff changeset
3022 return fp
30405
10514a92860e util: add iterfile to workaround a fileobj.__iter__ issue with EINTR
Jun Wu <quark@fb.com>
parents: 30369
diff changeset
3023
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
3024
51304
f15cb5111a1e pytype: move some type comment to proper annotation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51302
diff changeset
3025 def iterlines(iterator: Iterable[bytes]) -> Iterator[bytes]:
7879
5c4026a289a4 templater: ability to display diffstat for log-like commands
Alexander Solovyov <piranha at piranha.org.ua>
parents: 7875
diff changeset
3026 for chunk in iterator:
5c4026a289a4 templater: ability to display diffstat for log-like commands
Alexander Solovyov <piranha at piranha.org.ua>
parents: 7875
diff changeset
3027 for line in chunk.splitlines():
5c4026a289a4 templater: ability to display diffstat for log-like commands
Alexander Solovyov <piranha at piranha.org.ua>
parents: 7875
diff changeset
3028 yield line
9610
d78fe60f6bda make path expanding more consistent
Alexander Solovyov <piranha@piranha.org.ua>
parents: 9569
diff changeset
3029
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
3030
51304
f15cb5111a1e pytype: move some type comment to proper annotation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51302
diff changeset
3031 def expandpath(path: bytes) -> bytes:
9610
d78fe60f6bda make path expanding more consistent
Alexander Solovyov <piranha@piranha.org.ua>
parents: 9569
diff changeset
3032 return os.path.expanduser(os.path.expandvars(path))
10239
8e4be44a676f Find right hg command for detached process
Patrick Mezard <pmezard@gmail.com>
parents: 10199
diff changeset
3033
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
3034
13392
777cef34a890 dispatch: support for $ escaping in shell-alias definition
Roman Sokolov <sokolov.r.v@gmail.com>
parents: 13375
diff changeset
3035 def interpolate(prefix, mapping, s, fn=None, escape_prefix=False):
11988
8380ed691df8 util: add an interpolate() function to for replacing multiple values
Steve Losh <steve@stevelosh.com>
parents: 11946
diff changeset
3036 """Return the result of interpolating items in the mapping into string s.
8380ed691df8 util: add an interpolate() function to for replacing multiple values
Steve Losh <steve@stevelosh.com>
parents: 11946
diff changeset
3037
8380ed691df8 util: add an interpolate() function to for replacing multiple values
Steve Losh <steve@stevelosh.com>
parents: 11946
diff changeset
3038 prefix is a single character string, or a two character string with
8380ed691df8 util: add an interpolate() function to for replacing multiple values
Steve Losh <steve@stevelosh.com>
parents: 11946
diff changeset
3039 a backslash as the first character if the prefix needs to be escaped in
8380ed691df8 util: add an interpolate() function to for replacing multiple values
Steve Losh <steve@stevelosh.com>
parents: 11946
diff changeset
3040 a regular expression.
8380ed691df8 util: add an interpolate() function to for replacing multiple values
Steve Losh <steve@stevelosh.com>
parents: 11946
diff changeset
3041
8380ed691df8 util: add an interpolate() function to for replacing multiple values
Steve Losh <steve@stevelosh.com>
parents: 11946
diff changeset
3042 fn is an optional function that will be applied to the replacement text
8380ed691df8 util: add an interpolate() function to for replacing multiple values
Steve Losh <steve@stevelosh.com>
parents: 11946
diff changeset
3043 just before replacement.
13392
777cef34a890 dispatch: support for $ escaping in shell-alias definition
Roman Sokolov <sokolov.r.v@gmail.com>
parents: 13375
diff changeset
3044
777cef34a890 dispatch: support for $ escaping in shell-alias definition
Roman Sokolov <sokolov.r.v@gmail.com>
parents: 13375
diff changeset
3045 escape_prefix is an optional flag that allows using doubled prefix for
777cef34a890 dispatch: support for $ escaping in shell-alias definition
Roman Sokolov <sokolov.r.v@gmail.com>
parents: 13375
diff changeset
3046 its escaping.
11988
8380ed691df8 util: add an interpolate() function to for replacing multiple values
Steve Losh <steve@stevelosh.com>
parents: 11946
diff changeset
3047 """
8380ed691df8 util: add an interpolate() function to for replacing multiple values
Steve Losh <steve@stevelosh.com>
parents: 11946
diff changeset
3048 fn = fn or (lambda s: s)
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3049 patterns = b'|'.join(mapping.keys())
13392
777cef34a890 dispatch: support for $ escaping in shell-alias definition
Roman Sokolov <sokolov.r.v@gmail.com>
parents: 13375
diff changeset
3050 if escape_prefix:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3051 patterns += b'|' + prefix
13392
777cef34a890 dispatch: support for $ escaping in shell-alias definition
Roman Sokolov <sokolov.r.v@gmail.com>
parents: 13375
diff changeset
3052 if len(prefix) > 1:
777cef34a890 dispatch: support for $ escaping in shell-alias definition
Roman Sokolov <sokolov.r.v@gmail.com>
parents: 13375
diff changeset
3053 prefix_char = prefix[1:]
777cef34a890 dispatch: support for $ escaping in shell-alias definition
Roman Sokolov <sokolov.r.v@gmail.com>
parents: 13375
diff changeset
3054 else:
777cef34a890 dispatch: support for $ escaping in shell-alias definition
Roman Sokolov <sokolov.r.v@gmail.com>
parents: 13375
diff changeset
3055 prefix_char = prefix
777cef34a890 dispatch: support for $ escaping in shell-alias definition
Roman Sokolov <sokolov.r.v@gmail.com>
parents: 13375
diff changeset
3056 mapping[prefix_char] = prefix_char
35166
25c543944bc0 py3: add b'' to regular expressions which are raw strings
Pulkit Goyal <7895pulkit@gmail.com>
parents: 35053
diff changeset
3057 r = remod.compile(br'%s(%s)' % (prefix, patterns))
11988
8380ed691df8 util: add an interpolate() function to for replacing multiple values
Steve Losh <steve@stevelosh.com>
parents: 11946
diff changeset
3058 return r.sub(lambda x: fn(mapping[x.group()[1:]]), s)
8380ed691df8 util: add an interpolate() function to for replacing multiple values
Steve Losh <steve@stevelosh.com>
parents: 11946
diff changeset
3059
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
3060
18736
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
3061 timecount = unitcountfn(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3062 (1, 1e3, _(b'%.0f s')),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3063 (100, 1, _(b'%.1f s')),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3064 (10, 1, _(b'%.2f s')),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3065 (1, 1, _(b'%.3f s')),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3066 (100, 0.001, _(b'%.1f ms')),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3067 (10, 0.001, _(b'%.2f ms')),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3068 (1, 0.001, _(b'%.3f ms')),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3069 (100, 0.000001, _(b'%.1f us')),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3070 (10, 0.000001, _(b'%.2f us')),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3071 (1, 0.000001, _(b'%.3f us')),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3072 (100, 0.000000001, _(b'%.1f ns')),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3073 (10, 0.000000001, _(b'%.2f ns')),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3074 (1, 0.000000001, _(b'%.3f ns')),
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
3075 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
3076
18736
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
3077
38837
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
3078 @attr.s
49037
642e31cb55f0 py3: use class X: instead of class X(object):
Gregory Szorc <gregory.szorc@gmail.com>
parents: 49031
diff changeset
3079 class timedcmstats:
38837
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
3080 """Stats information produced by the timedcm context manager on entering."""
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
3081
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
3082 # the starting value of the timer as a float (meaning and resulution is
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
3083 # platform dependent, see util.timer)
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
3084 start = attr.ib(default=attr.Factory(lambda: timer()))
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
3085 # the number of seconds as a floating point value; starts at 0, updated when
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
3086 # the context is exited.
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
3087 elapsed = attr.ib(default=0)
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
3088 # the number of nested timedcm context managers.
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
3089 level = attr.ib(default=1)
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
3090
38852
9d49bb117dde util: make new timedcmstats class Python 3 compatible
Martijn Pieters <mj@zopatista.com>
parents: 38837
diff changeset
3091 def __bytes__(self):
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3092 return timecount(self.elapsed) if self.elapsed else b'<unknown>'
38837
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
3093
38852
9d49bb117dde util: make new timedcmstats class Python 3 compatible
Martijn Pieters <mj@zopatista.com>
parents: 38837
diff changeset
3094 __str__ = encoding.strmethod(__bytes__)
9d49bb117dde util: make new timedcmstats class Python 3 compatible
Martijn Pieters <mj@zopatista.com>
parents: 38837
diff changeset
3095
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
3096
38837
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
3097 @contextlib.contextmanager
39287
e00123f63410 util: make timedcm require the label (API)
Augie Fackler <augie@google.com>
parents: 39286
diff changeset
3098 def timedcm(whencefmt, *whenceargs):
38837
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
3099 """A context manager that produces timing information for a given context.
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
3100
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
3101 On entering a timedcmstats instance is produced.
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
3102
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
3103 This context manager is reentrant.
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
3104
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
3105 """
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
3106 # track nested context managers
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
3107 timedcm._nested += 1
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
3108 timing_stats = timedcmstats(level=timedcm._nested)
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
3109 try:
39285
497effb0a04a util: make timedcm context manager also emit trace events
Augie Fackler <augie@google.com>
parents: 39237
diff changeset
3110 with tracing.log(whencefmt, *whenceargs):
497effb0a04a util: make timedcm context manager also emit trace events
Augie Fackler <augie@google.com>
parents: 39237
diff changeset
3111 yield timing_stats
38837
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
3112 finally:
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
3113 timing_stats.elapsed = timer() - timing_stats.start
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
3114 timedcm._nested -= 1
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
3115
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
3116
38837
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
3117 timedcm._nested = 0
18736
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
3118
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
3119
18736
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
3120 def timed(func):
45957
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
3121 """Report the execution time of a function call to stderr.
18736
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
3122
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
3123 During development, use as a decorator when you need to measure
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
3124 the cost of a function, e.g. as follows:
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
3125
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
3126 @util.timed
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
3127 def foo(a, b, c):
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
3128 pass
45957
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
3129 """
18736
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
3130
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
3131 def wrapper(*args, **kwargs):
39286
331ab85e910b cleanup: make all uses of timedcm specify what they're timing
Augie Fackler <augie@google.com>
parents: 39285
diff changeset
3132 with timedcm(pycompat.bytestr(func.__name__)) as time_stats:
38837
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
3133 result = func(*args, **kwargs)
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
3134 stderr = procutil.stderr
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
3135 stderr.write(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3136 b'%s%s: %s\n'
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
3137 % (
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3138 b' ' * time_stats.level * 2,
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
3139 pycompat.bytestr(func.__name__),
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
3140 time_stats,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
3141 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
3142 )
38837
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
3143 return result
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
3144
18736
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
3145 return wrapper
19194
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint
Bryan O'Sullivan <bryano@fb.com>
parents: 18938
diff changeset
3146
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
3147
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
3148 _sizeunits = (
51762
ca7bde5dbafb black: format the codebase with 23.3.0
Rapha?l Gom?s <rgomes@octobus.net>
parents: 51761
diff changeset
3149 (b'm', 2**20),
ca7bde5dbafb black: format the codebase with 23.3.0
Rapha?l Gom?s <rgomes@octobus.net>
parents: 51761
diff changeset
3150 (b'k', 2**10),
ca7bde5dbafb black: format the codebase with 23.3.0
Rapha?l Gom?s <rgomes@octobus.net>
parents: 51761
diff changeset
3151 (b'g', 2**30),
ca7bde5dbafb black: format the codebase with 23.3.0
Rapha?l Gom?s <rgomes@octobus.net>
parents: 51761
diff changeset
3152 (b'kb', 2**10),
ca7bde5dbafb black: format the codebase with 23.3.0
Rapha?l Gom?s <rgomes@octobus.net>
parents: 51761
diff changeset
3153 (b'mb', 2**20),
ca7bde5dbafb black: format the codebase with 23.3.0
Rapha?l Gom?s <rgomes@octobus.net>
parents: 51761
diff changeset
3154 (b'gb', 2**30),
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3155 (b'b', 1),
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
3156 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
3157
19194
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint
Bryan O'Sullivan <bryano@fb.com>
parents: 18938
diff changeset
3158
51304
f15cb5111a1e pytype: move some type comment to proper annotation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51302
diff changeset
3159 def sizetoint(s: bytes) -> int:
45957
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
3160 """Convert a space specifier to a byte count.
19194
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint
Bryan O'Sullivan <bryano@fb.com>
parents: 18938
diff changeset
3161
34146
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
3162 >>> sizetoint(b'30')
19194
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint
Bryan O'Sullivan <bryano@fb.com>
parents: 18938
diff changeset
3163 30
34146
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
3164 >>> sizetoint(b'2.2kb')
19194
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint
Bryan O'Sullivan <bryano@fb.com>
parents: 18938
diff changeset
3165 2252
34146
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
3166 >>> sizetoint(b'6M')
19194
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint
Bryan O'Sullivan <bryano@fb.com>
parents: 18938
diff changeset
3167 6291456
45957
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
3168 """
19194
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint
Bryan O'Sullivan <bryano@fb.com>
parents: 18938
diff changeset
3169 t = s.strip().lower()
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint
Bryan O'Sullivan <bryano@fb.com>
parents: 18938
diff changeset
3170 try:
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint
Bryan O'Sullivan <bryano@fb.com>
parents: 18938
diff changeset
3171 for k, u in _sizeunits:
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint
Bryan O'Sullivan <bryano@fb.com>
parents: 18938
diff changeset
3172 if t.endswith(k):
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
3173 return int(float(t[: -len(k)]) * u)
19194
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint
Bryan O'Sullivan <bryano@fb.com>
parents: 18938
diff changeset
3174 return int(t)
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint
Bryan O'Sullivan <bryano@fb.com>
parents: 18938
diff changeset
3175 except ValueError:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3176 raise error.ParseError(_(b"couldn't parse size: %s") % s)
19211
3bfd7f1e7485 summary: augment output with info from extensions
Bryan O'Sullivan <bryano@fb.com>
parents: 19194
diff changeset
3177
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
3178
49037
642e31cb55f0 py3: use class X: instead of class X(object):
Gregory Szorc <gregory.szorc@gmail.com>
parents: 49031
diff changeset
3179 class hooks:
45957
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
3180 """A collection of hook functions that can be used to extend a
26098
ce26928cbe41 spelling: behaviour -> behavior
timeless@mozdev.org
parents: 25672
diff changeset
3181 function's behavior. Hooks are called in lexicographic order,
45957
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
3182 based on the names of their sources."""
19211
3bfd7f1e7485 summary: augment output with info from extensions
Bryan O'Sullivan <bryano@fb.com>
parents: 19194
diff changeset
3183
3bfd7f1e7485 summary: augment output with info from extensions
Bryan O'Sullivan <bryano@fb.com>
parents: 19194
diff changeset
3184 def __init__(self):
3bfd7f1e7485 summary: augment output with info from extensions
Bryan O'Sullivan <bryano@fb.com>
parents: 19194
diff changeset
3185 self._hooks = []
3bfd7f1e7485 summary: augment output with info from extensions
Bryan O'Sullivan <bryano@fb.com>
parents: 19194
diff changeset
3186
3bfd7f1e7485 summary: augment output with info from extensions
Bryan O'Sullivan <bryano@fb.com>
parents: 19194
diff changeset
3187 def add(self, source, hook):
3bfd7f1e7485 summary: augment output with info from extensions
Bryan O'Sullivan <bryano@fb.com>
parents: 19194
diff changeset
3188 self._hooks.append((source, hook))
3bfd7f1e7485 summary: augment output with info from extensions
Bryan O'Sullivan <bryano@fb.com>
parents: 19194
diff changeset
3189
3bfd7f1e7485 summary: augment output with info from extensions
Bryan O'Sullivan <bryano@fb.com>
parents: 19194
diff changeset
3190 def __call__(self, *args):
3bfd7f1e7485 summary: augment output with info from extensions
Bryan O'Sullivan <bryano@fb.com>
parents: 19194
diff changeset
3191 self._hooks.sort(key=lambda x: x[0])
21046
cc13addbd62b util: enable "hooks" to return list of the values returned from each hooks
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 21018
diff changeset
3192 results = []
19211
3bfd7f1e7485 summary: augment output with info from extensions
Bryan O'Sullivan <bryano@fb.com>
parents: 19194
diff changeset
3193 for source, hook in self._hooks:
21046
cc13addbd62b util: enable "hooks" to return list of the values returned from each hooks
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 21018
diff changeset
3194 results.append(hook(*args))
cc13addbd62b util: enable "hooks" to return list of the values returned from each hooks
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 21018
diff changeset
3195 return results
20244
47d0843647d1 util: introduce util.debugstacktrace for showing a stack trace without crashing
Mads Kiilerich <madski@unity3d.com>
parents: 20202
diff changeset
3196
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
3197
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3198 def getstackframes(skip=0, line=b' %-*s in %s\n', fileline=b'%s:%d', depth=0):
45957
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
3199 """Yields lines for a nicely formatted stacktrace.
31324
78ac7061f840 util: add debugstacktrace depth limit
Mads Kiilerich <madski@unity3d.com>
parents: 31323
diff changeset
3200 Skips the 'skip' last entries, then return the last 'depth' entries.
28497
906fece80cfa util: refactor getstackframes
timeless <timeless@mozdev.org>
parents: 28496
diff changeset
3201 Each file+linenumber is formatted according to fileline.
906fece80cfa util: refactor getstackframes
timeless <timeless@mozdev.org>
parents: 28496
diff changeset
3202 Each line is formatted according to line.
906fece80cfa util: refactor getstackframes
timeless <timeless@mozdev.org>
parents: 28496
diff changeset
3203 If line is None, it yields:
906fece80cfa util: refactor getstackframes
timeless <timeless@mozdev.org>
parents: 28496
diff changeset
3204 length of longest filepath+line number,
906fece80cfa util: refactor getstackframes
timeless <timeless@mozdev.org>
parents: 28496
diff changeset
3205 filepath+linenumber,
906fece80cfa util: refactor getstackframes
timeless <timeless@mozdev.org>
parents: 28496
diff changeset
3206 function
906fece80cfa util: refactor getstackframes
timeless <timeless@mozdev.org>
parents: 28496
diff changeset
3207
906fece80cfa util: refactor getstackframes
timeless <timeless@mozdev.org>
parents: 28496
diff changeset
3208 Not be used in production code but very convenient while developing.
45957
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
3209 """
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
3210 entries = [
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
3211 (fileline % (pycompat.sysbytes(fn), ln), pycompat.sysbytes(func))
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
3212 for fn, ln, func, _text in traceback.extract_stack()[: -skip - 1]
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
3213 ][-depth:]
28497
906fece80cfa util: refactor getstackframes
timeless <timeless@mozdev.org>
parents: 28496
diff changeset
3214 if entries:
906fece80cfa util: refactor getstackframes
timeless <timeless@mozdev.org>
parents: 28496
diff changeset
3215 fnmax = max(len(entry[0]) for entry in entries)
906fece80cfa util: refactor getstackframes
timeless <timeless@mozdev.org>
parents: 28496
diff changeset
3216 for fnln, func in entries:
906fece80cfa util: refactor getstackframes
timeless <timeless@mozdev.org>
parents: 28496
diff changeset
3217 if line is None:
906fece80cfa util: refactor getstackframes
timeless <timeless@mozdev.org>
parents: 28496
diff changeset
3218 yield (fnmax, fnln, func)
906fece80cfa util: refactor getstackframes
timeless <timeless@mozdev.org>
parents: 28496
diff changeset
3219 else:
906fece80cfa util: refactor getstackframes
timeless <timeless@mozdev.org>
parents: 28496
diff changeset
3220 yield line % (fnmax, fnln, func)
906fece80cfa util: refactor getstackframes
timeless <timeless@mozdev.org>
parents: 28496
diff changeset
3221
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
3222
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
3223 def debugstacktrace(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3224 msg=b'stacktrace',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3225 skip=0,
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3226 f=procutil.stderr,
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3227 otherf=procutil.stdout,
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3228 depth=0,
43774
02ededbef627 util: add an optional `prefix` argument to debugstacktrace
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43767
diff changeset
3229 prefix=b'',
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
3230 ):
45957
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
3231 """Writes a message to f (stderr) with a nicely formatted stacktrace.
31324
78ac7061f840 util: add debugstacktrace depth limit
Mads Kiilerich <madski@unity3d.com>
parents: 31323
diff changeset
3232 Skips the 'skip' entries closest to the call, then show 'depth' entries.
78ac7061f840 util: add debugstacktrace depth limit
Mads Kiilerich <madski@unity3d.com>
parents: 31323
diff changeset
3233 By default it will flush stdout first.
28496
b592564a803c util: reword debugstacktrace comment
timeless <timeless@mozdev.org>
parents: 28027
diff changeset
3234 It can be used everywhere and intentionally does not require an ui object.
20244
47d0843647d1 util: introduce util.debugstacktrace for showing a stack trace without crashing
Mads Kiilerich <madski@unity3d.com>
parents: 20202
diff changeset
3235 Not be used in production code but very convenient while developing.
45957
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
3236 """
20542
be27652675ce util: debugstacktrace, flush before and after writing
Mads Kiilerich <madski@unity3d.com>
parents: 20353
diff changeset
3237 if otherf:
be27652675ce util: debugstacktrace, flush before and after writing
Mads Kiilerich <madski@unity3d.com>
parents: 20353
diff changeset
3238 otherf.flush()
43774
02ededbef627 util: add an optional `prefix` argument to debugstacktrace
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43767
diff changeset
3239 f.write(b'%s%s at:\n' % (prefix, msg.rstrip()))
31324
78ac7061f840 util: add debugstacktrace depth limit
Mads Kiilerich <madski@unity3d.com>
parents: 31323
diff changeset
3240 for line in getstackframes(skip + 1, depth=depth):
43774
02ededbef627 util: add an optional `prefix` argument to debugstacktrace
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43767
diff changeset
3241 f.write(prefix + line)
20542
be27652675ce util: debugstacktrace, flush before and after writing
Mads Kiilerich <madski@unity3d.com>
parents: 20353
diff changeset
3242 f.flush()
20244
47d0843647d1 util: introduce util.debugstacktrace for showing a stack trace without crashing
Mads Kiilerich <madski@unity3d.com>
parents: 20202
diff changeset
3243
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
3244
20244
47d0843647d1 util: introduce util.debugstacktrace for showing a stack trace without crashing
Mads Kiilerich <madski@unity3d.com>
parents: 20202
diff changeset
3245 # convenient shortcut
47d0843647d1 util: introduce util.debugstacktrace for showing a stack trace without crashing
Mads Kiilerich <madski@unity3d.com>
parents: 20202
diff changeset
3246 dst = debugstacktrace
34554
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34468
diff changeset
3247
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
3248
34554
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34468
diff changeset
3249 def safename(f, tag, ctx, others=None):
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34468
diff changeset
3250 """
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34468
diff changeset
3251 Generate a name that it is safe to rename f to in the given context.
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34468
diff changeset
3252
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34468
diff changeset
3253 f: filename to rename
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34468
diff changeset
3254 tag: a string tag that will be included in the new name
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34468
diff changeset
3255 ctx: a context, in which the new name must not exist
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34468
diff changeset
3256 others: a set of other filenames that the new name must not be in
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34468
diff changeset
3257
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34468
diff changeset
3258 Returns a file name of the form oldname~tag[~number] which does not exist
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34468
diff changeset
3259 in the provided context and is not in the set of other names.
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34468
diff changeset
3260 """
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34468
diff changeset
3261 if others is None:
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34468
diff changeset
3262 others = set()
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34468
diff changeset
3263
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3264 fn = b'%s~%s' % (f, tag)
34554
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34468
diff changeset
3265 if fn not in ctx and fn not in others:
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34468
diff changeset
3266 return fn
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34468
diff changeset
3267 for n in itertools.count(1):
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3268 fn = b'%s~%s~%s' % (f, tag, n)
34554
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34468
diff changeset
3269 if fn not in ctx and fn not in others:
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34468
diff changeset
3270 return fn
35754
fb0be099063f util: move 'readexactly' in the util module
Boris Feld <boris.feld@octobus.net>
parents: 35582
diff changeset
3271
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
3272
35754
fb0be099063f util: move 'readexactly' in the util module
Boris Feld <boris.feld@octobus.net>
parents: 35582
diff changeset
3273 def readexactly(stream, n):
fb0be099063f util: move 'readexactly' in the util module
Boris Feld <boris.feld@octobus.net>
parents: 35582
diff changeset
3274 '''read n bytes from stream.read and abort if less was available'''
fb0be099063f util: move 'readexactly' in the util module
Boris Feld <boris.feld@octobus.net>
parents: 35582
diff changeset
3275 s = stream.read(n)
fb0be099063f util: move 'readexactly' in the util module
Boris Feld <boris.feld@octobus.net>
parents: 35582
diff changeset
3276 if len(s) < n:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
3277 raise error.Abort(
43117
8ff1ecfadcd1 cleanup: join string literals that are already on one line
Martin von Zweigbergk <martinvonz@google.com>
parents: 43106
diff changeset
3278 _(b"stream ended unexpectedly (got %d bytes, expected %d)")
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
3279 % (len(s), n)
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
3280 )
35754
fb0be099063f util: move 'readexactly' in the util module
Boris Feld <boris.feld@octobus.net>
parents: 35582
diff changeset
3281 return s
35755
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3282
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
3283
35755
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3284 def uvarintencode(value):
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3285 """Encode an unsigned integer value to a varint.
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3286
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3287 A varint is a variable length integer of 1 or more bytes. Each byte
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3288 except the last has the most significant bit set. The lower 7 bits of
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3289 each byte store the 2's complement representation, least significant group
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3290 first.
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3291
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3292 >>> uvarintencode(0)
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3293 '\\x00'
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3294 >>> uvarintencode(1)
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3295 '\\x01'
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3296 >>> uvarintencode(127)
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3297 '\\x7f'
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3298 >>> uvarintencode(1337)
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3299 '\\xb9\\n'
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3300 >>> uvarintencode(65536)
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3301 '\\x80\\x80\\x04'
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3302 >>> uvarintencode(-1)
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3303 Traceback (most recent call last):
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3304 ...
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3305 ProgrammingError: negative value for uvarint: -1
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3306 """
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3307 if value < 0:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3308 raise error.ProgrammingError(b'negative value for uvarint: %d' % value)
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
3309 bits = value & 0x7F
35755
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3310 value >>= 7
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3311 bytes = []
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3312 while value:
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3313 bytes.append(pycompat.bytechr(0x80 | bits))
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
3314 bits = value & 0x7F
35755
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3315 value >>= 7
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3316 bytes.append(pycompat.bytechr(bits))
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3317
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3318 return b''.join(bytes)
35755
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3319
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
3320
35755
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3321 def uvarintdecodestream(fh):
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3322 """Decode an unsigned variable length integer from a stream.
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3323
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3324 The passed argument is anything that has a ``.read(N)`` method.
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3325
49908
54114bba7c7e tests: drop some obsolete py2 handling in util.py doctest
Matt Harbison <matt_harbison@yahoo.com>
parents: 49721
diff changeset
3326 >>> from io import BytesIO
35755
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3327 >>> uvarintdecodestream(BytesIO(b'\\x00'))
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3328 0
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3329 >>> uvarintdecodestream(BytesIO(b'\\x01'))
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3330 1
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3331 >>> uvarintdecodestream(BytesIO(b'\\x7f'))
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3332 127
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3333 >>> uvarintdecodestream(BytesIO(b'\\xb9\\n'))
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3334 1337
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3335 >>> uvarintdecodestream(BytesIO(b'\\x80\\x80\\x04'))
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3336 65536
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3337 >>> uvarintdecodestream(BytesIO(b'\\x80'))
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3338 Traceback (most recent call last):
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3339 ...
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3340 Abort: stream ended unexpectedly (got 0 bytes, expected 1)
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3341 """
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3342 result = 0
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3343 shift = 0
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3344 while True:
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3345 byte = ord(readexactly(fh, 1))
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42782
diff changeset
3346 result |= (byte & 0x7F) << shift
35755
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3347 if not (byte & 0x80):
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3348 return result
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3349 shift += 7
44826
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44633
diff changeset
3350
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44633
diff changeset
3351
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44633
diff changeset
3352 # Passing the '' locale means that the locale should be set according to the
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44633
diff changeset
3353 # user settings (environment variables).
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44633
diff changeset
3354 # Python sometimes avoids setting the global locale settings. When interfacing
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44633
diff changeset
3355 # with C code (e.g. the curses module or the Subversion bindings), the global
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44633
diff changeset
3356 # locale settings must be initialized correctly. Python 2 does not initialize
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44633
diff changeset
3357 # the global locale settings on interpreter startup. Python 3 sometimes
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44633
diff changeset
3358 # initializes LC_CTYPE, but not consistently at least on Windows. Therefore we
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44633
diff changeset
3359 # explicitly initialize it to get consistent behavior if it's not already
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44633
diff changeset
3360 # initialized. Since CPython commit 177d921c8c03d30daa32994362023f777624b10d,
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44633
diff changeset
3361 # LC_CTYPE is always initialized. If we require Python 3.8+, we should re-check
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44633
diff changeset
3362 # if we can remove this code.
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44633
diff changeset
3363 @contextlib.contextmanager
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44633
diff changeset
3364 def with_lc_ctype():
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44633
diff changeset
3365 oldloc = locale.setlocale(locale.LC_CTYPE, None)
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44633
diff changeset
3366 if oldloc == 'C':
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44633
diff changeset
3367 try:
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44633
diff changeset
3368 try:
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44633
diff changeset
3369 locale.setlocale(locale.LC_CTYPE, '')
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44633
diff changeset
3370 except locale.Error:
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44633
diff changeset
3371 # The likely case is that the locale from the environment
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44633
diff changeset
3372 # variables is unknown.
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44633
diff changeset
3373 pass
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44633
diff changeset
3374 yield
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44633
diff changeset
3375 finally:
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44633
diff changeset
3376 locale.setlocale(locale.LC_CTYPE, oldloc)
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44633
diff changeset
3377 else:
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44633
diff changeset
3378 yield
45062
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45034
diff changeset
3379
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45034
diff changeset
3380
51304
f15cb5111a1e pytype: move some type comment to proper annotation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51302
diff changeset
3381 def _estimatememory() -> Optional[int]:
45062
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45034
diff changeset
3382 """Provide an estimate for the available system memory in Bytes.
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45034
diff changeset
3383
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45034
diff changeset
3384 If no estimate can be provided on the platform, returns None.
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45034
diff changeset
3385 """
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45034
diff changeset
3386 if pycompat.sysplatform.startswith(b'win'):
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45034
diff changeset
3387 # On Windows, use the GlobalMemoryStatusEx kernel function directly.
51917
8f3cbea2547c util: add a comment to suppress a PyCharm warning about a PEP 8 violation
Matt Harbison <matt_harbison@yahoo.com>
parents: 51901
diff changeset
3388 # noinspection PyPep8Naming
45062
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45034
diff changeset
3389 from ctypes import c_long as DWORD, c_ulonglong as DWORDLONG
46684
7711853110b9 typing: add some type annotations to mercurial/util.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 46114
diff changeset
3390 from ctypes.wintypes import ( # pytype: disable=import-error
7711853110b9 typing: add some type annotations to mercurial/util.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 46114
diff changeset
3391 Structure,
7711853110b9 typing: add some type annotations to mercurial/util.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 46114
diff changeset
3392 byref,
7711853110b9 typing: add some type annotations to mercurial/util.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 46114
diff changeset
3393 sizeof,
7711853110b9 typing: add some type annotations to mercurial/util.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 46114
diff changeset
3394 windll,
7711853110b9 typing: add some type annotations to mercurial/util.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 46114
diff changeset
3395 )
45062
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45034
diff changeset
3396
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45034
diff changeset
3397 class MEMORYSTATUSEX(Structure):
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45034
diff changeset
3398 _fields_ = [
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45034
diff changeset
3399 ('dwLength', DWORD),
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45034
diff changeset
3400 ('dwMemoryLoad', DWORD),
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45034
diff changeset
3401 ('ullTotalPhys', DWORDLONG),
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45034
diff changeset
3402 ('ullAvailPhys', DWORDLONG),
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45034
diff changeset
3403 ('ullTotalPageFile', DWORDLONG),
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45034
diff changeset
3404 ('ullAvailPageFile', DWORDLONG),
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45034
diff changeset
3405 ('ullTotalVirtual', DWORDLONG),
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45034
diff changeset
3406 ('ullAvailVirtual', DWORDLONG),
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45034
diff changeset
3407 ('ullExtendedVirtual', DWORDLONG),
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45034
diff changeset
3408 ]
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45034
diff changeset
3409
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45034
diff changeset
3410 x = MEMORYSTATUSEX()
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45034
diff changeset
3411 x.dwLength = sizeof(x)
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45034
diff changeset
3412 windll.kernel32.GlobalMemoryStatusEx(byref(x))
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45034
diff changeset
3413 return x.ullAvailPhys
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45034
diff changeset
3414
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45034
diff changeset
3415 # On newer Unix-like systems and Mac OSX, the sysconf interface
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45034
diff changeset
3416 # can be used. _SC_PAGE_SIZE is part of POSIX; _SC_PHYS_PAGES
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45034
diff changeset
3417 # seems to be implemented on most systems.
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45034
diff changeset
3418 try:
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45034
diff changeset
3419 pagesize = os.sysconf(os.sysconf_names['SC_PAGE_SIZE'])
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45034
diff changeset
3420 pages = os.sysconf(os.sysconf_names['SC_PHYS_PAGES'])
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45034
diff changeset
3421 return pagesize * pages
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45034
diff changeset
3422 except OSError: # sysconf can fail
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45034
diff changeset
3423 pass
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45034
diff changeset
3424 except KeyError: # unknown parameter
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45034
diff changeset
3425 pass