mercurial-scm/hg: mercurial/utils/compression.py annotate

annotate mercurial/utils/compression.py @ 42098:5753e5949b51

py3: add b'' prefixes to new doctests in match.py # skip-blame as just b'' prefixes Differential Revision: https://phab.mercurial-scm.org/D6221

author	Pulkit Goyal <pulkit@yandex-team.ru>
date	Wed, 10 Apr 2019 03:10:53 +0530
parents	bb271ec2fbfb
children	2372284d9457

rev	line source
42041 3e47d1ec9da5 util: extract compression code in `mercurial.utils.compression` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41834 diff changeset	1 # compression.py - Mercurial utility functions for compression
8226 8b2cd04a6e97 put license and copyright info into comment blocks Martin Geisler <mg@lazybytes.net> parents: 8225 diff changeset	2 #
8b2cd04a6e97 put license and copyright info into comment blocks Martin Geisler <mg@lazybytes.net> parents: 8225 diff changeset	3 # This software may be used and distributed according to the terms of the
10263 25e572394f5c Update license to GPLv2+ Matt Mackall <mpm@selenic.com> parents: 9996 diff changeset	4 # GNU General Public License version 2 or any later version.
1082 ce96e316278a Update util.py docstrings, fix walk test mpm@selenic.com parents: 1081 diff changeset	5
419 28511fc21073 [PATCH] file seperator handling for the other 'OS' mpm@selenic.com parents: diff changeset	6
34137 a8994d08e4a2 doctest: use print_function and convert bytes to unicode where needed Yuya Nishihara <yuya@tcha.org> parents: 34134 diff changeset	7 from __future__ import absolute_import, print_function
27358 ac839ee45b6a util: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27357 diff changeset	8
ac839ee45b6a util: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27357 diff changeset	9 import bz2
ac839ee45b6a util: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27357 diff changeset	10 import collections
26266 1e042e31bd0c changegroup: move all compressions utilities in util Pierre-Yves David <pierre-yves.david@fb.com> parents: 26201 diff changeset	11 import zlib
3769 96095d9ff1f8 Add encoding detection Matt Mackall <mpm@selenic.com> parents: 3767 diff changeset	12
42041 3e47d1ec9da5 util: extract compression code in `mercurial.utils.compression` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41834 diff changeset	13 from .. import (
27358 ac839ee45b6a util: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27357 diff changeset	14 error,
ac839ee45b6a util: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27357 diff changeset	15 i18n,
28818 6041fb8f2da8 pycompat: add empty and queue to handle py3 divergence timeless <timeless@mozdev.org> parents: 28497 diff changeset	16 pycompat,
27358 ac839ee45b6a util: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27357 diff changeset	17 )
42041 3e47d1ec9da5 util: extract compression code in `mercurial.utils.compression` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41834 diff changeset	18 from . import (
37083 f99d64e8a4e4 stringutil: move generic string helpers to new module Yuya Nishihara <yuya@tcha.org> parents: 37082 diff changeset	19 stringutil,
f99d64e8a4e4 stringutil: move generic string helpers to new module Yuya Nishihara <yuya@tcha.org> parents: 37082 diff changeset	20 )
3769 96095d9ff1f8 Add encoding detection Matt Mackall <mpm@selenic.com> parents: 3767 diff changeset	21
42041 3e47d1ec9da5 util: extract compression code in `mercurial.utils.compression` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41834 diff changeset	22 safehasattr = pycompat.safehasattr
32201 4462a981e8df base85: proxy through util module Yuya Nishihara <yuya@tcha.org> parents: 32154 diff changeset	23
14926 4e7e63fc685a util: eliminate wildcard imports Adrian Buehlmann <adrian@cadifra.com> parents: 14918 diff changeset	24
27358 ac839ee45b6a util: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27357 diff changeset	25 _ = i18n._
14926 4e7e63fc685a util: eliminate wildcard imports Adrian Buehlmann <adrian@cadifra.com> parents: 14918 diff changeset	26
42041 3e47d1ec9da5 util: extract compression code in `mercurial.utils.compression` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41834 diff changeset	27 # compression code
36524 bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36432 diff changeset	28
42041 3e47d1ec9da5 util: extract compression code in `mercurial.utils.compression` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41834 diff changeset	29 SERVERROLE = 'server'
3e47d1ec9da5 util: extract compression code in `mercurial.utils.compression` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41834 diff changeset	30 CLIENTROLE = 'client'
33793 bbbbd3c30bfc util: add base class for transactional context managers Martin von Zweigbergk <martinvonz@google.com> parents: 33737 diff changeset	31
42041 3e47d1ec9da5 util: extract compression code in `mercurial.utils.compression` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41834 diff changeset	32 compewireprotosupport = collections.namedtuple(r'compenginewireprotosupport',
3e47d1ec9da5 util: extract compression code in `mercurial.utils.compression` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41834 diff changeset	33 (r'name', r'serverpriority',
3e47d1ec9da5 util: extract compression code in `mercurial.utils.compression` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41834 diff changeset	34 r'clientpriority'))
9097 431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639] Matt Mackall <mpm@selenic.com> parents: 9089 diff changeset	35
8207 dd8d5be57d65 util: take propertycache from context.py Matt Mackall <mpm@selenic.com> parents: 8181 diff changeset	36 class propertycache(object):
dd8d5be57d65 util: take propertycache from context.py Matt Mackall <mpm@selenic.com> parents: 8181 diff changeset	37 def __init__(self, func):
dd8d5be57d65 util: take propertycache from context.py Matt Mackall <mpm@selenic.com> parents: 8181 diff changeset	38 self.func = func
dd8d5be57d65 util: take propertycache from context.py Matt Mackall <mpm@selenic.com> parents: 8181 diff changeset	39 self.name = func.__name__
dd8d5be57d65 util: take propertycache from context.py Matt Mackall <mpm@selenic.com> parents: 8181 diff changeset	40 def __get__(self, obj, type=None):
dd8d5be57d65 util: take propertycache from context.py Matt Mackall <mpm@selenic.com> parents: 8181 diff changeset	41 result = self.func(obj)
18013 98c867ac1330 clfilter: add a propertycache that must be unfiltered Pierre-Yves David <pierre-yves.david@logilab.fr> parents: 17962 diff changeset	42 self.cachevalue(obj, result)
8207 dd8d5be57d65 util: take propertycache from context.py Matt Mackall <mpm@selenic.com> parents: 8181 diff changeset	43 return result
dd8d5be57d65 util: take propertycache from context.py Matt Mackall <mpm@selenic.com> parents: 8181 diff changeset	44
18013 98c867ac1330 clfilter: add a propertycache that must be unfiltered Pierre-Yves David <pierre-yves.david@logilab.fr> parents: 17962 diff changeset	45 def cachevalue(self, obj, value):
19951 d51c4d85ec23 spelling: random spell checker fixes Mads Kiilerich <madski@unity3d.com> parents: 19852 diff changeset	46 # __dict__ assignment required to bypass __setattr__ (eg: repoview)
19845 a1237a4b437d repoview: make propertycache.setcache compatible with repoview Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 19461 diff changeset	47 obj.__dict__[self.name] = value
18013 98c867ac1330 clfilter: add a propertycache that must be unfiltered Pierre-Yves David <pierre-yves.david@logilab.fr> parents: 17962 diff changeset	48
30350 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	49 class compressormanager(object):
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	50 """Holds registrations of various compression engines.
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	51
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	52 This class essentially abstracts the differences between compression
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	53 engines to allow new compression formats to be added easily, possibly from
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	54 extensions.
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	55
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	56 Compressors are registered against the global instance by calling its
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	57 ``register()`` method.
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	58 """
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	59 def __init__(self):
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	60 self._engines = {}
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	61 # Bundle spec human name to engine name.
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	62 self._bundlenames = {}
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	63 # Internal bundle identifier to engine name.
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	64 self._bundletypes = {}
30798 f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	65 # Revlog header to engine name.
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	66 self._revlogheaders = {}
30761 7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	67 # Wire proto identifier to engine name.
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	68 self._wiretypes = {}
30350 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	69
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	70 def __getitem__(self, key):
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	71 return self._engines[key]
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	72
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	73 def __contains__(self, key):
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	74 return key in self._engines
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	75
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	76 def __iter__(self):
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	77 return iter(self._engines.keys())
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	78
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	79 def register(self, engine):
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	80 """Register a compression engine with the manager.
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	81
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	82 The argument must be a ``compressionengine`` instance.
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	83 """
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	84 if not isinstance(engine, compressionengine):
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	85 raise ValueError(_('argument must be a compressionengine'))
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	86
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	87 name = engine.name()
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	88
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	89 if name in self._engines:
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	90 raise error.Abort(_('compression engine %s already registered') %
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	91 name)
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	92
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	93 bundleinfo = engine.bundletype()
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	94 if bundleinfo:
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	95 bundlename, bundletype = bundleinfo
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	96
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	97 if bundlename in self._bundlenames:
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	98 raise error.Abort(_('bundle name %s already registered') %
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	99 bundlename)
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	100 if bundletype in self._bundletypes:
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	101 raise error.Abort(_('bundle type %s already registered by %s') %
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	102 (bundletype, self._bundletypes[bundletype]))
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	103
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	104 # No external facing name declared.
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	105 if bundlename:
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	106 self._bundlenames[bundlename] = name
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	107
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	108 self._bundletypes[bundletype] = name
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	109
30761 7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	110 wiresupport = engine.wireprotosupport()
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	111 if wiresupport:
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	112 wiretype = wiresupport.name
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	113 if wiretype in self._wiretypes:
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	114 raise error.Abort(_('wire protocol compression %s already '
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	115 'registered by %s') %
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	116 (wiretype, self._wiretypes[wiretype]))
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	117
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	118 self._wiretypes[wiretype] = name
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	119
30798 f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	120 revlogheader = engine.revlogheader()
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	121 if revlogheader and revlogheader in self._revlogheaders:
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	122 raise error.Abort(_('revlog header %s already registered by %s') %
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	123 (revlogheader, self._revlogheaders[revlogheader]))
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	124
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	125 if revlogheader:
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	126 self._revlogheaders[revlogheader] = name
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	127
30350 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	128 self._engines[name] = engine
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	129
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	130 @property
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	131 def supportedbundlenames(self):
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	132 return set(self._bundlenames.keys())
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	133
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	134 @property
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	135 def supportedbundletypes(self):
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	136 return set(self._bundletypes.keys())
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	137
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	138 def forbundlename(self, bundlename):
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	139 """Obtain a compression engine registered to a bundle name.
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	140
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	141 Will raise KeyError if the bundle type isn't registered.
30438 90933e4e44fd util: check for compression engine availability before returning Gregory Szorc <gregory.szorc@gmail.com> parents: 30437 diff changeset	142
90933e4e44fd util: check for compression engine availability before returning Gregory Szorc <gregory.szorc@gmail.com> parents: 30437 diff changeset	143 Will abort if the engine is known but not available.
30350 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	144 """
30438 90933e4e44fd util: check for compression engine availability before returning Gregory Szorc <gregory.szorc@gmail.com> parents: 30437 diff changeset	145 engine = self._engines[self._bundlenames[bundlename]]
90933e4e44fd util: check for compression engine availability before returning Gregory Szorc <gregory.szorc@gmail.com> parents: 30437 diff changeset	146 if not engine.available():
90933e4e44fd util: check for compression engine availability before returning Gregory Szorc <gregory.szorc@gmail.com> parents: 30437 diff changeset	147 raise error.Abort(_('compression engine %s could not be loaded') %
90933e4e44fd util: check for compression engine availability before returning Gregory Szorc <gregory.szorc@gmail.com> parents: 30437 diff changeset	148 engine.name())
90933e4e44fd util: check for compression engine availability before returning Gregory Szorc <gregory.szorc@gmail.com> parents: 30437 diff changeset	149 return engine
30350 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	150
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	151 def forbundletype(self, bundletype):
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	152 """Obtain a compression engine registered to a bundle type.
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	153
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	154 Will raise KeyError if the bundle type isn't registered.
30438 90933e4e44fd util: check for compression engine availability before returning Gregory Szorc <gregory.szorc@gmail.com> parents: 30437 diff changeset	155
90933e4e44fd util: check for compression engine availability before returning Gregory Szorc <gregory.szorc@gmail.com> parents: 30437 diff changeset	156 Will abort if the engine is known but not available.
30350 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	157 """
30438 90933e4e44fd util: check for compression engine availability before returning Gregory Szorc <gregory.szorc@gmail.com> parents: 30437 diff changeset	158 engine = self._engines[self._bundletypes[bundletype]]
90933e4e44fd util: check for compression engine availability before returning Gregory Szorc <gregory.szorc@gmail.com> parents: 30437 diff changeset	159 if not engine.available():
90933e4e44fd util: check for compression engine availability before returning Gregory Szorc <gregory.szorc@gmail.com> parents: 30437 diff changeset	160 raise error.Abort(_('compression engine %s could not be loaded') %
90933e4e44fd util: check for compression engine availability before returning Gregory Szorc <gregory.szorc@gmail.com> parents: 30437 diff changeset	161 engine.name())
90933e4e44fd util: check for compression engine availability before returning Gregory Szorc <gregory.szorc@gmail.com> parents: 30437 diff changeset	162 return engine
30350 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	163
30761 7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	164 def supportedwireengines(self, role, onlyavailable=True):
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	165 """Obtain compression engines that support the wire protocol.
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	166
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	167 Returns a list of engines in prioritized order, most desired first.
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	168
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	169 If ``onlyavailable`` is set, filter out engines that can't be
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	170 loaded.
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	171 """
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	172 assert role in (SERVERROLE, CLIENTROLE)
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	173
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	174 attr = 'serverpriority' if role == SERVERROLE else 'clientpriority'
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	175
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	176 engines = [self._engines[e] for e in self._wiretypes.values()]
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	177 if onlyavailable:
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	178 engines = [e for e in engines if e.available()]
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	179
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	180 def getkey(e):
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	181 # Sort first by priority, highest first. In case of tie, sort
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	182 # alphabetically. This is arbitrary, but ensures output is
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	183 # stable.
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	184 w = e.wireprotosupport()
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	185 return -1 * getattr(w, attr), w.name
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	186
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	187 return list(sorted(engines, key=getkey))
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	188
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	189 def forwiretype(self, wiretype):
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	190 engine = self._engines[self._wiretypes[wiretype]]
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	191 if not engine.available():
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	192 raise error.Abort(_('compression engine %s could not be loaded') %
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	193 engine.name())
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	194 return engine
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	195
30798 f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	196 def forrevlogheader(self, header):
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	197 """Obtain a compression engine registered to a revlog header.
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	198
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	199 Will raise KeyError if the revlog header value isn't registered.
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	200 """
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	201 return self._engines[self._revlogheaders[header]]
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	202
30350 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	203 compengines = compressormanager()
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	204
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	205 class compressionengine(object):
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	206 """Base class for compression engines.
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	207
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	208 Compression engines must implement the interface defined by this class.
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	209 """
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	210 def name(self):
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	211 """Returns the name of the compression engine.
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	212
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	213 This is the key the engine is registered under.
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	214
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	215 This method must be implemented.
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	216 """
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	217 raise NotImplementedError()
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	218
30437 64d7275445d0 util: expose an "available" API on compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30418 diff changeset	219 def available(self):
64d7275445d0 util: expose an "available" API on compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30418 diff changeset	220 """Whether the compression engine is available.
64d7275445d0 util: expose an "available" API on compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30418 diff changeset	221
64d7275445d0 util: expose an "available" API on compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30418 diff changeset	222 The intent of this method is to allow optional compression engines
64d7275445d0 util: expose an "available" API on compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30418 diff changeset	223 that may not be available in all installations (such as engines relying
64d7275445d0 util: expose an "available" API on compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30418 diff changeset	224 on C extensions that may not be present).
64d7275445d0 util: expose an "available" API on compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30418 diff changeset	225 """
64d7275445d0 util: expose an "available" API on compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30418 diff changeset	226 return True
64d7275445d0 util: expose an "available" API on compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30418 diff changeset	227
30350 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	228 def bundletype(self):
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	229 """Describes bundle identifiers for this engine.
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	230
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	231 If this compression engine isn't supported for bundles, returns None.
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	232
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	233 If this engine can be used for bundles, returns a 2-tuple of strings of
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	234 the user-facing "bundle spec" compression name and an internal
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	235 identifier used to denote the compression format within bundles. To
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	236 exclude the name from external usage, set the first element to ``None``.
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	237
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	238 If bundle compression is supported, the class must also implement
30359 673f0fdc1046 util: remove compressorobj API from compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30356 diff changeset	239 ``compressstream`` and `decompressorreader``.
31792 55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	240
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	241 The docstring of this method is used in the help system to tell users
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	242 about this engine.
30350 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	243 """
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	244 return None
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	245
30761 7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	246 def wireprotosupport(self):
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	247 """Declare support for this compression format on the wire protocol.
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	248
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	249 If this compression engine isn't supported for compressing wire
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	250 protocol payloads, returns None.
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	251
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	252 Otherwise, returns ``compenginewireprotosupport`` with the following
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	253 fields:
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	254
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	255 * String format identifier
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	256 * Integer priority for the server
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	257 * Integer priority for the client
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	258
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	259 The integer priorities are used to order the advertisement of format
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	260 support by server and client. The highest integer is advertised
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	261 first. Integers with non-positive values aren't advertised.
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	262
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	263 The priority values are somewhat arbitrary and only used for default
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	264 ordering. The relative order can be changed via config options.
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	265
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	266 If wire protocol compression is supported, the class must also implement
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	267 ``compressstream`` and ``decompressorreader``.
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	268 """
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	269 return None
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	270
30798 f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	271 def revlogheader(self):
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	272 """Header added to revlog chunks that identifies this engine.
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	273
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	274 If this engine can be used to compress revlogs, this method should
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	275 return the bytes used to identify chunks compressed with this engine.
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	276 Else, the method should return ``None`` to indicate it does not
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	277 participate in revlog compression.
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	278 """
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	279 return None
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	280
30356 c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	281 def compressstream(self, it, opts=None):
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	282 """Compress an iterator of chunks.
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	283
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	284 The method receives an iterator (ideally a generator) of chunks of
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	285 bytes to be compressed. It returns an iterator (ideally a generator)
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	286 of bytes of chunks representing the compressed output.
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	287
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	288 Optionally accepts an argument defining how to perform compression.
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	289 Each engine treats this argument differently.
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	290 """
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	291 raise NotImplementedError()
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	292
30350 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	293 def decompressorreader(self, fh):
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	294 """Perform decompression on a file object.
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	295
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	296 Argument is an object with a ``read(size)`` method that returns
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	297 compressed data. Return value is an object with a ``read(size)`` that
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	298 returns uncompressed data.
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	299 """
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	300 raise NotImplementedError()
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	301
30794 31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	302 def revlogcompressor(self, opts=None):
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	303 """Obtain an object that can be used to compress revlog entries.
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	304
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	305 The object has a ``compress(data)`` method that compresses binary
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	306 data. This method returns compressed binary data or ``None`` if
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	307 the data could not be compressed (too small, not compressible, etc).
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	308 The returned data should have a header uniquely identifying this
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	309 compression format so decompression can be routed to this engine.
30798 f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	310 This header should be identified by the ``revlogheader()`` return
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	311 value.
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	312
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	313 The object has a ``decompress(data)`` method that decompresses
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	314 data. The method will only be called if ``data`` begins with
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	315 ``revlogheader()``. The method should return the raw, uncompressed
39777 b63dee7bd0d9 global: replace most uses of RevlogError with StorageError (API) Gregory Szorc <gregory.szorc@gmail.com> parents: 39571 diff changeset	316 data or raise a ``StorageError``.
30794 31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	317
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	318 The object is reusable but is not thread safe.
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	319 """
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	320 raise NotImplementedError()
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	321
38713 27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	322 class _CompressedStreamReader(object):
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	323 def __init__(self, fh):
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	324 if safehasattr(fh, 'unbufferedread'):
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	325 self._reader = fh.unbufferedread
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	326 else:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	327 self._reader = fh.read
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	328 self._pending = []
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	329 self._pos = 0
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	330 self._eof = False
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	331
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	332 def _decompress(self, chunk):
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	333 raise NotImplementedError()
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	334
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	335 def read(self, l):
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	336 buf = []
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	337 while True:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	338 while self._pending:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	339 if len(self._pending[0]) > l + self._pos:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	340 newbuf = self._pending[0]
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	341 buf.append(newbuf[self._pos:self._pos + l])
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	342 self._pos += l
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	343 return ''.join(buf)
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	344
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	345 newbuf = self._pending.pop(0)
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	346 if self._pos:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	347 buf.append(newbuf[self._pos:])
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	348 l -= len(newbuf) - self._pos
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	349 else:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	350 buf.append(newbuf)
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	351 l -= len(newbuf)
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	352 self._pos = 0
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	353
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	354 if self._eof:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	355 return ''.join(buf)
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	356 chunk = self._reader(65536)
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	357 self._decompress(chunk)
39209 1af95139e5ec util: improve handling of truncated compressed streams Joerg Sonnenberger <joerg@bec.de> parents: 39060 diff changeset	358 if not chunk and not self._pending and not self._eof:
1af95139e5ec util: improve handling of truncated compressed streams Joerg Sonnenberger <joerg@bec.de> parents: 39060 diff changeset	359 # No progress and no new data, bail out
1af95139e5ec util: improve handling of truncated compressed streams Joerg Sonnenberger <joerg@bec.de> parents: 39060 diff changeset	360 return ''.join(buf)
38713 27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	361
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	362 class _GzipCompressedStreamReader(_CompressedStreamReader):
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	363 def __init__(self, fh):
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	364 super(_GzipCompressedStreamReader, self).__init__(fh)
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	365 self._decompobj = zlib.decompressobj()
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	366 def _decompress(self, chunk):
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	367 newbuf = self._decompobj.decompress(chunk)
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	368 if newbuf:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	369 self._pending.append(newbuf)
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	370 d = self._decompobj.copy()
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	371 try:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	372 d.decompress('x')
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	373 d.flush()
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	374 if d.unused_data == 'x':
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	375 self._eof = True
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	376 except zlib.error:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	377 pass
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	378
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	379 class _BZ2CompressedStreamReader(_CompressedStreamReader):
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	380 def __init__(self, fh):
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	381 super(_BZ2CompressedStreamReader, self).__init__(fh)
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	382 self._decompobj = bz2.BZ2Decompressor()
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	383 def _decompress(self, chunk):
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	384 newbuf = self._decompobj.decompress(chunk)
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	385 if newbuf:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	386 self._pending.append(newbuf)
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	387 try:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	388 while True:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	389 newbuf = self._decompobj.decompress('')
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	390 if newbuf:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	391 self._pending.append(newbuf)
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	392 else:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	393 break
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	394 except EOFError:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	395 self._eof = True
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	396
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	397 class _TruncatedBZ2CompressedStreamReader(_BZ2CompressedStreamReader):
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	398 def __init__(self, fh):
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	399 super(_TruncatedBZ2CompressedStreamReader, self).__init__(fh)
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	400 newbuf = self._decompobj.decompress('BZ')
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	401 if newbuf:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	402 self._pending.append(newbuf)
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	403
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	404 class _ZstdCompressedStreamReader(_CompressedStreamReader):
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	405 def __init__(self, fh, zstd):
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	406 super(_ZstdCompressedStreamReader, self).__init__(fh)
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	407 self._zstd = zstd
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	408 self._decompobj = zstd.ZstdDecompressor().decompressobj()
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	409 def _decompress(self, chunk):
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	410 newbuf = self._decompobj.decompress(chunk)
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	411 if newbuf:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	412 self._pending.append(newbuf)
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	413 try:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	414 while True:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	415 newbuf = self._decompobj.decompress('')
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	416 if newbuf:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	417 self._pending.append(newbuf)
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	418 else:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	419 break
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	420 except self._zstd.ZstdError:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	421 self._eof = True
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	422
30350 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	423 class _zlibengine(compressionengine):
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	424 def name(self):
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	425 return 'zlib'
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	426
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	427 def bundletype(self):
31792 55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	428 """zlib compression using the DEFLATE algorithm.
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	429
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	430 All Mercurial clients should support this format. The compression
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	431 algorithm strikes a reasonable balance between compression ratio
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	432 and size.
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	433 """
30350 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	434 return 'gzip', 'GZ'
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	435
30761 7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	436 def wireprotosupport(self):
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	437 return compewireprotosupport('zlib', 20, 20)
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	438
30798 f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	439 def revlogheader(self):
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	440 return 'x'
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	441
30356 c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	442 def compressstream(self, it, opts=None):
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	443 opts = opts or {}
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	444
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	445 z = zlib.compressobj(opts.get('level', -1))
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	446 for chunk in it:
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	447 data = z.compress(chunk)
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	448 # Not all calls to compress emit data. It is cheaper to inspect
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	449 # here than to feed empty chunks through generator.
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	450 if data:
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	451 yield data
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	452
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	453 yield z.flush()
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	454
30350 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	455 def decompressorreader(self, fh):
38713 27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	456 return _GzipCompressedStreamReader(fh)
30350 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	457
30794 31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	458 class zlibrevlogcompressor(object):
42042 aaececb4b066 compression: accept level management for zlib compression Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42041 diff changeset	459
aaececb4b066 compression: accept level management for zlib compression Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42041 diff changeset	460 def __init__(self, level=None):
aaececb4b066 compression: accept level management for zlib compression Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42041 diff changeset	461 self._level = level
aaececb4b066 compression: accept level management for zlib compression Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42041 diff changeset	462
30794 31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	463 def compress(self, data):
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	464 insize = len(data)
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	465 # Caller handles empty input case.
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	466 assert insize > 0
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	467
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	468 if insize < 44:
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	469 return None
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	470
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	471 elif insize <= 1000000:
42042 aaececb4b066 compression: accept level management for zlib compression Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42041 diff changeset	472 if self._level is None:
aaececb4b066 compression: accept level management for zlib compression Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42041 diff changeset	473 compressed = zlib.compress(data)
aaececb4b066 compression: accept level management for zlib compression Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42041 diff changeset	474 else:
aaececb4b066 compression: accept level management for zlib compression Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42041 diff changeset	475 compressed = zlib.compress(data, self._level)
30794 31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	476 if len(compressed) < insize:
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	477 return compressed
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	478 return None
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	479
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	480 # zlib makes an internal copy of the input buffer, doubling
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	481 # memory usage for large inputs. So do streaming compression
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	482 # on large inputs.
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	483 else:
42042 aaececb4b066 compression: accept level management for zlib compression Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42041 diff changeset	484 if self._level is None:
aaececb4b066 compression: accept level management for zlib compression Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42041 diff changeset	485 z = zlib.compressobj()
aaececb4b066 compression: accept level management for zlib compression Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42041 diff changeset	486 else:
aaececb4b066 compression: accept level management for zlib compression Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42041 diff changeset	487 z = zlib.compressobj(level=self._level)
30794 31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	488 parts = []
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	489 pos = 0
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	490 while pos < insize:
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	491 pos2 = pos + 2**20
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	492 parts.append(z.compress(data[pos:pos2]))
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	493 pos = pos2
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	494 parts.append(z.flush())
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	495
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	496 if sum(map(len, parts)) < insize:
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	497 return ''.join(parts)
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	498 return None
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	499
30798 f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	500 def decompress(self, data):
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	501 try:
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	502 return zlib.decompress(data)
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	503 except zlib.error as e:
39777 b63dee7bd0d9 global: replace most uses of RevlogError with StorageError (API) Gregory Szorc <gregory.szorc@gmail.com> parents: 39571 diff changeset	504 raise error.StorageError(_('revlog decompress error: %s') %
b63dee7bd0d9 global: replace most uses of RevlogError with StorageError (API) Gregory Szorc <gregory.szorc@gmail.com> parents: 39571 diff changeset	505 stringutil.forcebytestr(e))
30798 f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	506
30794 31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	507 def revlogcompressor(self, opts=None):
42043 1fac9b931d46 compression: introduce a `storage.revlog.zlib.level` configuration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42042 diff changeset	508 level = None
1fac9b931d46 compression: introduce a `storage.revlog.zlib.level` configuration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42042 diff changeset	509 if opts is not None:
1fac9b931d46 compression: introduce a `storage.revlog.zlib.level` configuration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42042 diff changeset	510 level = opts.get('zlib.level')
1fac9b931d46 compression: introduce a `storage.revlog.zlib.level` configuration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42042 diff changeset	511 return self.zlibrevlogcompressor(level)
30794 31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	512
30350 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	513 compengines.register(_zlibengine())
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	514
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	515 class _bz2engine(compressionengine):
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	516 def name(self):
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	517 return 'bz2'
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	518
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	519 def bundletype(self):
31792 55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	520 """An algorithm that produces smaller bundles than ``gzip``.
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	521
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	522 All Mercurial clients should support this format.
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	523
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	524 This engine will likely produce smaller bundles than ``gzip`` but
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	525 will be significantly slower, both during compression and
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	526 decompression.
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	527
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	528 If available, the ``zstd`` engine can yield similar or better
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	529 compression at much higher speeds.
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	530 """
30350 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	531 return 'bzip2', 'BZ'
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	532
30761 7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	533 # We declare a protocol name but don't advertise by default because
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	534 # it is slow.
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	535 def wireprotosupport(self):
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	536 return compewireprotosupport('bzip2', 0, 0)
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	537
30356 c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	538 def compressstream(self, it, opts=None):
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	539 opts = opts or {}
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	540 z = bz2.BZ2Compressor(opts.get('level', 9))
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	541 for chunk in it:
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	542 data = z.compress(chunk)
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	543 if data:
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	544 yield data
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	545
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	546 yield z.flush()
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	547
30350 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	548 def decompressorreader(self, fh):
38713 27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	549 return _BZ2CompressedStreamReader(fh)
30350 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	550
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	551 compengines.register(_bz2engine())
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	552
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	553 class _truncatedbz2engine(compressionengine):
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	554 def name(self):
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	555 return 'bz2truncated'
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	556
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	557 def bundletype(self):
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	558 return None, '_truncatedBZ'
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	559
30359 673f0fdc1046 util: remove compressorobj API from compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30356 diff changeset	560 # We don't implement compressstream because it is hackily handled elsewhere.
30350 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	561
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	562 def decompressorreader(self, fh):
38713 27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	563 return _TruncatedBZ2CompressedStreamReader(fh)
30350 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	564
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	565 compengines.register(_truncatedbz2engine())
30265 6a8aff737a17 util: put compression code next to each other Gregory Szorc <gregory.szorc@gmail.com> parents: 30181 diff changeset	566
30350 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	567 class _noopengine(compressionengine):
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	568 def name(self):
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	569 return 'none'
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	570
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	571 def bundletype(self):
31792 55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	572 """No compression is performed.
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	573
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	574 Use this compression engine to explicitly disable compression.
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	575 """
30350 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	576 return 'none', 'UN'
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	577
30761 7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	578 # Clients always support uncompressed payloads. Servers don't because
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	579 # unless you are on a fast network, uncompressed payloads can easily
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	580 # saturate your network pipe.
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	581 def wireprotosupport(self):
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	582 return compewireprotosupport('none', 0, 10)
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	583
30798 f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	584 # We don't implement revlogheader because it is handled specially
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	585 # in the revlog class.
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	586
30356 c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	587 def compressstream(self, it, opts=None):
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	588 return it
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	589
30350 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	590 def decompressorreader(self, fh):
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	591 return fh
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	592
30794 31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	593 class nooprevlogcompressor(object):
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	594 def compress(self, data):
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	595 return None
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	596
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	597 def revlogcompressor(self, opts=None):
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	598 return self.nooprevlogcompressor()
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	599
30350 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	600 compengines.register(_noopengine())
26266 1e042e31bd0c changegroup: move all compressions utilities in util Pierre-Yves David <pierre-yves.david@fb.com> parents: 26201 diff changeset	601
30442 41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	602 class _zstdengine(compressionengine):
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	603 def name(self):
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	604 return 'zstd'
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	605
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	606 @propertycache
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	607 def _module(self):
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	608 # Not all installs have the zstd module available. So defer importing
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	609 # until first access.
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	610 try:
42041 3e47d1ec9da5 util: extract compression code in `mercurial.utils.compression` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41834 diff changeset	611 from .. import zstd
30442 41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	612 # Force delayed import.
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	613 zstd.__version__
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	614 return zstd
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	615 except ImportError:
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	616 return None
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	617
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	618 def available(self):
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	619 return bool(self._module)
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	620
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	621 def bundletype(self):
31792 55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	622 """A modern compression algorithm that is fast and highly flexible.
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	623
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	624 Only supported by Mercurial 4.1 and newer clients.
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	625
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	626 With the default settings, zstd compression is both faster and yields
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	627 better compression than ``gzip``. It also frequently yields better
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	628 compression than ``bzip2`` while operating at much higher speeds.
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	629
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	630 If this engine is available and backwards compatibility is not a
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	631 concern, it is likely the best available engine.
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	632 """
30442 41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	633 return 'zstd', 'ZS'
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	634
30761 7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	635 def wireprotosupport(self):
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	636 return compewireprotosupport('zstd', 50, 50)
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	637
30798 f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	638 def revlogheader(self):
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	639 return '\x28'
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	640
30442 41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	641 def compressstream(self, it, opts=None):
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	642 opts = opts or {}
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	643 # zstd level 3 is almost always significantly faster than zlib
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	644 # while providing no worse compression. It strikes a good balance
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	645 # between speed and compression.
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	646 level = opts.get('level', 3)
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	647
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	648 zstd = self._module
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	649 z = zstd.ZstdCompressor(level=level).compressobj()
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	650 for chunk in it:
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	651 data = z.compress(chunk)
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	652 if data:
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	653 yield data
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	654
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	655 yield z.flush()
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	656
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	657 def decompressorreader(self, fh):
38713 27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	658 return _ZstdCompressedStreamReader(fh, self._module)
30442 41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	659
30794 31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	660 class zstdrevlogcompressor(object):
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	661 def __init__(self, zstd, level=3):
37496 1765ed63db40 util: drop write_content_size=True Gregory Szorc <gregory.szorc@gmail.com> parents: 37455 diff changeset	662 # TODO consider omitting frame magic to save 4 bytes.
1765ed63db40 util: drop write_content_size=True Gregory Szorc <gregory.szorc@gmail.com> parents: 37455 diff changeset	663 # This writes content sizes into the frame header. That is
1765ed63db40 util: drop write_content_size=True Gregory Szorc <gregory.szorc@gmail.com> parents: 37455 diff changeset	664 # extra storage. But it allows a correct size memory allocation
1765ed63db40 util: drop write_content_size=True Gregory Szorc <gregory.szorc@gmail.com> parents: 37455 diff changeset	665 # to hold the result.
1765ed63db40 util: drop write_content_size=True Gregory Szorc <gregory.szorc@gmail.com> parents: 37455 diff changeset	666 self._cctx = zstd.ZstdCompressor(level=level)
30798 f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	667 self._dctx = zstd.ZstdDecompressor()
30794 31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	668 self._compinsize = zstd.COMPRESSION_RECOMMENDED_INPUT_SIZE
30798 f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	669 self._decompinsize = zstd.DECOMPRESSION_RECOMMENDED_INPUT_SIZE
30794 31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	670
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	671 def compress(self, data):
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	672 insize = len(data)
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	673 # Caller handles empty input case.
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	674 assert insize > 0
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	675
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	676 if insize < 50:
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	677 return None
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	678
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	679 elif insize <= 1000000:
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	680 compressed = self._cctx.compress(data)
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	681 if len(compressed) < insize:
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	682 return compressed
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	683 return None
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	684 else:
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	685 z = self._cctx.compressobj()
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	686 chunks = []
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	687 pos = 0
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	688 while pos < insize:
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	689 pos2 = pos + self._compinsize
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	690 chunk = z.compress(data[pos:pos2])
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	691 if chunk:
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	692 chunks.append(chunk)
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	693 pos = pos2
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	694 chunks.append(z.flush())
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	695
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	696 if sum(map(len, chunks)) < insize:
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	697 return ''.join(chunks)
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	698 return None
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	699
30798 f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	700 def decompress(self, data):
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	701 insize = len(data)
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	702
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	703 try:
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	704 # This was measured to be faster than other streaming
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	705 # decompressors.
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	706 dobj = self._dctx.decompressobj()
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	707 chunks = []
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	708 pos = 0
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	709 while pos < insize:
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	710 pos2 = pos + self._decompinsize
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	711 chunk = dobj.decompress(data[pos:pos2])
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	712 if chunk:
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	713 chunks.append(chunk)
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	714 pos = pos2
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	715 # Frame should be exhausted, so no finish() API.
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	716
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	717 return ''.join(chunks)
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	718 except Exception as e:
39777 b63dee7bd0d9 global: replace most uses of RevlogError with StorageError (API) Gregory Szorc <gregory.szorc@gmail.com> parents: 39571 diff changeset	719 raise error.StorageError(_('revlog decompress error: %s') %
b63dee7bd0d9 global: replace most uses of RevlogError with StorageError (API) Gregory Szorc <gregory.szorc@gmail.com> parents: 39571 diff changeset	720 stringutil.forcebytestr(e))
30798 f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	721
30794 31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	722 def revlogcompressor(self, opts=None):
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	723 opts = opts or {}
42044 bb271ec2fbfb compression: introduce a `storage.revlog.zstd.level` configuration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42043 diff changeset	724 level = opts.get('zstd.level')
bb271ec2fbfb compression: introduce a `storage.revlog.zstd.level` configuration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42043 diff changeset	725 if level is None:
bb271ec2fbfb compression: introduce a `storage.revlog.zstd.level` configuration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42043 diff changeset	726 level = opts.get('level')
bb271ec2fbfb compression: introduce a `storage.revlog.zstd.level` configuration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42043 diff changeset	727 if level is None:
bb271ec2fbfb compression: introduce a `storage.revlog.zstd.level` configuration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42043 diff changeset	728 level = 3
bb271ec2fbfb compression: introduce a `storage.revlog.zstd.level` configuration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42043 diff changeset	729 return self.zstdrevlogcompressor(self._module, level=level)
30794 31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	730
30442 41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	731 compengines.register(_zstdengine())
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	732
31792 55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	733 def bundlecompressiontopics():
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	734 """Obtains a list of available bundle compressions for use in help."""
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	735 # help.makeitemsdocs() expects a dict of names to items with a .__doc__.
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	736 items = {}
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	737
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	738 # We need to format the docstring. So use a dummy object/type to hold it
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	739 # rather than mutating the original.
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	740 class docobject(object):
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	741 pass
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	742
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	743 for name in compengines:
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	744 engine = compengines[name]
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	745
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	746 if not engine.available():
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	747 continue
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	748
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	749 bt = engine.bundletype()
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	750 if not bt or not bt[0]:
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	751 continue
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	752
40251 3c89227788a2 py3: build help of compression engines in bytes Yuya Nishihara <yuya@tcha.org> parents: 40029 diff changeset	753 doc = b'``%s``\n %s' % (bt[0], pycompat.getdoc(engine.bundletype))
31792 55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	754
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	755 value = docobject()
40251 3c89227788a2 py3: build help of compression engines in bytes Yuya Nishihara <yuya@tcha.org> parents: 40029 diff changeset	756 value.__doc__ = pycompat.sysstr(doc)
33818 ed04d7254a91 i18n: use saved object to get actual function information if available FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 33799 diff changeset	757 value._origdoc = engine.bundletype.__doc__
ed04d7254a91 i18n: use saved object to get actual function information if available FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 33799 diff changeset	758 value._origfunc = engine.bundletype
31792 55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	759
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	760 items[bt[0]] = value
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	761
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	762 return items
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	763
33820 fa7e30efe05a i18n: get translation entries for description of each compression engines FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 33818 diff changeset	764 i18nfunctions = bundlecompressiontopics().values()

Mercurial > public > mercurial-scm > hg

annotate mercurial/utils/compression.py @ 42098:5753e5949b51