mercurial-scm/hg: mercurial/utils/compression.py annotate

annotate mercurial/utils/compression.py @ 45892:06b64fabf91c

copies: cache the ancestor checking call when tracing copy A good share of the time spent in this function is spent doing ancestors checking. To avoid spending time in duplicated call, we cache the result of calls. In the slower case, this provide a quite significant performance boost. Below are the result for a set of selected pairs (many of them pathological): (And further down is another table that summarize the current state of filelog based vs changeset base copy tracing) The benchmark have been configured to be killed after 6 minutes of runtime, which mean that any detect slower than 2 minutes will be marked as "killed". This drop some useful information about how much slower these case are? but also prevent 99% of the benchmark time to be spent on case that can be labelled "very slow" anyway. Repo Case Source-Rev Dest-Rev Old-Time New-Time Difference Factor ------------------------------------------------------------------------------------------------------------------------------------ mercurial x_revs_x_added_0_copies ad6b123de1c7 39cfcef4f463 : 0.000044 s, 0.000044 s, +0.000000 s, ? 1.0000 mercurial x_revs_x_added_x_copies 2b1c78674230 0c1d10351869 : 0.000138 s, 0.000138 s, +0.000000 s, ? 1.0000 mercurial x000_revs_x000_added_x_copies 81f8ff2a9bf2 dd3267698d84 : 0.005067 s, 0.005052 s, -0.000015 s, ? 0.9970 pypy x_revs_x_added_0_copies aed021ee8ae8 099ed31b181b : 0.000218 s, 0.000219 s, +0.000001 s, ? 1.0046 pypy x_revs_x000_added_0_copies 4aa4e1f8e19a 359343b9ac0e : 0.000053 s, 0.000055 s, +0.000002 s, ? 1.0377 pypy x_revs_x_added_x_copies ac52eb7bbbb0 72e022663155 : 0.000125 s, 0.000128 s, +0.000003 s, ? 1.0240 pypy x_revs_x00_added_x_copies c3b14617fbd7 ace7255d9a26 : 0.001098 s, 0.001089 s, -0.000009 s, ? 0.9918 pypy x_revs_x000_added_x000_copies df6f7a526b60 a83dc6a2d56f : 0.017546 s, 0.017407 s, -0.000139 s, ? 0.9921 pypy x000_revs_xx00_added_0_copies 89a76aede314 2f22446ff07e : 0.096723 s, 0.094175 s, -0.002548 s, ? 0.9737 pypy x000_revs_x000_added_x_copies 8a3b5bfd266e 2c68e87c3efe : 0.271796 s, 0.238009 s, -0.033787 s, ? 0.8757 pypy x000_revs_x000_added_x000_copies 89a76aede314 7b3dda341c84 : 0.128602 s, 0.125876 s, -0.002726 s, ? 0.9788 pypy x0000_revs_x_added_0_copies d1defd0dc478 c9cb1334cc78 : 7.086742 s, 3.581556 s, -3.505186 s, ? 0.5054 pypy x0000_revs_xx000_added_0_copies bf2c629d0071 4ffed77c095c : 0.016634 s, 0.016721 s, +0.000087 s, ? 1.0052 pypy x0000_revs_xx000_added_x000_copies 08ea3258278e d9fa043f30c0 : 0.254225 s, 0.242367 s, -0.011858 s, ? 0.9534 netbeans x_revs_x_added_0_copies fb0955ffcbcd a01e9239f9e7 : 0.000166 s, 0.000165 s, -0.000001 s, ? 0.9940 netbeans x_revs_x000_added_0_copies 6f360122949f 20eb231cc7d0 : 0.000118 s, 0.000114 s, -0.000004 s, ? 0.9661 netbeans x_revs_x_added_x_copies 1ada3faf6fb6 5a39d12eecf4 : 0.000296 s, 0.000296 s, +0.000000 s, ? 1.0000 netbeans x_revs_x00_added_x_copies 35be93ba1e2c 9eec5e90c05f : 0.001137 s, 0.001124 s, -0.000013 s, ? 0.9886 netbeans x000_revs_xx00_added_0_copies eac3045b4fdd 51d4ae7f1290 : 0.014133 s, 0.013060 s, -0.001073 s, ? 0.9241 netbeans x000_revs_x000_added_x_copies e2063d266acd 6081d72689dc : 0.016988 s, 0.017112 s, +0.000124 s, ? 1.0073 netbeans x000_revs_x000_added_x000_copies ff453e9fee32 411350406ec2 : 0.676361 s, 0.660350 s, -0.016011 s, ? 0.9763 netbeans x0000_revs_xx000_added_x000_copies 588c2d1ced70 1aad62e59ddd : 12.515149 s, 10.032499 s, -2.482650 s, ? 0.8016 mozilla-central x_revs_x_added_0_copies 3697f962bb7b 7015fcdd43a2 : 0.000186 s, 0.000189 s, +0.000003 s, ? 1.0161 mozilla-central x_revs_x000_added_0_copies dd390860c6c9 40d0c5bed75d : 0.000459 s, 0.000462 s, +0.000003 s, ? 1.0065 mozilla-central x_revs_x_added_x_copies 8d198483ae3b 14207ffc2b2f : 0.000273 s, 0.000270 s, -0.000003 s, ? 0.9890 mozilla-central x_revs_x00_added_x_copies 98cbc58cc6bc 446a150332c3 : 0.001503 s, 0.001474 s, -0.000029 s, ? 0.9807 mozilla-central x_revs_x000_added_x000_copies 3c684b4b8f68 0a5e72d1b479 : 0.004862 s, 0.004806 s, -0.000056 s, ? 0.9885 mozilla-central x_revs_x0000_added_x0000_copies effb563bb7e5 c07a39dc4e80 : 0.088291 s, 0.085150 s, -0.003141 s, ? 0.9644 mozilla-central x000_revs_xx00_added_0_copies 6100d773079a 04a55431795e : 0.007113 s, 0.007064 s, -0.000049 s, ? 0.9931 mozilla-central x000_revs_x000_added_x_copies 9f17a6fc04f9 2d37b966abed : 0.004687 s, 0.004741 s, +0.000054 s, ? 1.0115 mozilla-central x000_revs_x000_added_x000_copies 7c97034feb78 4407bd0c6330 : 0.198710 s, 0.190133 s, -0.008577 s, ? 0.9568 mozilla-central x0000_revs_xx000_added_0_copies 9eec5917337d 67118cc6dcad : 0.036068 s, 0.035651 s, -0.000417 s, ? 0.9884 mozilla-central x0000_revs_xx000_added_x000_copies f78c615a656c 96a38b690156 : 0.465362 s, 0.440694 s, -0.024668 s, ? 0.9470 mozilla-central x00000_revs_x0000_added_x0000_copies 6832ae71433c 4c222a1d9a00 : 24.519684 s, 18.454163 s, -6.065521 s, ? 0.7526 mozilla-central x00000_revs_x00000_added_x000_copies 76caed42cf7c 1daa622bbe42 : 42.711897 s, 31.562719 s, -11.149178 s, ? 0.7390 mozilla-try x_revs_x_added_0_copies aaf6dde0deb8 9790f499805a : 0.001201 s, 0.001189 s, -0.000012 s, ? 0.9900 mozilla-try x_revs_x000_added_0_copies d8d0222927b4 5bb8ce8c7450 : 0.001216 s, 0.001204 s, -0.000012 s, ? 0.9901 mozilla-try x_revs_x_added_x_copies 092fcca11bdb 936255a0384a : 0.000595 s, 0.000586 s, -0.000009 s, ? 0.9849 mozilla-try x_revs_x00_added_x_copies b53d2fadbdb5 017afae788ec : 0.001856 s, 0.001845 s, -0.000011 s, ? 0.9941 mozilla-try x_revs_x000_added_x000_copies 20408ad61ce5 6f0ee96e21ad : 0.064936 s, 0.063822 s, -0.001114 s, ? 0.9828 mozilla-try x_revs_x0000_added_x0000_copies effb563bb7e5 c07a39dc4e80 : 0.090601 s, 0.088038 s, -0.002563 s, ? 0.9717 mozilla-try x000_revs_xx00_added_0_copies 6100d773079a 04a55431795e : 0.007510 s, 0.007389 s, -0.000121 s, ? 0.9839 mozilla-try x000_revs_x000_added_x_copies 9f17a6fc04f9 2d37b966abed : 0.004911 s, 0.004868 s, -0.000043 s, ? 0.9912 mozilla-try x000_revs_x000_added_x000_copies 1346fd0130e4 4c65cbdabc1f : 0.233231 s, 0.222450 s, -0.010781 s, ? 0.9538 mozilla-try x0000_revs_x_added_0_copies 63519bfd42ee a36a2a865d92 : 0.419989 s, 0.370675 s, -0.049314 s, ? 0.8826 mozilla-try x0000_revs_x_added_x_copies 9fe69ff0762d bcabf2a78927 : 0.401521 s, 0.358020 s, -0.043501 s, ? 0.8917 mozilla-try x0000_revs_xx000_added_x_copies 156f6e2674f2 4d0f2c178e66 : 0.179555 s, 0.145235 s, -0.034320 s, ? 0.8089 mozilla-try x0000_revs_xx000_added_0_copies 9eec5917337d 67118cc6dcad : 0.038004 s, 0.037606 s, -0.000398 s, ? 0.9895 mozilla-try x0000_revs_xx000_added_x000_copies 89294cd501d9 7ccb2fc7ccb5 : 52.838482 s, 7.382439 s, -45.456043 s, ? 0.1397 mozilla-try x0000_revs_x0000_added_x0000_copies e928c65095ed e951f4ad123a : 8.705874 s, 7.273506 s, -1.432368 s, ? 0.8355 mozilla-try x00000_revs_x00000_added_0_copies dc8a3ca7010e d16fde900c9c : 1.126708 s, 1.074593 s, -0.052115 s, ? 0.9537 mozilla-try x00000_revs_x0000_added_x0000_copies 8d3fafa80d4b eb884023b810 : 83.854020 s, 27.746195 s, -56.107825 s, ? 0.3309 Below is a table comparing the runtime of the current "filelog centric" algorithm, with the "changeset centric" one, we just modified. The changeset centric algorithm is a significant win in many scenario, but they are still various cases where it is quite slower. When many revision has to be considered the cost of retrieving the copy information, creating new dictionaries, merging dictionaries and checking if revision are ancestors of each other can slow things down. The rest of this series, will introduce a rust version of the copy tracing code to deal with most of theses issues. Repo Case Source-Rev Dest-Rev filelog sidedata Difference Factor --------------------------------------------------------------------------------------------------------------------------------------- mercurial x_revs_x_added_0_copies ad6b123de1c7 39cfcef4f463 : 0.000914 s, 0.000044 s, - 0.000870 s, ? 0.048140 mercurial x_revs_x_added_x_copies 2b1c78674230 0c1d10351869 : 0.001812 s, 0.000138 s, - 0.001674 s, ? 0.076159 mercurial x000_revs_x000_added_x_copies 81f8ff2a9bf2 dd3267698d84 : 0.017954 s, 0.005052 s, - 0.012902 s, ? 0.281386 pypy x_revs_x_added_0_copies aed021ee8ae8 099ed31b181b : 0.001509 s, 0.000219 s, - 0.001290 s, ? 0.145129 pypy x_revs_x000_added_0_copies 4aa4e1f8e19a 359343b9ac0e : 0.206881 s, 0.000055 s, - 0.206826 s, ? 0.000266 pypy x_revs_x_added_x_copies ac52eb7bbbb0 72e022663155 : 0.016951 s, 0.000128 s, - 0.016823 s, ? 0.007551 pypy x_revs_x00_added_x_copies c3b14617fbd7 ace7255d9a26 : 0.019096 s, 0.001089 s, - 0.018007 s, ? 0.057028 pypy x_revs_x000_added_x000_copies df6f7a526b60 a83dc6a2d56f : 0.762506 s, 0.017407 s, - 0.745099 s, ? 0.022829 pypy x000_revs_xx00_added_0_copies 89a76aede314 2f22446ff07e : 1.179211 s, 0.094175 s, - 1.085036 s, ? 0.079863 pypy x000_revs_x000_added_x_copies 8a3b5bfd266e 2c68e87c3efe : 1.249058 s, 0.238009 s, - 1.011049 s, ? 0.190551 pypy x000_revs_x000_added_x000_copies 89a76aede314 7b3dda341c84 : 1.614107 s, 0.125876 s, - 1.488231 s, ? 0.077985 pypy x0000_revs_x_added_0_copies d1defd0dc478 c9cb1334cc78 : 0.001064 s, 3.581556 s, + 3.580492 s, ? 3366.124060 pypy x0000_revs_xx000_added_0_copies bf2c629d0071 4ffed77c095c : 1.061275 s, 0.016721 s, - 1.044554 s, ? 0.015756 pypy x0000_revs_xx000_added_x000_copies 08ea3258278e d9fa043f30c0 : 1.341119 s, 0.242367 s, - 1.098752 s, ? 0.180720 netbeans x_revs_x_added_0_copies fb0955ffcbcd a01e9239f9e7 : 0.027803 s, 0.000165 s, - 0.027638 s, ? 0.005935 netbeans x_revs_x000_added_0_copies 6f360122949f 20eb231cc7d0 : 0.130014 s, 0.000114 s, - 0.129900 s, ? 0.000877 netbeans x_revs_x_added_x_copies 1ada3faf6fb6 5a39d12eecf4 : 0.024990 s, 0.000296 s, - 0.024694 s, ? 0.011845 netbeans x_revs_x00_added_x_copies 35be93ba1e2c 9eec5e90c05f : 0.052201 s, 0.001124 s, - 0.051077 s, ? 0.021532 netbeans x000_revs_xx00_added_0_copies eac3045b4fdd 51d4ae7f1290 : 0.037642 s, 0.013060 s, - 0.024582 s, ? 0.346953 netbeans x000_revs_x000_added_x_copies e2063d266acd 6081d72689dc : 0.197086 s, 0.017112 s, - 0.179974 s, ? 0.086825 netbeans x000_revs_x000_added_x000_copies ff453e9fee32 411350406ec2 : 0.935148 s, 0.660350 s, - 0.274798 s, ? 0.706145 netbeans x0000_revs_xx000_added_x000_copies 588c2d1ced70 1aad62e59ddd : 3.920674 s, 10.032499 s, + 6.111825 s, ? 2.558871 mozilla-central x_revs_x_added_0_copies 3697f962bb7b 7015fcdd43a2 : 0.024232 s, 0.000189 s, - 0.024043 s, ? 0.007800 mozilla-central x_revs_x000_added_0_copies dd390860c6c9 40d0c5bed75d : 0.141483 s, 0.000462 s, - 0.141021 s, ? 0.003265 mozilla-central x_revs_x_added_x_copies 8d198483ae3b 14207ffc2b2f : 0.025775 s, 0.000270 s, - 0.025505 s, ? 0.010475 mozilla-central x_revs_x00_added_x_copies 98cbc58cc6bc 446a150332c3 : 0.084922 s, 0.001474 s, - 0.083448 s, ? 0.017357 mozilla-central x_revs_x000_added_x000_copies 3c684b4b8f68 0a5e72d1b479 : 0.194784 s, 0.004806 s, - 0.189978 s, ? 0.024673 mozilla-central x_revs_x0000_added_x0000_copies effb563bb7e5 c07a39dc4e80 : 2.161103 s, 0.085150 s, - 2.075953 s, ? 0.039401 mozilla-central x000_revs_xx00_added_0_copies 6100d773079a 04a55431795e : 0.089347 s, 0.007064 s, - 0.082283 s, ? 0.079063 mozilla-central x000_revs_x000_added_x_copies 9f17a6fc04f9 2d37b966abed : 0.732171 s, 0.004741 s, - 0.727430 s, ? 0.006475 mozilla-central x000_revs_x000_added_x000_copies 7c97034feb78 4407bd0c6330 : 1.157287 s, 0.190133 s, - 0.967154 s, ? 0.164292 mozilla-central x0000_revs_xx000_added_0_copies 9eec5917337d 67118cc6dcad : 6.726568 s, 0.035651 s, - 6.690917 s, ? 0.005300 mozilla-central x0000_revs_xx000_added_x000_copies f78c615a656c 96a38b690156 : 3.266229 s, 0.440694 s, - 2.825535 s, ? 0.134924 mozilla-central x00000_revs_x0000_added_x0000_copies 6832ae71433c 4c222a1d9a00 : 15.860534 s, 18.454163 s, + 2.593629 s, ? 1.163527 mozilla-central x00000_revs_x00000_added_x000_copies 76caed42cf7c 1daa622bbe42 : 20.450475 s, 31.562719 s, +11.112244 s, ? 1.543373 mozilla-try x_revs_x_added_0_copies aaf6dde0deb8 9790f499805a : 0.080442 s, 0.001189 s, - 0.079253 s, ? 0.014781 mozilla-try x_revs_x000_added_0_copies d8d0222927b4 5bb8ce8c7450 : 0.497672 s, 0.001204 s, - 0.496468 s, ? 0.002419 mozilla-try x_revs_x_added_x_copies 092fcca11bdb 936255a0384a : 0.021183 s, 0.000586 s, - 0.020597 s, ? 0.027664 mozilla-try x_revs_x00_added_x_copies b53d2fadbdb5 017afae788ec : 0.230991 s, 0.001845 s, - 0.229146 s, ? 0.007987 mozilla-try x_revs_x000_added_x000_copies 20408ad61ce5 6f0ee96e21ad : 1.118461 s, 0.063822 s, - 1.054639 s, ? 0.057062 mozilla-try x_revs_x0000_added_x0000_copies effb563bb7e5 c07a39dc4e80 : 2.206083 s, 0.088038 s, - 2.118045 s, ? 0.039907 mozilla-try x000_revs_xx00_added_0_copies 6100d773079a 04a55431795e : 0.089404 s, 0.007389 s, - 0.082015 s, ? 0.082647 mozilla-try x000_revs_x000_added_x_copies 9f17a6fc04f9 2d37b966abed : 0.733043 s, 0.004868 s, - 0.728175 s, ? 0.006641 mozilla-try x000_revs_x000_added_x000_copies 1346fd0130e4 4c65cbdabc1f : 1.163367 s, 0.222450 s, - 0.940917 s, ? 0.191212 mozilla-try x0000_revs_x_added_0_copies 63519bfd42ee a36a2a865d92 : 0.085456 s, 0.370675 s, + 0.285219 s, ? 4.337612 mozilla-try x0000_revs_x_added_x_copies 9fe69ff0762d bcabf2a78927 : 0.083601 s, 0.358020 s, + 0.274419 s, ? 4.282485 mozilla-try x0000_revs_xx000_added_x_copies 156f6e2674f2 4d0f2c178e66 : 7.366614 s, 0.145235 s, - 7.221379 s, ? 0.019715 mozilla-try x0000_revs_xx000_added_0_copies 9eec5917337d 67118cc6dcad : 6.664464 s, 0.037606 s, - 6.626858 s, ? 0.005643 mozilla-try x0000_revs_xx000_added_x000_copies 89294cd501d9 7ccb2fc7ccb5 : 7.467836 s, 7.382439 s, - 0.085397 s, ? 0.988565 mozilla-try x0000_revs_x0000_added_x0000_copies e928c65095ed e951f4ad123a : 9.801294 s, 7.273506 s, - 2.527788 s, ? 0.742097 mozilla-try x00000_revs_x_added_0_copies 6a320851d377 1ebb79acd503 : 0.091886 s, killed mozilla-try x00000_revs_x00000_added_0_copies dc8a3ca7010e d16fde900c9c : 26.491140 s, 1.074593 s, -25.416547 s, ? 0.040564 mozilla-try x00000_revs_x_added_x_copies 5173c4b6f97c 95d83ee7242d : 0.092863 s, killed mozilla-try x00000_revs_x000_added_x_copies 9126823d0e9c ca82787bb23c : 0.226823 s, killed mozilla-try x00000_revs_x0000_added_x0000_copies 8d3fafa80d4b eb884023b810 : 18.914630 s, 27.746195 s, + 8.831565 s, ? 1.466917 mozilla-try x00000_revs_x00000_added_x0000_copies 1b661134e2ca 1ae03d022d6d : 21.198903 s, killed mozilla-try x00000_revs_x00000_added_x000_copies 9b2a99adc05e 8e29777b48e6 : 24.952268 s, killed Differential Revision: https://phab.mercurial-scm.org/D9296

author	Pierre-Yves David <pierre-yves.david@octobus.net>
date	Mon, 02 Nov 2020 11:03:56 +0100
parents	9f70512ae2cf
children	89a2afe31e82

rev	line source
42041 3e47d1ec9da5 util: extract compression code in `mercurial.utils.compression` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41834 diff changeset	1 # compression.py - Mercurial utility functions for compression
8226 8b2cd04a6e97 put license and copyright info into comment blocks Martin Geisler <mg@lazybytes.net> parents: 8225 diff changeset	2 #
8b2cd04a6e97 put license and copyright info into comment blocks Martin Geisler <mg@lazybytes.net> parents: 8225 diff changeset	3 # This software may be used and distributed according to the terms of the
10263 25e572394f5c Update license to GPLv2+ Matt Mackall <mpm@selenic.com> parents: 9996 diff changeset	4 # GNU General Public License version 2 or any later version.
1082 ce96e316278a Update util.py docstrings, fix walk test mpm@selenic.com parents: 1081 diff changeset	5
419 28511fc21073 [PATCH] file seperator handling for the other 'OS' mpm@selenic.com parents: diff changeset	6
34137 a8994d08e4a2 doctest: use print_function and convert bytes to unicode where needed Yuya Nishihara <yuya@tcha.org> parents: 34134 diff changeset	7 from __future__ import absolute_import, print_function
27358 ac839ee45b6a util: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27357 diff changeset	8
ac839ee45b6a util: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27357 diff changeset	9 import bz2
ac839ee45b6a util: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27357 diff changeset	10 import collections
26266 1e042e31bd0c changegroup: move all compressions utilities in util Pierre-Yves David <pierre-yves.david@fb.com> parents: 26201 diff changeset	11 import zlib
3769 96095d9ff1f8 Add encoding detection Matt Mackall <mpm@selenic.com> parents: 3767 diff changeset	12
43089 c59eb1560c44 py3: manually import getattr where it is needed Gregory Szorc <gregory.szorc@gmail.com> parents: 43077 diff changeset	13 from ..pycompat import getattr
42041 3e47d1ec9da5 util: extract compression code in `mercurial.utils.compression` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41834 diff changeset	14 from .. import (
27358 ac839ee45b6a util: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27357 diff changeset	15 error,
ac839ee45b6a util: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27357 diff changeset	16 i18n,
28818 6041fb8f2da8 pycompat: add empty and queue to handle py3 divergence timeless <timeless@mozdev.org> parents: 28497 diff changeset	17 pycompat,
27358 ac839ee45b6a util: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27357 diff changeset	18 )
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42044 diff changeset	19 from . import stringutil
3769 96095d9ff1f8 Add encoding detection Matt Mackall <mpm@selenic.com> parents: 3767 diff changeset	20
42041 3e47d1ec9da5 util: extract compression code in `mercurial.utils.compression` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41834 diff changeset	21 safehasattr = pycompat.safehasattr
32201 4462a981e8df base85: proxy through util module Yuya Nishihara <yuya@tcha.org> parents: 32154 diff changeset	22
14926 4e7e63fc685a util: eliminate wildcard imports Adrian Buehlmann <adrian@cadifra.com> parents: 14918 diff changeset	23
27358 ac839ee45b6a util: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 27357 diff changeset	24 _ = i18n._
14926 4e7e63fc685a util: eliminate wildcard imports Adrian Buehlmann <adrian@cadifra.com> parents: 14918 diff changeset	25
42041 3e47d1ec9da5 util: extract compression code in `mercurial.utils.compression` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 41834 diff changeset	26 # compression code
36524 bfe38f787d5b util: add a file object proxy that can notify observers Gregory Szorc <gregory.szorc@gmail.com> parents: 36432 diff changeset	27
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	28 SERVERROLE = b'server'
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	29 CLIENTROLE = b'client'
33793 bbbbd3c30bfc util: add base class for transactional context managers Martin von Zweigbergk <martinvonz@google.com> parents: 33737 diff changeset	30
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42044 diff changeset	31 compewireprotosupport = collections.namedtuple(
43506 9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings Augie Fackler <augie@google.com> parents: 43475 diff changeset	32 'compenginewireprotosupport', ('name', 'serverpriority', 'clientpriority'),
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42044 diff changeset	33 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42044 diff changeset	34
9097 431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639] Matt Mackall <mpm@selenic.com> parents: 9089 diff changeset	35
8207 dd8d5be57d65 util: take propertycache from context.py Matt Mackall <mpm@selenic.com> parents: 8181 diff changeset	36 class propertycache(object):
dd8d5be57d65 util: take propertycache from context.py Matt Mackall <mpm@selenic.com> parents: 8181 diff changeset	37 def __init__(self, func):
dd8d5be57d65 util: take propertycache from context.py Matt Mackall <mpm@selenic.com> parents: 8181 diff changeset	38 self.func = func
dd8d5be57d65 util: take propertycache from context.py Matt Mackall <mpm@selenic.com> parents: 8181 diff changeset	39 self.name = func.__name__
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42044 diff changeset	40
8207 dd8d5be57d65 util: take propertycache from context.py Matt Mackall <mpm@selenic.com> parents: 8181 diff changeset	41 def __get__(self, obj, type=None):
dd8d5be57d65 util: take propertycache from context.py Matt Mackall <mpm@selenic.com> parents: 8181 diff changeset	42 result = self.func(obj)
18013 98c867ac1330 clfilter: add a propertycache that must be unfiltered Pierre-Yves David <pierre-yves.david@logilab.fr> parents: 17962 diff changeset	43 self.cachevalue(obj, result)
8207 dd8d5be57d65 util: take propertycache from context.py Matt Mackall <mpm@selenic.com> parents: 8181 diff changeset	44 return result
dd8d5be57d65 util: take propertycache from context.py Matt Mackall <mpm@selenic.com> parents: 8181 diff changeset	45
18013 98c867ac1330 clfilter: add a propertycache that must be unfiltered Pierre-Yves David <pierre-yves.david@logilab.fr> parents: 17962 diff changeset	46 def cachevalue(self, obj, value):
19951 d51c4d85ec23 spelling: random spell checker fixes Mads Kiilerich <madski@unity3d.com> parents: 19852 diff changeset	47 # __dict__ assignment required to bypass __setattr__ (eg: repoview)
19845 a1237a4b437d repoview: make propertycache.setcache compatible with repoview Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 19461 diff changeset	48 obj.__dict__[self.name] = value
18013 98c867ac1330 clfilter: add a propertycache that must be unfiltered Pierre-Yves David <pierre-yves.david@logilab.fr> parents: 17962 diff changeset	49
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42044 diff changeset	50
30350 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	51 class compressormanager(object):
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	52 """Holds registrations of various compression engines.
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	53
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	54 This class essentially abstracts the differences between compression
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	55 engines to allow new compression formats to be added easily, possibly from
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	56 extensions.
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	57
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	58 Compressors are registered against the global instance by calling its
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	59 ``register()`` method.
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	60 """
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42044 diff changeset	61
30350 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	62 def __init__(self):
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	63 self._engines = {}
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	64 # Bundle spec human name to engine name.
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	65 self._bundlenames = {}
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	66 # Internal bundle identifier to engine name.
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	67 self._bundletypes = {}
30798 f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	68 # Revlog header to engine name.
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	69 self._revlogheaders = {}
30761 7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	70 # Wire proto identifier to engine name.
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	71 self._wiretypes = {}
30350 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	72
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	73 def __getitem__(self, key):
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	74 return self._engines[key]
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	75
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	76 def __contains__(self, key):
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	77 return key in self._engines
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	78
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	79 def __iter__(self):
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	80 return iter(self._engines.keys())
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	81
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	82 def register(self, engine):
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	83 """Register a compression engine with the manager.
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	84
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	85 The argument must be a ``compressionengine`` instance.
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	86 """
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	87 if not isinstance(engine, compressionengine):
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	88 raise ValueError(_(b'argument must be a compressionengine'))
30350 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	89
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	90 name = engine.name()
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	91
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	92 if name in self._engines:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42044 diff changeset	93 raise error.Abort(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	94 _(b'compression engine %s already registered') % name
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42044 diff changeset	95 )
30350 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	96
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	97 bundleinfo = engine.bundletype()
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	98 if bundleinfo:
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	99 bundlename, bundletype = bundleinfo
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	100
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	101 if bundlename in self._bundlenames:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42044 diff changeset	102 raise error.Abort(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	103 _(b'bundle name %s already registered') % bundlename
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42044 diff changeset	104 )
30350 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	105 if bundletype in self._bundletypes:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42044 diff changeset	106 raise error.Abort(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	107 _(b'bundle type %s already registered by %s')
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42044 diff changeset	108 % (bundletype, self._bundletypes[bundletype])
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42044 diff changeset	109 )
30350 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	110
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	111 # No external facing name declared.
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	112 if bundlename:
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	113 self._bundlenames[bundlename] = name
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	114
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	115 self._bundletypes[bundletype] = name
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	116
30761 7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	117 wiresupport = engine.wireprotosupport()
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	118 if wiresupport:
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	119 wiretype = wiresupport.name
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	120 if wiretype in self._wiretypes:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42044 diff changeset	121 raise error.Abort(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42044 diff changeset	122 _(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	123 b'wire protocol compression %s already '
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	124 b'registered by %s'
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42044 diff changeset	125 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42044 diff changeset	126 % (wiretype, self._wiretypes[wiretype])
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42044 diff changeset	127 )
30761 7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	128
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	129 self._wiretypes[wiretype] = name
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	130
30798 f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	131 revlogheader = engine.revlogheader()
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	132 if revlogheader and revlogheader in self._revlogheaders:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42044 diff changeset	133 raise error.Abort(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	134 _(b'revlog header %s already registered by %s')
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42044 diff changeset	135 % (revlogheader, self._revlogheaders[revlogheader])
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42044 diff changeset	136 )
30798 f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	137
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	138 if revlogheader:
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	139 self._revlogheaders[revlogheader] = name
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	140
30350 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	141 self._engines[name] = engine
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	142
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	143 @property
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	144 def supportedbundlenames(self):
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	145 return set(self._bundlenames.keys())
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	146
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	147 @property
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	148 def supportedbundletypes(self):
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	149 return set(self._bundletypes.keys())
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	150
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	151 def forbundlename(self, bundlename):
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	152 """Obtain a compression engine registered to a bundle name.
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	153
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	154 Will raise KeyError if the bundle type isn't registered.
30438 90933e4e44fd util: check for compression engine availability before returning Gregory Szorc <gregory.szorc@gmail.com> parents: 30437 diff changeset	155
90933e4e44fd util: check for compression engine availability before returning Gregory Szorc <gregory.szorc@gmail.com> parents: 30437 diff changeset	156 Will abort if the engine is known but not available.
30350 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	157 """
30438 90933e4e44fd util: check for compression engine availability before returning Gregory Szorc <gregory.szorc@gmail.com> parents: 30437 diff changeset	158 engine = self._engines[self._bundlenames[bundlename]]
90933e4e44fd util: check for compression engine availability before returning Gregory Szorc <gregory.szorc@gmail.com> parents: 30437 diff changeset	159 if not engine.available():
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42044 diff changeset	160 raise error.Abort(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	161 _(b'compression engine %s could not be loaded') % engine.name()
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42044 diff changeset	162 )
30438 90933e4e44fd util: check for compression engine availability before returning Gregory Szorc <gregory.szorc@gmail.com> parents: 30437 diff changeset	163 return engine
30350 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	164
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	165 def forbundletype(self, bundletype):
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	166 """Obtain a compression engine registered to a bundle type.
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	167
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	168 Will raise KeyError if the bundle type isn't registered.
30438 90933e4e44fd util: check for compression engine availability before returning Gregory Szorc <gregory.szorc@gmail.com> parents: 30437 diff changeset	169
90933e4e44fd util: check for compression engine availability before returning Gregory Szorc <gregory.szorc@gmail.com> parents: 30437 diff changeset	170 Will abort if the engine is known but not available.
30350 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	171 """
30438 90933e4e44fd util: check for compression engine availability before returning Gregory Szorc <gregory.szorc@gmail.com> parents: 30437 diff changeset	172 engine = self._engines[self._bundletypes[bundletype]]
90933e4e44fd util: check for compression engine availability before returning Gregory Szorc <gregory.szorc@gmail.com> parents: 30437 diff changeset	173 if not engine.available():
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42044 diff changeset	174 raise error.Abort(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	175 _(b'compression engine %s could not be loaded') % engine.name()
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42044 diff changeset	176 )
30438 90933e4e44fd util: check for compression engine availability before returning Gregory Szorc <gregory.szorc@gmail.com> parents: 30437 diff changeset	177 return engine
30350 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	178
30761 7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	179 def supportedwireengines(self, role, onlyavailable=True):
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	180 """Obtain compression engines that support the wire protocol.
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	181
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	182 Returns a list of engines in prioritized order, most desired first.
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	183
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	184 If ``onlyavailable`` is set, filter out engines that can't be
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	185 loaded.
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	186 """
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	187 assert role in (SERVERROLE, CLIENTROLE)
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	188
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	189 attr = b'serverpriority' if role == SERVERROLE else b'clientpriority'
30761 7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	190
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	191 engines = [self._engines[e] for e in self._wiretypes.values()]
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	192 if onlyavailable:
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	193 engines = [e for e in engines if e.available()]
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	194
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	195 def getkey(e):
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	196 # Sort first by priority, highest first. In case of tie, sort
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	197 # alphabetically. This is arbitrary, but ensures output is
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	198 # stable.
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	199 w = e.wireprotosupport()
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	200 return -1 * getattr(w, attr), w.name
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	201
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	202 return list(sorted(engines, key=getkey))
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	203
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	204 def forwiretype(self, wiretype):
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	205 engine = self._engines[self._wiretypes[wiretype]]
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	206 if not engine.available():
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42044 diff changeset	207 raise error.Abort(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	208 _(b'compression engine %s could not be loaded') % engine.name()
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42044 diff changeset	209 )
30761 7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	210 return engine
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	211
30798 f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	212 def forrevlogheader(self, header):
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	213 """Obtain a compression engine registered to a revlog header.
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	214
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	215 Will raise KeyError if the revlog header value isn't registered.
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	216 """
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	217 return self._engines[self._revlogheaders[header]]
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	218
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42044 diff changeset	219
30350 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	220 compengines = compressormanager()
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	221
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42044 diff changeset	222
30350 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	223 class compressionengine(object):
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	224 """Base class for compression engines.
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	225
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	226 Compression engines must implement the interface defined by this class.
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	227 """
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42044 diff changeset	228
30350 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	229 def name(self):
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	230 """Returns the name of the compression engine.
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	231
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	232 This is the key the engine is registered under.
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	233
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	234 This method must be implemented.
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	235 """
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	236 raise NotImplementedError()
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	237
30437 64d7275445d0 util: expose an "available" API on compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30418 diff changeset	238 def available(self):
64d7275445d0 util: expose an "available" API on compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30418 diff changeset	239 """Whether the compression engine is available.
64d7275445d0 util: expose an "available" API on compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30418 diff changeset	240
64d7275445d0 util: expose an "available" API on compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30418 diff changeset	241 The intent of this method is to allow optional compression engines
64d7275445d0 util: expose an "available" API on compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30418 diff changeset	242 that may not be available in all installations (such as engines relying
64d7275445d0 util: expose an "available" API on compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30418 diff changeset	243 on C extensions that may not be present).
64d7275445d0 util: expose an "available" API on compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30418 diff changeset	244 """
64d7275445d0 util: expose an "available" API on compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30418 diff changeset	245 return True
64d7275445d0 util: expose an "available" API on compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30418 diff changeset	246
30350 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	247 def bundletype(self):
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	248 """Describes bundle identifiers for this engine.
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	249
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	250 If this compression engine isn't supported for bundles, returns None.
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	251
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	252 If this engine can be used for bundles, returns a 2-tuple of strings of
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	253 the user-facing "bundle spec" compression name and an internal
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	254 identifier used to denote the compression format within bundles. To
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	255 exclude the name from external usage, set the first element to ``None``.
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	256
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	257 If bundle compression is supported, the class must also implement
30359 673f0fdc1046 util: remove compressorobj API from compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30356 diff changeset	258 ``compressstream`` and `decompressorreader``.
31792 55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	259
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	260 The docstring of this method is used in the help system to tell users
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	261 about this engine.
30350 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	262 """
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	263 return None
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	264
30761 7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	265 def wireprotosupport(self):
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	266 """Declare support for this compression format on the wire protocol.
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	267
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	268 If this compression engine isn't supported for compressing wire
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	269 protocol payloads, returns None.
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	270
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	271 Otherwise, returns ``compenginewireprotosupport`` with the following
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	272 fields:
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	273
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	274 * String format identifier
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	275 * Integer priority for the server
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	276 * Integer priority for the client
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	277
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	278 The integer priorities are used to order the advertisement of format
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	279 support by server and client. The highest integer is advertised
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	280 first. Integers with non-positive values aren't advertised.
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	281
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	282 The priority values are somewhat arbitrary and only used for default
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	283 ordering. The relative order can be changed via config options.
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	284
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	285 If wire protocol compression is supported, the class must also implement
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	286 ``compressstream`` and ``decompressorreader``.
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	287 """
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	288 return None
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	289
30798 f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	290 def revlogheader(self):
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	291 """Header added to revlog chunks that identifies this engine.
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	292
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	293 If this engine can be used to compress revlogs, this method should
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	294 return the bytes used to identify chunks compressed with this engine.
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	295 Else, the method should return ``None`` to indicate it does not
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	296 participate in revlog compression.
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	297 """
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	298 return None
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	299
30356 c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	300 def compressstream(self, it, opts=None):
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	301 """Compress an iterator of chunks.
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	302
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	303 The method receives an iterator (ideally a generator) of chunks of
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	304 bytes to be compressed. It returns an iterator (ideally a generator)
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	305 of bytes of chunks representing the compressed output.
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	306
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	307 Optionally accepts an argument defining how to perform compression.
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	308 Each engine treats this argument differently.
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	309 """
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	310 raise NotImplementedError()
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	311
30350 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	312 def decompressorreader(self, fh):
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	313 """Perform decompression on a file object.
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	314
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	315 Argument is an object with a ``read(size)`` method that returns
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	316 compressed data. Return value is an object with a ``read(size)`` that
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	317 returns uncompressed data.
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	318 """
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	319 raise NotImplementedError()
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	320
30794 31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	321 def revlogcompressor(self, opts=None):
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	322 """Obtain an object that can be used to compress revlog entries.
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	323
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	324 The object has a ``compress(data)`` method that compresses binary
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	325 data. This method returns compressed binary data or ``None`` if
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	326 the data could not be compressed (too small, not compressible, etc).
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	327 The returned data should have a header uniquely identifying this
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	328 compression format so decompression can be routed to this engine.
30798 f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	329 This header should be identified by the ``revlogheader()`` return
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	330 value.
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	331
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	332 The object has a ``decompress(data)`` method that decompresses
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	333 data. The method will only be called if ``data`` begins with
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	334 ``revlogheader()``. The method should return the raw, uncompressed
39777 b63dee7bd0d9 global: replace most uses of RevlogError with StorageError (API) Gregory Szorc <gregory.szorc@gmail.com> parents: 39571 diff changeset	335 data or raise a ``StorageError``.
30794 31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	336
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	337 The object is reusable but is not thread safe.
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	338 """
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	339 raise NotImplementedError()
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	340
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42044 diff changeset	341
38713 27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	342 class _CompressedStreamReader(object):
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	343 def __init__(self, fh):
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	344 if safehasattr(fh, 'unbufferedread'):
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	345 self._reader = fh.unbufferedread
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	346 else:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	347 self._reader = fh.read
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	348 self._pending = []
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	349 self._pos = 0
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	350 self._eof = False
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	351
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	352 def _decompress(self, chunk):
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	353 raise NotImplementedError()
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	354
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	355 def read(self, l):
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	356 buf = []
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	357 while True:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	358 while self._pending:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	359 if len(self._pending[0]) > l + self._pos:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	360 newbuf = self._pending[0]
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42044 diff changeset	361 buf.append(newbuf[self._pos : self._pos + l])
38713 27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	362 self._pos += l
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	363 return b''.join(buf)
38713 27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	364
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	365 newbuf = self._pending.pop(0)
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	366 if self._pos:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42044 diff changeset	367 buf.append(newbuf[self._pos :])
38713 27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	368 l -= len(newbuf) - self._pos
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	369 else:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	370 buf.append(newbuf)
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	371 l -= len(newbuf)
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	372 self._pos = 0
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	373
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	374 if self._eof:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	375 return b''.join(buf)
38713 27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	376 chunk = self._reader(65536)
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	377 self._decompress(chunk)
39209 1af95139e5ec util: improve handling of truncated compressed streams Joerg Sonnenberger <joerg@bec.de> parents: 39060 diff changeset	378 if not chunk and not self._pending and not self._eof:
1af95139e5ec util: improve handling of truncated compressed streams Joerg Sonnenberger <joerg@bec.de> parents: 39060 diff changeset	379 # No progress and no new data, bail out
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	380 return b''.join(buf)
38713 27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	381
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42044 diff changeset	382
38713 27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	383 class _GzipCompressedStreamReader(_CompressedStreamReader):
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	384 def __init__(self, fh):
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	385 super(_GzipCompressedStreamReader, self).__init__(fh)
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	386 self._decompobj = zlib.decompressobj()
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42044 diff changeset	387
38713 27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	388 def _decompress(self, chunk):
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	389 newbuf = self._decompobj.decompress(chunk)
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	390 if newbuf:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	391 self._pending.append(newbuf)
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	392 d = self._decompobj.copy()
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	393 try:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	394 d.decompress(b'x')
38713 27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	395 d.flush()
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	396 if d.unused_data == b'x':
38713 27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	397 self._eof = True
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	398 except zlib.error:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	399 pass
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	400
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42044 diff changeset	401
38713 27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	402 class _BZ2CompressedStreamReader(_CompressedStreamReader):
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	403 def __init__(self, fh):
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	404 super(_BZ2CompressedStreamReader, self).__init__(fh)
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	405 self._decompobj = bz2.BZ2Decompressor()
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42044 diff changeset	406
38713 27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	407 def _decompress(self, chunk):
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	408 newbuf = self._decompobj.decompress(chunk)
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	409 if newbuf:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	410 self._pending.append(newbuf)
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	411 try:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	412 while True:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	413 newbuf = self._decompobj.decompress(b'')
38713 27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	414 if newbuf:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	415 self._pending.append(newbuf)
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	416 else:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	417 break
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	418 except EOFError:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	419 self._eof = True
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	420
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42044 diff changeset	421
38713 27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	422 class _TruncatedBZ2CompressedStreamReader(_BZ2CompressedStreamReader):
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	423 def __init__(self, fh):
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	424 super(_TruncatedBZ2CompressedStreamReader, self).__init__(fh)
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	425 newbuf = self._decompobj.decompress(b'BZ')
38713 27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	426 if newbuf:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	427 self._pending.append(newbuf)
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	428
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42044 diff changeset	429
38713 27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	430 class _ZstdCompressedStreamReader(_CompressedStreamReader):
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	431 def __init__(self, fh, zstd):
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	432 super(_ZstdCompressedStreamReader, self).__init__(fh)
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	433 self._zstd = zstd
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	434 self._decompobj = zstd.ZstdDecompressor().decompressobj()
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42044 diff changeset	435
38713 27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	436 def _decompress(self, chunk):
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	437 newbuf = self._decompobj.decompress(chunk)
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	438 if newbuf:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	439 self._pending.append(newbuf)
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	440 try:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	441 while True:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	442 newbuf = self._decompobj.decompress(b'')
38713 27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	443 if newbuf:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	444 self._pending.append(newbuf)
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	445 else:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	446 break
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	447 except self._zstd.ZstdError:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	448 self._eof = True
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	449
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42044 diff changeset	450
30350 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	451 class _zlibengine(compressionengine):
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	452 def name(self):
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	453 return b'zlib'
30350 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	454
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	455 def bundletype(self):
31792 55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	456 """zlib compression using the DEFLATE algorithm.
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	457
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	458 All Mercurial clients should support this format. The compression
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	459 algorithm strikes a reasonable balance between compression ratio
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	460 and size.
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	461 """
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	462 return b'gzip', b'GZ'
30350 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	463
30761 7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	464 def wireprotosupport(self):
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	465 return compewireprotosupport(b'zlib', 20, 20)
30761 7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	466
30798 f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	467 def revlogheader(self):
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	468 return b'x'
30798 f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	469
30356 c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	470 def compressstream(self, it, opts=None):
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	471 opts = opts or {}
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	472
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	473 z = zlib.compressobj(opts.get(b'level', -1))
30356 c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	474 for chunk in it:
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	475 data = z.compress(chunk)
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	476 # Not all calls to compress emit data. It is cheaper to inspect
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	477 # here than to feed empty chunks through generator.
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	478 if data:
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	479 yield data
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	480
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	481 yield z.flush()
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	482
30350 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	483 def decompressorreader(self, fh):
38713 27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	484 return _GzipCompressedStreamReader(fh)
30350 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	485
30794 31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	486 class zlibrevlogcompressor(object):
42042 aaececb4b066 compression: accept level management for zlib compression Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42041 diff changeset	487 def __init__(self, level=None):
aaececb4b066 compression: accept level management for zlib compression Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42041 diff changeset	488 self._level = level
aaececb4b066 compression: accept level management for zlib compression Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42041 diff changeset	489
30794 31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	490 def compress(self, data):
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	491 insize = len(data)
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	492 # Caller handles empty input case.
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	493 assert insize > 0
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	494
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	495 if insize < 44:
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	496 return None
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	497
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	498 elif insize <= 1000000:
42042 aaececb4b066 compression: accept level management for zlib compression Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42041 diff changeset	499 if self._level is None:
aaececb4b066 compression: accept level management for zlib compression Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42041 diff changeset	500 compressed = zlib.compress(data)
aaececb4b066 compression: accept level management for zlib compression Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42041 diff changeset	501 else:
aaececb4b066 compression: accept level management for zlib compression Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42041 diff changeset	502 compressed = zlib.compress(data, self._level)
30794 31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	503 if len(compressed) < insize:
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	504 return compressed
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	505 return None
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	506
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	507 # zlib makes an internal copy of the input buffer, doubling
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	508 # memory usage for large inputs. So do streaming compression
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	509 # on large inputs.
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	510 else:
42042 aaececb4b066 compression: accept level management for zlib compression Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42041 diff changeset	511 if self._level is None:
aaececb4b066 compression: accept level management for zlib compression Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42041 diff changeset	512 z = zlib.compressobj()
aaececb4b066 compression: accept level management for zlib compression Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42041 diff changeset	513 else:
aaececb4b066 compression: accept level management for zlib compression Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42041 diff changeset	514 z = zlib.compressobj(level=self._level)
30794 31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	515 parts = []
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	516 pos = 0
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	517 while pos < insize:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42044 diff changeset	518 pos2 = pos + 2 ** 20
30794 31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	519 parts.append(z.compress(data[pos:pos2]))
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	520 pos = pos2
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	521 parts.append(z.flush())
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	522
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	523 if sum(map(len, parts)) < insize:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	524 return b''.join(parts)
30794 31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	525 return None
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	526
30798 f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	527 def decompress(self, data):
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	528 try:
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	529 return zlib.decompress(data)
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	530 except zlib.error as e:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42044 diff changeset	531 raise error.StorageError(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	532 _(b'revlog decompress error: %s')
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42044 diff changeset	533 % stringutil.forcebytestr(e)
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42044 diff changeset	534 )
30798 f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	535
30794 31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	536 def revlogcompressor(self, opts=None):
42043 1fac9b931d46 compression: introduce a `storage.revlog.zlib.level` configuration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42042 diff changeset	537 level = None
1fac9b931d46 compression: introduce a `storage.revlog.zlib.level` configuration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42042 diff changeset	538 if opts is not None:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	539 level = opts.get(b'zlib.level')
42043 1fac9b931d46 compression: introduce a `storage.revlog.zlib.level` configuration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42042 diff changeset	540 return self.zlibrevlogcompressor(level)
30794 31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	541
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42044 diff changeset	542
30350 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	543 compengines.register(_zlibengine())
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	544
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42044 diff changeset	545
30350 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	546 class _bz2engine(compressionengine):
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	547 def name(self):
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	548 return b'bz2'
30350 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	549
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	550 def bundletype(self):
31792 55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	551 """An algorithm that produces smaller bundles than ``gzip``.
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	552
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	553 All Mercurial clients should support this format.
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	554
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	555 This engine will likely produce smaller bundles than ``gzip`` but
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	556 will be significantly slower, both during compression and
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	557 decompression.
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	558
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	559 If available, the ``zstd`` engine can yield similar or better
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	560 compression at much higher speeds.
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	561 """
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	562 return b'bzip2', b'BZ'
30350 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	563
30761 7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	564 # We declare a protocol name but don't advertise by default because
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	565 # it is slow.
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	566 def wireprotosupport(self):
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	567 return compewireprotosupport(b'bzip2', 0, 0)
30761 7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	568
30356 c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	569 def compressstream(self, it, opts=None):
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	570 opts = opts or {}
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	571 z = bz2.BZ2Compressor(opts.get(b'level', 9))
30356 c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	572 for chunk in it:
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	573 data = z.compress(chunk)
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	574 if data:
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	575 yield data
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	576
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	577 yield z.flush()
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	578
30350 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	579 def decompressorreader(self, fh):
38713 27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	580 return _BZ2CompressedStreamReader(fh)
30350 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	581
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42044 diff changeset	582
30350 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	583 compengines.register(_bz2engine())
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	584
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42044 diff changeset	585
30350 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	586 class _truncatedbz2engine(compressionengine):
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	587 def name(self):
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	588 return b'bz2truncated'
30350 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	589
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	590 def bundletype(self):
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	591 return None, b'_truncatedBZ'
30350 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	592
30359 673f0fdc1046 util: remove compressorobj API from compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30356 diff changeset	593 # We don't implement compressstream because it is hackily handled elsewhere.
30350 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	594
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	595 def decompressorreader(self, fh):
38713 27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	596 return _TruncatedBZ2CompressedStreamReader(fh)
30350 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	597
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42044 diff changeset	598
30350 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	599 compengines.register(_truncatedbz2engine())
30265 6a8aff737a17 util: put compression code next to each other Gregory Szorc <gregory.szorc@gmail.com> parents: 30181 diff changeset	600
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42044 diff changeset	601
30350 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	602 class _noopengine(compressionengine):
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	603 def name(self):
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	604 return b'none'
30350 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	605
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	606 def bundletype(self):
31792 55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	607 """No compression is performed.
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	608
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	609 Use this compression engine to explicitly disable compression.
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	610 """
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	611 return b'none', b'UN'
30350 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	612
30761 7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	613 # Clients always support uncompressed payloads. Servers don't because
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	614 # unless you are on a fast network, uncompressed payloads can easily
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	615 # saturate your network pipe.
7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	616 def wireprotosupport(self):
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	617 return compewireprotosupport(b'none', 0, 10)
30761 7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	618
30798 f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	619 # We don't implement revlogheader because it is handled specially
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	620 # in the revlog class.
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	621
30356 c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	622 def compressstream(self, it, opts=None):
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	623 return it
c86109eface7 util: add a stream compression API to compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30355 diff changeset	624
30350 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	625 def decompressorreader(self, fh):
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	626 return fh
358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	627
30794 31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	628 class nooprevlogcompressor(object):
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	629 def compress(self, data):
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	630 return None
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	631
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	632 def revlogcompressor(self, opts=None):
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	633 return self.nooprevlogcompressor()
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	634
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42044 diff changeset	635
30350 358cda0af6ee util: create new abstraction for compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30332 diff changeset	636 compengines.register(_noopengine())
26266 1e042e31bd0c changegroup: move all compressions utilities in util Pierre-Yves David <pierre-yves.david@fb.com> parents: 26201 diff changeset	637
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42044 diff changeset	638
30442 41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	639 class _zstdengine(compressionengine):
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	640 def name(self):
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	641 return b'zstd'
30442 41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	642
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	643 @propertycache
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	644 def _module(self):
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	645 # Not all installs have the zstd module available. So defer importing
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	646 # until first access.
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	647 try:
43475 9ce76155e0da compression: tell pytype to not sweat a missing `zstd` module Augie Fackler <augie@google.com> parents: 43089 diff changeset	648 from .. import zstd # pytype: disable=import-error
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42044 diff changeset	649
30442 41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	650 # Force delayed import.
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	651 zstd.__version__
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	652 return zstd
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	653 except ImportError:
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	654 return None
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	655
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	656 def available(self):
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	657 return bool(self._module)
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	658
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	659 def bundletype(self):
31792 55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	660 """A modern compression algorithm that is fast and highly flexible.
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	661
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	662 Only supported by Mercurial 4.1 and newer clients.
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	663
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	664 With the default settings, zstd compression is both faster and yields
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	665 better compression than ``gzip``. It also frequently yields better
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	666 compression than ``bzip2`` while operating at much higher speeds.
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	667
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	668 If this engine is available and backwards compatibility is not a
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	669 concern, it is likely the best available engine.
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	670 """
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	671 return b'zstd', b'ZS'
30442 41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	672
30761 7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	673 def wireprotosupport(self):
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	674 return compewireprotosupport(b'zstd', 50, 50)
30761 7283719e2bfd util: declare wire protocol support of compression engines Gregory Szorc <gregory.szorc@gmail.com> parents: 30745 diff changeset	675
30798 f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	676 def revlogheader(self):
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	677 return b'\x28'
30798 f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	678
30442 41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	679 def compressstream(self, it, opts=None):
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	680 opts = opts or {}
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	681 # zstd level 3 is almost always significantly faster than zlib
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	682 # while providing no worse compression. It strikes a good balance
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	683 # between speed and compression.
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	684 level = opts.get(b'level', 3)
30442 41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	685
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	686 zstd = self._module
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	687 z = zstd.ZstdCompressor(level=level).compressobj()
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	688 for chunk in it:
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	689 data = z.compress(chunk)
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	690 if data:
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	691 yield data
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	692
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	693 yield z.flush()
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	694
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	695 def decompressorreader(self, fh):
38713 27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression Joerg Sonnenberger <joerg@bec.de> parents: 38575 diff changeset	696 return _ZstdCompressedStreamReader(fh, self._module)
30442 41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	697
30794 31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	698 class zstdrevlogcompressor(object):
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	699 def __init__(self, zstd, level=3):
37496 1765ed63db40 util: drop write_content_size=True Gregory Szorc <gregory.szorc@gmail.com> parents: 37455 diff changeset	700 # TODO consider omitting frame magic to save 4 bytes.
1765ed63db40 util: drop write_content_size=True Gregory Szorc <gregory.szorc@gmail.com> parents: 37455 diff changeset	701 # This writes content sizes into the frame header. That is
1765ed63db40 util: drop write_content_size=True Gregory Szorc <gregory.szorc@gmail.com> parents: 37455 diff changeset	702 # extra storage. But it allows a correct size memory allocation
1765ed63db40 util: drop write_content_size=True Gregory Szorc <gregory.szorc@gmail.com> parents: 37455 diff changeset	703 # to hold the result.
1765ed63db40 util: drop write_content_size=True Gregory Szorc <gregory.szorc@gmail.com> parents: 37455 diff changeset	704 self._cctx = zstd.ZstdCompressor(level=level)
30798 f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	705 self._dctx = zstd.ZstdDecompressor()
30794 31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	706 self._compinsize = zstd.COMPRESSION_RECOMMENDED_INPUT_SIZE
30798 f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	707 self._decompinsize = zstd.DECOMPRESSION_RECOMMENDED_INPUT_SIZE
30794 31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	708
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	709 def compress(self, data):
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	710 insize = len(data)
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	711 # Caller handles empty input case.
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	712 assert insize > 0
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	713
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	714 if insize < 50:
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	715 return None
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	716
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	717 elif insize <= 1000000:
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	718 compressed = self._cctx.compress(data)
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	719 if len(compressed) < insize:
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	720 return compressed
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	721 return None
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	722 else:
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	723 z = self._cctx.compressobj()
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	724 chunks = []
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	725 pos = 0
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	726 while pos < insize:
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	727 pos2 = pos + self._compinsize
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	728 chunk = z.compress(data[pos:pos2])
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	729 if chunk:
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	730 chunks.append(chunk)
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	731 pos = pos2
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	732 chunks.append(z.flush())
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	733
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	734 if sum(map(len, chunks)) < insize:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	735 return b''.join(chunks)
30794 31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	736 return None
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	737
30798 f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	738 def decompress(self, data):
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	739 insize = len(data)
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	740
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	741 try:
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	742 # This was measured to be faster than other streaming
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	743 # decompressors.
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	744 dobj = self._dctx.decompressobj()
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	745 chunks = []
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	746 pos = 0
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	747 while pos < insize:
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	748 pos2 = pos + self._decompinsize
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	749 chunk = dobj.decompress(data[pos:pos2])
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	750 if chunk:
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	751 chunks.append(chunk)
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	752 pos = pos2
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	753 # Frame should be exhausted, so no finish() API.
f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	754
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	755 return b''.join(chunks)
30798 f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	756 except Exception as e:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42044 diff changeset	757 raise error.StorageError(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	758 _(b'revlog decompress error: %s')
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42044 diff changeset	759 % stringutil.forcebytestr(e)
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42044 diff changeset	760 )
30798 f50c0db50025 util: compression APIs to support revlog decompression Gregory Szorc <gregory.szorc@gmail.com> parents: 30794 diff changeset	761
30794 31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	762 def revlogcompressor(self, opts=None):
31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	763 opts = opts or {}
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	764 level = opts.get(b'zstd.level')
42044 bb271ec2fbfb compression: introduce a `storage.revlog.zstd.level` configuration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42043 diff changeset	765 if level is None:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	766 level = opts.get(b'level')
42044 bb271ec2fbfb compression: introduce a `storage.revlog.zstd.level` configuration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42043 diff changeset	767 if level is None:
bb271ec2fbfb compression: introduce a `storage.revlog.zstd.level` configuration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42043 diff changeset	768 level = 3
bb271ec2fbfb compression: introduce a `storage.revlog.zstd.level` configuration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 42043 diff changeset	769 return self.zstdrevlogcompressor(self._module, level=level)
30794 31e1f0d4ab44 util: compression APIs to support revlog compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30773 diff changeset	770
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42044 diff changeset	771
30442 41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	772 compengines.register(_zstdengine())
41a8106789ca util: implement zstd compression engine Gregory Szorc <gregory.szorc@gmail.com> parents: 30438 diff changeset	773
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42044 diff changeset	774
31792 55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	775 def bundlecompressiontopics():
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	776 """Obtains a list of available bundle compressions for use in help."""
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	777 # help.makeitemsdocs() expects a dict of names to items with a .__doc__.
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	778 items = {}
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	779
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	780 # We need to format the docstring. So use a dummy object/type to hold it
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	781 # rather than mutating the original.
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	782 class docobject(object):
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	783 pass
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	784
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	785 for name in compengines:
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	786 engine = compengines[name]
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	787
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	788 if not engine.available():
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	789 continue
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	790
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	791 bt = engine.bundletype()
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	792 if not bt or not bt[0]:
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	793 continue
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	794
40251 3c89227788a2 py3: build help of compression engines in bytes Yuya Nishihara <yuya@tcha.org> parents: 40029 diff changeset	795 doc = b'``%s``\n %s' % (bt[0], pycompat.getdoc(engine.bundletype))
31792 55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	796
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	797 value = docobject()
40251 3c89227788a2 py3: build help of compression engines in bytes Yuya Nishihara <yuya@tcha.org> parents: 40029 diff changeset	798 value.__doc__ = pycompat.sysstr(doc)
33818 ed04d7254a91 i18n: use saved object to get actual function information if available FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 33799 diff changeset	799 value._origdoc = engine.bundletype.__doc__
ed04d7254a91 i18n: use saved object to get actual function information if available FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 33799 diff changeset	800 value._origfunc = engine.bundletype
31792 55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	801
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	802 items[bt[0]] = value
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	803
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	804 return items
55c0c91f55e6 util: document bundle compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31777 diff changeset	805
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42044 diff changeset	806
33820 fa7e30efe05a i18n: get translation entries for description of each compression engines FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 33818 diff changeset	807 i18nfunctions = bundlecompressiontopics().values()

Mercurial > public > mercurial-scm > hg

annotate mercurial/utils/compression.py @ 45892:06b64fabf91c