mercurial-scm/hg: mercurial/exchangev2.py annotate

annotate mercurial/exchangev2.py @ 45892:06b64fabf91c

copies: cache the ancestor checking call when tracing copy A good share of the time spent in this function is spent doing ancestors checking. To avoid spending time in duplicated call, we cache the result of calls. In the slower case, this provide a quite significant performance boost. Below are the result for a set of selected pairs (many of them pathological): (And further down is another table that summarize the current state of filelog based vs changeset base copy tracing) The benchmark have been configured to be killed after 6 minutes of runtime, which mean that any detect slower than 2 minutes will be marked as "killed". This drop some useful information about how much slower these case are? but also prevent 99% of the benchmark time to be spent on case that can be labelled "very slow" anyway. Repo Case Source-Rev Dest-Rev Old-Time New-Time Difference Factor ------------------------------------------------------------------------------------------------------------------------------------ mercurial x_revs_x_added_0_copies ad6b123de1c7 39cfcef4f463 : 0.000044 s, 0.000044 s, +0.000000 s, ? 1.0000 mercurial x_revs_x_added_x_copies 2b1c78674230 0c1d10351869 : 0.000138 s, 0.000138 s, +0.000000 s, ? 1.0000 mercurial x000_revs_x000_added_x_copies 81f8ff2a9bf2 dd3267698d84 : 0.005067 s, 0.005052 s, -0.000015 s, ? 0.9970 pypy x_revs_x_added_0_copies aed021ee8ae8 099ed31b181b : 0.000218 s, 0.000219 s, +0.000001 s, ? 1.0046 pypy x_revs_x000_added_0_copies 4aa4e1f8e19a 359343b9ac0e : 0.000053 s, 0.000055 s, +0.000002 s, ? 1.0377 pypy x_revs_x_added_x_copies ac52eb7bbbb0 72e022663155 : 0.000125 s, 0.000128 s, +0.000003 s, ? 1.0240 pypy x_revs_x00_added_x_copies c3b14617fbd7 ace7255d9a26 : 0.001098 s, 0.001089 s, -0.000009 s, ? 0.9918 pypy x_revs_x000_added_x000_copies df6f7a526b60 a83dc6a2d56f : 0.017546 s, 0.017407 s, -0.000139 s, ? 0.9921 pypy x000_revs_xx00_added_0_copies 89a76aede314 2f22446ff07e : 0.096723 s, 0.094175 s, -0.002548 s, ? 0.9737 pypy x000_revs_x000_added_x_copies 8a3b5bfd266e 2c68e87c3efe : 0.271796 s, 0.238009 s, -0.033787 s, ? 0.8757 pypy x000_revs_x000_added_x000_copies 89a76aede314 7b3dda341c84 : 0.128602 s, 0.125876 s, -0.002726 s, ? 0.9788 pypy x0000_revs_x_added_0_copies d1defd0dc478 c9cb1334cc78 : 7.086742 s, 3.581556 s, -3.505186 s, ? 0.5054 pypy x0000_revs_xx000_added_0_copies bf2c629d0071 4ffed77c095c : 0.016634 s, 0.016721 s, +0.000087 s, ? 1.0052 pypy x0000_revs_xx000_added_x000_copies 08ea3258278e d9fa043f30c0 : 0.254225 s, 0.242367 s, -0.011858 s, ? 0.9534 netbeans x_revs_x_added_0_copies fb0955ffcbcd a01e9239f9e7 : 0.000166 s, 0.000165 s, -0.000001 s, ? 0.9940 netbeans x_revs_x000_added_0_copies 6f360122949f 20eb231cc7d0 : 0.000118 s, 0.000114 s, -0.000004 s, ? 0.9661 netbeans x_revs_x_added_x_copies 1ada3faf6fb6 5a39d12eecf4 : 0.000296 s, 0.000296 s, +0.000000 s, ? 1.0000 netbeans x_revs_x00_added_x_copies 35be93ba1e2c 9eec5e90c05f : 0.001137 s, 0.001124 s, -0.000013 s, ? 0.9886 netbeans x000_revs_xx00_added_0_copies eac3045b4fdd 51d4ae7f1290 : 0.014133 s, 0.013060 s, -0.001073 s, ? 0.9241 netbeans x000_revs_x000_added_x_copies e2063d266acd 6081d72689dc : 0.016988 s, 0.017112 s, +0.000124 s, ? 1.0073 netbeans x000_revs_x000_added_x000_copies ff453e9fee32 411350406ec2 : 0.676361 s, 0.660350 s, -0.016011 s, ? 0.9763 netbeans x0000_revs_xx000_added_x000_copies 588c2d1ced70 1aad62e59ddd : 12.515149 s, 10.032499 s, -2.482650 s, ? 0.8016 mozilla-central x_revs_x_added_0_copies 3697f962bb7b 7015fcdd43a2 : 0.000186 s, 0.000189 s, +0.000003 s, ? 1.0161 mozilla-central x_revs_x000_added_0_copies dd390860c6c9 40d0c5bed75d : 0.000459 s, 0.000462 s, +0.000003 s, ? 1.0065 mozilla-central x_revs_x_added_x_copies 8d198483ae3b 14207ffc2b2f : 0.000273 s, 0.000270 s, -0.000003 s, ? 0.9890 mozilla-central x_revs_x00_added_x_copies 98cbc58cc6bc 446a150332c3 : 0.001503 s, 0.001474 s, -0.000029 s, ? 0.9807 mozilla-central x_revs_x000_added_x000_copies 3c684b4b8f68 0a5e72d1b479 : 0.004862 s, 0.004806 s, -0.000056 s, ? 0.9885 mozilla-central x_revs_x0000_added_x0000_copies effb563bb7e5 c07a39dc4e80 : 0.088291 s, 0.085150 s, -0.003141 s, ? 0.9644 mozilla-central x000_revs_xx00_added_0_copies 6100d773079a 04a55431795e : 0.007113 s, 0.007064 s, -0.000049 s, ? 0.9931 mozilla-central x000_revs_x000_added_x_copies 9f17a6fc04f9 2d37b966abed : 0.004687 s, 0.004741 s, +0.000054 s, ? 1.0115 mozilla-central x000_revs_x000_added_x000_copies 7c97034feb78 4407bd0c6330 : 0.198710 s, 0.190133 s, -0.008577 s, ? 0.9568 mozilla-central x0000_revs_xx000_added_0_copies 9eec5917337d 67118cc6dcad : 0.036068 s, 0.035651 s, -0.000417 s, ? 0.9884 mozilla-central x0000_revs_xx000_added_x000_copies f78c615a656c 96a38b690156 : 0.465362 s, 0.440694 s, -0.024668 s, ? 0.9470 mozilla-central x00000_revs_x0000_added_x0000_copies 6832ae71433c 4c222a1d9a00 : 24.519684 s, 18.454163 s, -6.065521 s, ? 0.7526 mozilla-central x00000_revs_x00000_added_x000_copies 76caed42cf7c 1daa622bbe42 : 42.711897 s, 31.562719 s, -11.149178 s, ? 0.7390 mozilla-try x_revs_x_added_0_copies aaf6dde0deb8 9790f499805a : 0.001201 s, 0.001189 s, -0.000012 s, ? 0.9900 mozilla-try x_revs_x000_added_0_copies d8d0222927b4 5bb8ce8c7450 : 0.001216 s, 0.001204 s, -0.000012 s, ? 0.9901 mozilla-try x_revs_x_added_x_copies 092fcca11bdb 936255a0384a : 0.000595 s, 0.000586 s, -0.000009 s, ? 0.9849 mozilla-try x_revs_x00_added_x_copies b53d2fadbdb5 017afae788ec : 0.001856 s, 0.001845 s, -0.000011 s, ? 0.9941 mozilla-try x_revs_x000_added_x000_copies 20408ad61ce5 6f0ee96e21ad : 0.064936 s, 0.063822 s, -0.001114 s, ? 0.9828 mozilla-try x_revs_x0000_added_x0000_copies effb563bb7e5 c07a39dc4e80 : 0.090601 s, 0.088038 s, -0.002563 s, ? 0.9717 mozilla-try x000_revs_xx00_added_0_copies 6100d773079a 04a55431795e : 0.007510 s, 0.007389 s, -0.000121 s, ? 0.9839 mozilla-try x000_revs_x000_added_x_copies 9f17a6fc04f9 2d37b966abed : 0.004911 s, 0.004868 s, -0.000043 s, ? 0.9912 mozilla-try x000_revs_x000_added_x000_copies 1346fd0130e4 4c65cbdabc1f : 0.233231 s, 0.222450 s, -0.010781 s, ? 0.9538 mozilla-try x0000_revs_x_added_0_copies 63519bfd42ee a36a2a865d92 : 0.419989 s, 0.370675 s, -0.049314 s, ? 0.8826 mozilla-try x0000_revs_x_added_x_copies 9fe69ff0762d bcabf2a78927 : 0.401521 s, 0.358020 s, -0.043501 s, ? 0.8917 mozilla-try x0000_revs_xx000_added_x_copies 156f6e2674f2 4d0f2c178e66 : 0.179555 s, 0.145235 s, -0.034320 s, ? 0.8089 mozilla-try x0000_revs_xx000_added_0_copies 9eec5917337d 67118cc6dcad : 0.038004 s, 0.037606 s, -0.000398 s, ? 0.9895 mozilla-try x0000_revs_xx000_added_x000_copies 89294cd501d9 7ccb2fc7ccb5 : 52.838482 s, 7.382439 s, -45.456043 s, ? 0.1397 mozilla-try x0000_revs_x0000_added_x0000_copies e928c65095ed e951f4ad123a : 8.705874 s, 7.273506 s, -1.432368 s, ? 0.8355 mozilla-try x00000_revs_x00000_added_0_copies dc8a3ca7010e d16fde900c9c : 1.126708 s, 1.074593 s, -0.052115 s, ? 0.9537 mozilla-try x00000_revs_x0000_added_x0000_copies 8d3fafa80d4b eb884023b810 : 83.854020 s, 27.746195 s, -56.107825 s, ? 0.3309 Below is a table comparing the runtime of the current "filelog centric" algorithm, with the "changeset centric" one, we just modified. The changeset centric algorithm is a significant win in many scenario, but they are still various cases where it is quite slower. When many revision has to be considered the cost of retrieving the copy information, creating new dictionaries, merging dictionaries and checking if revision are ancestors of each other can slow things down. The rest of this series, will introduce a rust version of the copy tracing code to deal with most of theses issues. Repo Case Source-Rev Dest-Rev filelog sidedata Difference Factor --------------------------------------------------------------------------------------------------------------------------------------- mercurial x_revs_x_added_0_copies ad6b123de1c7 39cfcef4f463 : 0.000914 s, 0.000044 s, - 0.000870 s, ? 0.048140 mercurial x_revs_x_added_x_copies 2b1c78674230 0c1d10351869 : 0.001812 s, 0.000138 s, - 0.001674 s, ? 0.076159 mercurial x000_revs_x000_added_x_copies 81f8ff2a9bf2 dd3267698d84 : 0.017954 s, 0.005052 s, - 0.012902 s, ? 0.281386 pypy x_revs_x_added_0_copies aed021ee8ae8 099ed31b181b : 0.001509 s, 0.000219 s, - 0.001290 s, ? 0.145129 pypy x_revs_x000_added_0_copies 4aa4e1f8e19a 359343b9ac0e : 0.206881 s, 0.000055 s, - 0.206826 s, ? 0.000266 pypy x_revs_x_added_x_copies ac52eb7bbbb0 72e022663155 : 0.016951 s, 0.000128 s, - 0.016823 s, ? 0.007551 pypy x_revs_x00_added_x_copies c3b14617fbd7 ace7255d9a26 : 0.019096 s, 0.001089 s, - 0.018007 s, ? 0.057028 pypy x_revs_x000_added_x000_copies df6f7a526b60 a83dc6a2d56f : 0.762506 s, 0.017407 s, - 0.745099 s, ? 0.022829 pypy x000_revs_xx00_added_0_copies 89a76aede314 2f22446ff07e : 1.179211 s, 0.094175 s, - 1.085036 s, ? 0.079863 pypy x000_revs_x000_added_x_copies 8a3b5bfd266e 2c68e87c3efe : 1.249058 s, 0.238009 s, - 1.011049 s, ? 0.190551 pypy x000_revs_x000_added_x000_copies 89a76aede314 7b3dda341c84 : 1.614107 s, 0.125876 s, - 1.488231 s, ? 0.077985 pypy x0000_revs_x_added_0_copies d1defd0dc478 c9cb1334cc78 : 0.001064 s, 3.581556 s, + 3.580492 s, ? 3366.124060 pypy x0000_revs_xx000_added_0_copies bf2c629d0071 4ffed77c095c : 1.061275 s, 0.016721 s, - 1.044554 s, ? 0.015756 pypy x0000_revs_xx000_added_x000_copies 08ea3258278e d9fa043f30c0 : 1.341119 s, 0.242367 s, - 1.098752 s, ? 0.180720 netbeans x_revs_x_added_0_copies fb0955ffcbcd a01e9239f9e7 : 0.027803 s, 0.000165 s, - 0.027638 s, ? 0.005935 netbeans x_revs_x000_added_0_copies 6f360122949f 20eb231cc7d0 : 0.130014 s, 0.000114 s, - 0.129900 s, ? 0.000877 netbeans x_revs_x_added_x_copies 1ada3faf6fb6 5a39d12eecf4 : 0.024990 s, 0.000296 s, - 0.024694 s, ? 0.011845 netbeans x_revs_x00_added_x_copies 35be93ba1e2c 9eec5e90c05f : 0.052201 s, 0.001124 s, - 0.051077 s, ? 0.021532 netbeans x000_revs_xx00_added_0_copies eac3045b4fdd 51d4ae7f1290 : 0.037642 s, 0.013060 s, - 0.024582 s, ? 0.346953 netbeans x000_revs_x000_added_x_copies e2063d266acd 6081d72689dc : 0.197086 s, 0.017112 s, - 0.179974 s, ? 0.086825 netbeans x000_revs_x000_added_x000_copies ff453e9fee32 411350406ec2 : 0.935148 s, 0.660350 s, - 0.274798 s, ? 0.706145 netbeans x0000_revs_xx000_added_x000_copies 588c2d1ced70 1aad62e59ddd : 3.920674 s, 10.032499 s, + 6.111825 s, ? 2.558871 mozilla-central x_revs_x_added_0_copies 3697f962bb7b 7015fcdd43a2 : 0.024232 s, 0.000189 s, - 0.024043 s, ? 0.007800 mozilla-central x_revs_x000_added_0_copies dd390860c6c9 40d0c5bed75d : 0.141483 s, 0.000462 s, - 0.141021 s, ? 0.003265 mozilla-central x_revs_x_added_x_copies 8d198483ae3b 14207ffc2b2f : 0.025775 s, 0.000270 s, - 0.025505 s, ? 0.010475 mozilla-central x_revs_x00_added_x_copies 98cbc58cc6bc 446a150332c3 : 0.084922 s, 0.001474 s, - 0.083448 s, ? 0.017357 mozilla-central x_revs_x000_added_x000_copies 3c684b4b8f68 0a5e72d1b479 : 0.194784 s, 0.004806 s, - 0.189978 s, ? 0.024673 mozilla-central x_revs_x0000_added_x0000_copies effb563bb7e5 c07a39dc4e80 : 2.161103 s, 0.085150 s, - 2.075953 s, ? 0.039401 mozilla-central x000_revs_xx00_added_0_copies 6100d773079a 04a55431795e : 0.089347 s, 0.007064 s, - 0.082283 s, ? 0.079063 mozilla-central x000_revs_x000_added_x_copies 9f17a6fc04f9 2d37b966abed : 0.732171 s, 0.004741 s, - 0.727430 s, ? 0.006475 mozilla-central x000_revs_x000_added_x000_copies 7c97034feb78 4407bd0c6330 : 1.157287 s, 0.190133 s, - 0.967154 s, ? 0.164292 mozilla-central x0000_revs_xx000_added_0_copies 9eec5917337d 67118cc6dcad : 6.726568 s, 0.035651 s, - 6.690917 s, ? 0.005300 mozilla-central x0000_revs_xx000_added_x000_copies f78c615a656c 96a38b690156 : 3.266229 s, 0.440694 s, - 2.825535 s, ? 0.134924 mozilla-central x00000_revs_x0000_added_x0000_copies 6832ae71433c 4c222a1d9a00 : 15.860534 s, 18.454163 s, + 2.593629 s, ? 1.163527 mozilla-central x00000_revs_x00000_added_x000_copies 76caed42cf7c 1daa622bbe42 : 20.450475 s, 31.562719 s, +11.112244 s, ? 1.543373 mozilla-try x_revs_x_added_0_copies aaf6dde0deb8 9790f499805a : 0.080442 s, 0.001189 s, - 0.079253 s, ? 0.014781 mozilla-try x_revs_x000_added_0_copies d8d0222927b4 5bb8ce8c7450 : 0.497672 s, 0.001204 s, - 0.496468 s, ? 0.002419 mozilla-try x_revs_x_added_x_copies 092fcca11bdb 936255a0384a : 0.021183 s, 0.000586 s, - 0.020597 s, ? 0.027664 mozilla-try x_revs_x00_added_x_copies b53d2fadbdb5 017afae788ec : 0.230991 s, 0.001845 s, - 0.229146 s, ? 0.007987 mozilla-try x_revs_x000_added_x000_copies 20408ad61ce5 6f0ee96e21ad : 1.118461 s, 0.063822 s, - 1.054639 s, ? 0.057062 mozilla-try x_revs_x0000_added_x0000_copies effb563bb7e5 c07a39dc4e80 : 2.206083 s, 0.088038 s, - 2.118045 s, ? 0.039907 mozilla-try x000_revs_xx00_added_0_copies 6100d773079a 04a55431795e : 0.089404 s, 0.007389 s, - 0.082015 s, ? 0.082647 mozilla-try x000_revs_x000_added_x_copies 9f17a6fc04f9 2d37b966abed : 0.733043 s, 0.004868 s, - 0.728175 s, ? 0.006641 mozilla-try x000_revs_x000_added_x000_copies 1346fd0130e4 4c65cbdabc1f : 1.163367 s, 0.222450 s, - 0.940917 s, ? 0.191212 mozilla-try x0000_revs_x_added_0_copies 63519bfd42ee a36a2a865d92 : 0.085456 s, 0.370675 s, + 0.285219 s, ? 4.337612 mozilla-try x0000_revs_x_added_x_copies 9fe69ff0762d bcabf2a78927 : 0.083601 s, 0.358020 s, + 0.274419 s, ? 4.282485 mozilla-try x0000_revs_xx000_added_x_copies 156f6e2674f2 4d0f2c178e66 : 7.366614 s, 0.145235 s, - 7.221379 s, ? 0.019715 mozilla-try x0000_revs_xx000_added_0_copies 9eec5917337d 67118cc6dcad : 6.664464 s, 0.037606 s, - 6.626858 s, ? 0.005643 mozilla-try x0000_revs_xx000_added_x000_copies 89294cd501d9 7ccb2fc7ccb5 : 7.467836 s, 7.382439 s, - 0.085397 s, ? 0.988565 mozilla-try x0000_revs_x0000_added_x0000_copies e928c65095ed e951f4ad123a : 9.801294 s, 7.273506 s, - 2.527788 s, ? 0.742097 mozilla-try x00000_revs_x_added_0_copies 6a320851d377 1ebb79acd503 : 0.091886 s, killed mozilla-try x00000_revs_x00000_added_0_copies dc8a3ca7010e d16fde900c9c : 26.491140 s, 1.074593 s, -25.416547 s, ? 0.040564 mozilla-try x00000_revs_x_added_x_copies 5173c4b6f97c 95d83ee7242d : 0.092863 s, killed mozilla-try x00000_revs_x000_added_x_copies 9126823d0e9c ca82787bb23c : 0.226823 s, killed mozilla-try x00000_revs_x0000_added_x0000_copies 8d3fafa80d4b eb884023b810 : 18.914630 s, 27.746195 s, + 8.831565 s, ? 1.466917 mozilla-try x00000_revs_x00000_added_x0000_copies 1b661134e2ca 1ae03d022d6d : 21.198903 s, killed mozilla-try x00000_revs_x00000_added_x000_copies 9b2a99adc05e 8e29777b48e6 : 24.952268 s, killed Differential Revision: https://phab.mercurial-scm.org/D9296

author	Pierre-Yves David <pierre-yves.david@octobus.net>
date	Mon, 02 Nov 2020 11:03:56 +0100
parents	5d65e04b6a80
children	89a2afe31e82

rev	line source
39629 a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1 # exchangev2.py - repository exchange for wire protocol version 2
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	2 #
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	3 # Copyright 2018 Gregory Szorc <gregory.szorc@gmail.com>
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	4 #
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	5 # This software may be used and distributed according to the terms of the
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	6 # GNU General Public License version 2 or any later version.
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	7
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	8 from __future__ import absolute_import
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	9
39640 039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	10 import collections
39631 b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	11 import weakref
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	12
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	13 from .i18n import _
39629 a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	14 from .node import (
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	15 nullid,
39631 b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	16 short,
39629 a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	17 )
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	18 from . import (
39635 349482d726ee exchangev2: fetch and apply bookmarks Gregory Szorc <gregory.szorc@gmail.com> parents: 39633 diff changeset	19 bookmarks,
39638 d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	20 error,
39631 b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	21 mdiff,
40327 55836a34f41b exchangev2: recognize narrow patterns when pulling Gregory Szorc <gregory.szorc@gmail.com> parents: 40179 diff changeset	22 narrowspec,
39633 ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39631 diff changeset	23 phases,
39631 b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	24 pycompat,
42813 268662aac075 interfaces: create a new folder for interfaces and move repository.py in it Pulkit Goyal <pulkit@yandex-team.ru> parents: 41217 diff changeset	25 setdiscovery,
268662aac075 interfaces: create a new folder for interfaces and move repository.py in it Pulkit Goyal <pulkit@yandex-team.ru> parents: 41217 diff changeset	26 )
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	27 from .interfaces import repository
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	28
39629 a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	29
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	30 def pull(pullop):
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	31 """Pull using wire protocol version 2."""
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	32 repo = pullop.repo
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	33 remote = pullop.remote
40330 00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	34
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	35 usingrawchangelogandmanifest = _checkuserawstorefiledata(pullop)
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	36
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	37 # If this is a clone and it was requested to perform a "stream clone",
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	38 # we obtain the raw files data from the remote then fall back to an
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	39 # incremental pull. This is somewhat hacky and is not nearly robust enough
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	40 # for long-term usage.
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	41 if usingrawchangelogandmanifest:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	42 with repo.transaction(b'clone'):
40330 00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	43 _fetchrawstorefiles(repo, remote)
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	44 repo.invalidate(clearfilecache=True)
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	45
39631 b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	46 tr = pullop.trmanager.transaction()
39629 a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	47
40327 55836a34f41b exchangev2: recognize narrow patterns when pulling Gregory Szorc <gregory.szorc@gmail.com> parents: 40179 diff changeset	48 # We don't use the repo's narrow matcher here because the patterns passed
55836a34f41b exchangev2: recognize narrow patterns when pulling Gregory Szorc <gregory.szorc@gmail.com> parents: 40179 diff changeset	49 # to exchange.pull() could be different.
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	50 narrowmatcher = narrowspec.match(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	51 repo.root,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	52 # Empty maps to nevermatcher. So always
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	53 # set includes if missing.
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	54 pullop.includepats or {b'path:.'},
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	55 pullop.excludepats,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	56 )
40327 55836a34f41b exchangev2: recognize narrow patterns when pulling Gregory Szorc <gregory.szorc@gmail.com> parents: 40179 diff changeset	57
55836a34f41b exchangev2: recognize narrow patterns when pulling Gregory Szorc <gregory.szorc@gmail.com> parents: 40179 diff changeset	58 if pullop.includepats or pullop.excludepats:
55836a34f41b exchangev2: recognize narrow patterns when pulling Gregory Szorc <gregory.szorc@gmail.com> parents: 40179 diff changeset	59 pathfilter = {}
55836a34f41b exchangev2: recognize narrow patterns when pulling Gregory Szorc <gregory.szorc@gmail.com> parents: 40179 diff changeset	60 if pullop.includepats:
55836a34f41b exchangev2: recognize narrow patterns when pulling Gregory Szorc <gregory.szorc@gmail.com> parents: 40179 diff changeset	61 pathfilter[b'include'] = sorted(pullop.includepats)
55836a34f41b exchangev2: recognize narrow patterns when pulling Gregory Szorc <gregory.szorc@gmail.com> parents: 40179 diff changeset	62 if pullop.excludepats:
55836a34f41b exchangev2: recognize narrow patterns when pulling Gregory Szorc <gregory.szorc@gmail.com> parents: 40179 diff changeset	63 pathfilter[b'exclude'] = sorted(pullop.excludepats)
55836a34f41b exchangev2: recognize narrow patterns when pulling Gregory Szorc <gregory.szorc@gmail.com> parents: 40179 diff changeset	64 else:
55836a34f41b exchangev2: recognize narrow patterns when pulling Gregory Szorc <gregory.szorc@gmail.com> parents: 40179 diff changeset	65 pathfilter = None
55836a34f41b exchangev2: recognize narrow patterns when pulling Gregory Szorc <gregory.szorc@gmail.com> parents: 40179 diff changeset	66
39629 a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	67 # Figure out what needs to be fetched.
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	68 common, fetch, remoteheads = _pullchangesetdiscovery(
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	69 repo, remote, pullop.heads, abortwhenunrelated=pullop.force
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	70 )
39629 a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	71
39633 ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39631 diff changeset	72 # And fetch the data.
39631 b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	73 pullheads = pullop.heads or remoteheads
39633 ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39631 diff changeset	74 csetres = _fetchchangesets(repo, tr, remote, common, fetch, pullheads)
ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39631 diff changeset	75
ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39631 diff changeset	76 # New revisions are written to the changelog. But all other updates
ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39631 diff changeset	77 # are deferred. Do those now.
ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39631 diff changeset	78
ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39631 diff changeset	79 # Ensure all new changesets are draft by default. If the repo is
ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39631 diff changeset	80 # publishing, the phase will be adjusted by the loop below.
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	81 if csetres[b'added']:
45790 5d65e04b6a80 phases: convert registernew users to use revision sets Joerg Sonnenberger <joerg@bec.de> parents: 45788 diff changeset	82 phases.registernew(
5d65e04b6a80 phases: convert registernew users to use revision sets Joerg Sonnenberger <joerg@bec.de> parents: 45788 diff changeset	83 repo, tr, phases.draft, [repo[n].rev() for n in csetres[b'added']]
5d65e04b6a80 phases: convert registernew users to use revision sets Joerg Sonnenberger <joerg@bec.de> parents: 45788 diff changeset	84 )
39633 ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39631 diff changeset	85
ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39631 diff changeset	86 # And adjust the phase of all changesets accordingly.
45117 b1e51ef4e536 phases: sparsify phase lists Joerg Sonnenberger <joerg@bec.de> parents: 43546 diff changeset	87 for phasenumber, phase in phases.phasenames.items():
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	88 if phase == b'secret' or not csetres[b'nodesbyphase'][phase]:
39633 ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39631 diff changeset	89 continue
ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39631 diff changeset	90
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	91 phases.advanceboundary(
45117 b1e51ef4e536 phases: sparsify phase lists Joerg Sonnenberger <joerg@bec.de> parents: 43546 diff changeset	92 repo, tr, phasenumber, csetres[b'nodesbyphase'][phase],
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	93 )
39631 b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	94
39635 349482d726ee exchangev2: fetch and apply bookmarks Gregory Szorc <gregory.szorc@gmail.com> parents: 39633 diff changeset	95 # Write bookmark updates.
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	96 bookmarks.updatefromremote(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	97 repo.ui,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	98 repo,
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	99 csetres[b'bookmarks'],
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	100 remote.url(),
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	101 pullop.gettransaction,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	102 explicit=pullop.explicitbookmarks,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	103 )
39635 349482d726ee exchangev2: fetch and apply bookmarks Gregory Szorc <gregory.szorc@gmail.com> parents: 39633 diff changeset	104
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	105 manres = _fetchmanifests(repo, tr, remote, csetres[b'manifestnodes'])
39640 039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	106
40393 229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	107 # We don't properly support shallow changeset and manifest yet. So we apply
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	108 # depth limiting locally.
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	109 if pullop.depth:
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	110 relevantcsetnodes = set()
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	111 clnode = repo.changelog.node
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	112
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	113 for rev in repo.revs(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	114 b'ancestors(%ln, %s)', pullheads, pullop.depth - 1
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	115 ):
40393 229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	116 relevantcsetnodes.add(clnode(rev))
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	117
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	118 csetrelevantfilter = lambda n: n in relevantcsetnodes
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	119
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	120 else:
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	121 csetrelevantfilter = lambda n: True
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	122
40330 00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	123 # If obtaining the raw store files, we need to scan the full repo to
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	124 # derive all the changesets, manifests, and linkrevs.
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	125 if usingrawchangelogandmanifest:
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	126 csetsforfiles = []
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	127 mnodesforfiles = []
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	128 manifestlinkrevs = {}
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	129
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	130 for rev in repo:
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	131 ctx = repo[rev]
40393 229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	132 node = ctx.node()
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	133
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	134 if not csetrelevantfilter(node):
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	135 continue
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	136
40330 00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	137 mnode = ctx.manifestnode()
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	138
40393 229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	139 csetsforfiles.append(node)
40330 00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	140 mnodesforfiles.append(mnode)
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	141 manifestlinkrevs[mnode] = rev
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	142
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	143 else:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	144 csetsforfiles = [n for n in csetres[b'added'] if csetrelevantfilter(n)]
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	145 mnodesforfiles = manres[b'added']
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	146 manifestlinkrevs = manres[b'linkrevs']
40330 00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	147
39640 039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	148 # Find all file nodes referenced by added manifests and fetch those
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	149 # revisions.
40330 00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	150 fnodes = _derivefilesfrommanifests(repo, narrowmatcher, mnodesforfiles)
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	151 _fetchfilesfromcsets(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	152 repo,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	153 tr,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	154 remote,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	155 pathfilter,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	156 fnodes,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	157 csetsforfiles,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	158 manifestlinkrevs,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	159 shallow=bool(pullop.depth),
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	160 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	161
40330 00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	162
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	163 def _checkuserawstorefiledata(pullop):
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	164 """Check whether we should use rawstorefiledata command to retrieve data."""
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	165
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	166 repo = pullop.repo
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	167 remote = pullop.remote
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	168
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	169 # Command to obtain raw store data isn't available.
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	170 if b'rawstorefiledata' not in remote.apidescriptor[b'commands']:
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	171 return False
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	172
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	173 # Only honor if user requested stream clone operation.
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	174 if not pullop.streamclonerequested:
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	175 return False
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	176
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	177 # Only works on empty repos.
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	178 if len(repo):
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	179 return False
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	180
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	181 # TODO This is super hacky. There needs to be a storage API for this. We
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	182 # also need to check for compatibility with the remote.
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	183 if b'revlogv1' not in repo.requirements:
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	184 return False
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	185
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	186 return True
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	187
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	188
40330 00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	189 def _fetchrawstorefiles(repo, remote):
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	190 with remote.commandexecutor() as e:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	191 objs = e.callcommand(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	192 b'rawstorefiledata', {b'files': [b'changelog', b'manifestlog'],}
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	193 ).result()
40330 00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	194
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	195 # First object is a summary of files data that follows.
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	196 overall = next(objs)
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	197
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	198 progress = repo.ui.makeprogress(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	199 _(b'clone'), total=overall[b'totalsize'], unit=_(b'bytes')
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	200 )
40330 00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	201 with progress:
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	202 progress.update(0)
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	203
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	204 # Next are pairs of file metadata, data.
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	205 while True:
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	206 try:
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	207 filemeta = next(objs)
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	208 except StopIteration:
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	209 break
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	210
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	211 for k in (b'location', b'path', b'size'):
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	212 if k not in filemeta:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	213 raise error.Abort(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	214 _(b'remote file data missing key: %s') % k
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	215 )
40330 00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	216
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	217 if filemeta[b'location'] == b'store':
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	218 vfs = repo.svfs
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	219 else:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	220 raise error.Abort(
43117 8ff1ecfadcd1 cleanup: join string literals that are already on one line Martin von Zweigbergk <martinvonz@google.com> parents: 43106 diff changeset	221 _(b'invalid location for raw file data: %s')
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	222 % filemeta[b'location']
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	223 )
40330 00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	224
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	225 bytesremaining = filemeta[b'size']
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	226
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	227 with vfs.open(filemeta[b'path'], b'wb') as fh:
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	228 while True:
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	229 try:
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	230 chunk = next(objs)
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	231 except StopIteration:
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	232 break
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	233
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	234 bytesremaining -= len(chunk)
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	235
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	236 if bytesremaining < 0:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	237 raise error.Abort(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	238 _(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	239 b'received invalid number of bytes for file '
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	240 b'data; expected %d, got extra'
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	241 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	242 % filemeta[b'size']
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	243 )
40330 00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	244
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	245 progress.increment(step=len(chunk))
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	246 fh.write(chunk)
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	247
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	248 try:
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	249 if chunk.islast:
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	250 break
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	251 except AttributeError:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	252 raise error.Abort(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	253 _(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	254 b'did not receive indefinite length bytestring '
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	255 b'for file data'
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	256 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	257 )
40330 00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	258
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	259 if bytesremaining:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	260 raise error.Abort(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	261 _(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	262 b'received invalid number of bytes for'
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	263 b'file data; expected %d got %d'
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	264 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	265 % (
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	266 filemeta[b'size'],
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	267 filemeta[b'size'] - bytesremaining,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	268 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	269 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	270
39638 d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	271
39629 a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	272 def _pullchangesetdiscovery(repo, remote, heads, abortwhenunrelated=True):
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	273 """Determine which changesets need to be pulled."""
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	274
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	275 if heads:
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	276 knownnode = repo.changelog.hasnode
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	277 if all(knownnode(head) for head in heads):
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	278 return heads, False, heads
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	279
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	280 # TODO wire protocol version 2 is capable of more efficient discovery
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	281 # than setdiscovery. Consider implementing something better.
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	282 common, fetch, remoteheads = setdiscovery.findcommonheads(
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	283 repo.ui, repo, remote, abortwhenunrelated=abortwhenunrelated
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	284 )
39629 a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	285
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	286 common = set(common)
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	287 remoteheads = set(remoteheads)
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	288
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	289 # If a remote head is filtered locally, put it back in the common set.
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	290 # See the comment in exchange._pulldiscoverychangegroup() for more.
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	291
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	292 if fetch and remoteheads:
43546 a166fadf5c3b index: use `index.has_node` in `exchangev2._pullchangesetdiscovery` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 43117 diff changeset	293 has_node = repo.unfiltered().changelog.index.has_node
39629 a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	294
43546 a166fadf5c3b index: use `index.has_node` in `exchangev2._pullchangesetdiscovery` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 43117 diff changeset	295 common \|= {head for head in remoteheads if has_node(head)}
39629 a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	296
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	297 if set(remoteheads).issubset(common):
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	298 fetch = []
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	299
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	300 common.discard(nullid)
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	301
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	302 return common, fetch, remoteheads
39631 b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	303
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	304
39631 b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	305 def _fetchchangesets(repo, tr, remote, common, fetch, remoteheads):
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	306 # TODO consider adding a step here where we obtain the DAG shape first
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	307 # (or ask the server to slice changesets into chunks for us) so that
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	308 # we can perform multiple fetches in batches. This will facilitate
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	309 # resuming interrupted clones, higher server-side cache hit rates due
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	310 # to smaller segments, etc.
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	311 with remote.commandexecutor() as e:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	312 objs = e.callcommand(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	313 b'changesetdata',
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	314 {
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	315 b'revisions': [
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	316 {
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	317 b'type': b'changesetdagrange',
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	318 b'roots': sorted(common),
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	319 b'heads': sorted(remoteheads),
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	320 }
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	321 ],
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	322 b'fields': {b'bookmarks', b'parents', b'phase', b'revision'},
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	323 },
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	324 ).result()
39631 b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	325
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	326 # The context manager waits on all response data when exiting. So
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	327 # we need to remain in the context manager in order to stream data.
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	328 return _processchangesetdata(repo, tr, objs)
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	329
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	330
39631 b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	331 def _processchangesetdata(repo, tr, objs):
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	332 repo.hook(b'prechangegroup', throw=True, **pycompat.strkwargs(tr.hookargs))
39631 b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	333
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	334 urepo = repo.unfiltered()
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	335 cl = urepo.changelog
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	336
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	337 cl.delayupdate(tr)
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	338
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	339 # The first emitted object is a header describing the data that
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	340 # follows.
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	341 meta = next(objs)
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	342
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	343 progress = repo.ui.makeprogress(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	344 _(b'changesets'), unit=_(b'chunks'), total=meta.get(b'totalitems')
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	345 )
39631 b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	346
39638 d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	347 manifestnodes = {}
45788 a5206e71c536 revlog: extend addgroup() with callback for duplicates Joerg Sonnenberger <joerg@bec.de> parents: 45117 diff changeset	348 added = []
39638 d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	349
39631 b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	350 def linkrev(node):
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	351 repo.ui.debug(b'add changeset %s\n' % short(node))
39631 b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	352 # Linkrev for changelog is always self.
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	353 return len(cl)
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	354
45788 a5206e71c536 revlog: extend addgroup() with callback for duplicates Joerg Sonnenberger <joerg@bec.de> parents: 45117 diff changeset	355 def ondupchangeset(cl, node):
a5206e71c536 revlog: extend addgroup() with callback for duplicates Joerg Sonnenberger <joerg@bec.de> parents: 45117 diff changeset	356 added.append(node)
a5206e71c536 revlog: extend addgroup() with callback for duplicates Joerg Sonnenberger <joerg@bec.de> parents: 45117 diff changeset	357
39631 b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	358 def onchangeset(cl, node):
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	359 progress.increment()
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	360
39638 d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	361 revision = cl.changelogrevision(node)
45788 a5206e71c536 revlog: extend addgroup() with callback for duplicates Joerg Sonnenberger <joerg@bec.de> parents: 45117 diff changeset	362 added.append(node)
39638 d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	363
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	364 # We need to preserve the mapping of changelog revision to node
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	365 # so we can set the linkrev accordingly when manifests are added.
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	366 manifestnodes[cl.rev(node)] = revision.manifest
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	367
45117 b1e51ef4e536 phases: sparsify phase lists Joerg Sonnenberger <joerg@bec.de> parents: 43546 diff changeset	368 nodesbyphase = {phase: set() for phase in phases.phasenames.values()}
39635 349482d726ee exchangev2: fetch and apply bookmarks Gregory Szorc <gregory.szorc@gmail.com> parents: 39633 diff changeset	369 remotebookmarks = {}
39633 ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39631 diff changeset	370
39631 b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	371 # addgroup() expects a 7-tuple describing revisions. This normalizes
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	372 # the wire data to that format.
39633 ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39631 diff changeset	373 #
ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39631 diff changeset	374 # This loop also aggregates non-revision metadata, such as phase
ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39631 diff changeset	375 # data.
39631 b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	376 def iterrevisions():
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	377 for cset in objs:
39633 ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39631 diff changeset	378 node = cset[b'node']
ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39631 diff changeset	379
ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39631 diff changeset	380 if b'phase' in cset:
ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39631 diff changeset	381 nodesbyphase[cset[b'phase']].add(node)
ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39631 diff changeset	382
39635 349482d726ee exchangev2: fetch and apply bookmarks Gregory Szorc <gregory.szorc@gmail.com> parents: 39633 diff changeset	383 for mark in cset.get(b'bookmarks', []):
349482d726ee exchangev2: fetch and apply bookmarks Gregory Szorc <gregory.szorc@gmail.com> parents: 39633 diff changeset	384 remotebookmarks[mark] = node
349482d726ee exchangev2: fetch and apply bookmarks Gregory Szorc <gregory.szorc@gmail.com> parents: 39633 diff changeset	385
39636 399ddd3227a4 wireprotov2: add TODOs around extending changesetdata fields Gregory Szorc <gregory.szorc@gmail.com> parents: 39635 diff changeset	386 # TODO add mechanism for extensions to examine records so they
399ddd3227a4 wireprotov2: add TODOs around extending changesetdata fields Gregory Szorc <gregory.szorc@gmail.com> parents: 39635 diff changeset	387 # can siphon off custom data fields.
399ddd3227a4 wireprotov2: add TODOs around extending changesetdata fields Gregory Szorc <gregory.szorc@gmail.com> parents: 39635 diff changeset	388
39814 d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39641 diff changeset	389 extrafields = {}
d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39641 diff changeset	390
d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39641 diff changeset	391 for field, size in cset.get(b'fieldsfollowing', []):
d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39641 diff changeset	392 extrafields[field] = next(objs)
d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39641 diff changeset	393
39633 ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39631 diff changeset	394 # Some entries might only be metadata only updates.
39814 d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39641 diff changeset	395 if b'revision' not in extrafields:
39633 ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39631 diff changeset	396 continue
ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39631 diff changeset	397
39814 d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39641 diff changeset	398 data = extrafields[b'revision']
39631 b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	399
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	400 yield (
39633 ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39631 diff changeset	401 node,
39631 b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	402 cset[b'parents'][0],
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	403 cset[b'parents'][1],
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	404 # Linknode is always itself for changesets.
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	405 cset[b'node'],
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	406 # We always send full revisions. So delta base is not set.
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	407 nullid,
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	408 mdiff.trivialdiffheader(len(data)) + data,
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	409 # Flags not yet supported.
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	410 0,
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	411 )
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	412
45788 a5206e71c536 revlog: extend addgroup() with callback for duplicates Joerg Sonnenberger <joerg@bec.de> parents: 45117 diff changeset	413 cl.addgroup(
a5206e71c536 revlog: extend addgroup() with callback for duplicates Joerg Sonnenberger <joerg@bec.de> parents: 45117 diff changeset	414 iterrevisions(),
a5206e71c536 revlog: extend addgroup() with callback for duplicates Joerg Sonnenberger <joerg@bec.de> parents: 45117 diff changeset	415 linkrev,
a5206e71c536 revlog: extend addgroup() with callback for duplicates Joerg Sonnenberger <joerg@bec.de> parents: 45117 diff changeset	416 weakref.proxy(tr),
a5206e71c536 revlog: extend addgroup() with callback for duplicates Joerg Sonnenberger <joerg@bec.de> parents: 45117 diff changeset	417 addrevisioncb=onchangeset,
a5206e71c536 revlog: extend addgroup() with callback for duplicates Joerg Sonnenberger <joerg@bec.de> parents: 45117 diff changeset	418 duplicaterevisioncb=ondupchangeset,
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	419 )
39631 b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	420
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	421 progress.complete()
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	422
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	423 return {
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	424 b'added': added,
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	425 b'nodesbyphase': nodesbyphase,
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	426 b'bookmarks': remotebookmarks,
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	427 b'manifestnodes': manifestnodes,
39631 b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	428 }
39638 d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	429
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	430
39638 d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	431 def _fetchmanifests(repo, tr, remote, manifestnodes):
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	432 rootmanifest = repo.manifestlog.getstorage(b'')
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	433
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	434 # Some manifests can be shared between changesets. Filter out revisions
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	435 # we already know about.
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	436 fetchnodes = []
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	437 linkrevs = {}
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	438 seen = set()
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	439
43106 d783f945a701 py3: finish porting iteritems() to pycompat and remove source transformer Gregory Szorc <gregory.szorc@gmail.com> parents: 43104 diff changeset	440 for clrev, node in sorted(pycompat.iteritems(manifestnodes)):
39638 d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	441 if node in seen:
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	442 continue
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	443
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	444 try:
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	445 rootmanifest.rev(node)
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	446 except error.LookupError:
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	447 fetchnodes.append(node)
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	448 linkrevs[node] = clrev
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	449
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	450 seen.add(node)
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	451
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	452 # TODO handle tree manifests
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	453
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	454 # addgroup() expects 7-tuple describing revisions. This normalizes
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	455 # the wire data to that format.
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	456 def iterrevisions(objs, progress):
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	457 for manifest in objs:
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	458 node = manifest[b'node']
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	459
39814 d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39641 diff changeset	460 extrafields = {}
d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39641 diff changeset	461
d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39641 diff changeset	462 for field, size in manifest.get(b'fieldsfollowing', []):
d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39641 diff changeset	463 extrafields[field] = next(objs)
d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39641 diff changeset	464
d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39641 diff changeset	465 if b'delta' in extrafields:
39638 d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	466 basenode = manifest[b'deltabasenode']
39814 d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39641 diff changeset	467 delta = extrafields[b'delta']
d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39641 diff changeset	468 elif b'revision' in extrafields:
39638 d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	469 basenode = nullid
39814 d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39641 diff changeset	470 revision = extrafields[b'revision']
39638 d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	471 delta = mdiff.trivialdiffheader(len(revision)) + revision
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	472 else:
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	473 continue
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	474
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	475 yield (
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	476 node,
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	477 manifest[b'parents'][0],
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	478 manifest[b'parents'][1],
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	479 # The value passed in is passed to the lookup function passed
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	480 # to addgroup(). We already have a map of manifest node to
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	481 # changelog revision number. So we just pass in the
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	482 # manifest node here and use linkrevs.__getitem__ as the
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	483 # resolution function.
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	484 node,
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	485 basenode,
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	486 delta,
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	487 # Flags not yet supported.
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	488 0,
39638 d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	489 )
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	490
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	491 progress.increment()
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	492
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	493 progress = repo.ui.makeprogress(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	494 _(b'manifests'), unit=_(b'chunks'), total=len(fetchnodes)
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	495 )
39638 d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	496
40173 b797150a1ab9 exchangev2: honor server advertised manifestdata recommended batch size Gregory Szorc <gregory.szorc@gmail.com> parents: 40035 diff changeset	497 commandmeta = remote.apidescriptor[b'commands'][b'manifestdata']
b797150a1ab9 exchangev2: honor server advertised manifestdata recommended batch size Gregory Szorc <gregory.szorc@gmail.com> parents: 40035 diff changeset	498 batchsize = commandmeta.get(b'recommendedbatchsize', 10000)
39638 d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	499 # TODO make size configurable on client?
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	500
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	501 # We send commands 1 at a time to the remote. This is not the most
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	502 # efficient because we incur a round trip at the end of each batch.
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	503 # However, the existing frame-based reactor keeps consuming server
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	504 # data in the background. And this results in response data buffering
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	505 # in memory. This can consume gigabytes of memory.
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	506 # TODO send multiple commands in a request once background buffering
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	507 # issues are resolved.
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	508
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	509 added = []
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	510
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	511 for i in pycompat.xrange(0, len(fetchnodes), batchsize):
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	512 batch = [node for node in fetchnodes[i : i + batchsize]]
39638 d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	513 if not batch:
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	514 continue
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	515
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	516 with remote.commandexecutor() as e:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	517 objs = e.callcommand(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	518 b'manifestdata',
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	519 {
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	520 b'tree': b'',
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	521 b'nodes': batch,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	522 b'fields': {b'parents', b'revision'},
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	523 b'haveparents': True,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	524 },
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	525 ).result()
39638 d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	526
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	527 # Chomp off header object.
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	528 next(objs)
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	529
45788 a5206e71c536 revlog: extend addgroup() with callback for duplicates Joerg Sonnenberger <joerg@bec.de> parents: 45117 diff changeset	530 def onchangeset(cl, node):
a5206e71c536 revlog: extend addgroup() with callback for duplicates Joerg Sonnenberger <joerg@bec.de> parents: 45117 diff changeset	531 added.append(node)
a5206e71c536 revlog: extend addgroup() with callback for duplicates Joerg Sonnenberger <joerg@bec.de> parents: 45117 diff changeset	532
a5206e71c536 revlog: extend addgroup() with callback for duplicates Joerg Sonnenberger <joerg@bec.de> parents: 45117 diff changeset	533 rootmanifest.addgroup(
a5206e71c536 revlog: extend addgroup() with callback for duplicates Joerg Sonnenberger <joerg@bec.de> parents: 45117 diff changeset	534 iterrevisions(objs, progress),
a5206e71c536 revlog: extend addgroup() with callback for duplicates Joerg Sonnenberger <joerg@bec.de> parents: 45117 diff changeset	535 linkrevs.__getitem__,
a5206e71c536 revlog: extend addgroup() with callback for duplicates Joerg Sonnenberger <joerg@bec.de> parents: 45117 diff changeset	536 weakref.proxy(tr),
a5206e71c536 revlog: extend addgroup() with callback for duplicates Joerg Sonnenberger <joerg@bec.de> parents: 45117 diff changeset	537 addrevisioncb=onchangeset,
a5206e71c536 revlog: extend addgroup() with callback for duplicates Joerg Sonnenberger <joerg@bec.de> parents: 45117 diff changeset	538 duplicaterevisioncb=onchangeset,
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	539 )
39638 d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	540
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	541 progress.complete()
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	542
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	543 return {
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	544 b'added': added,
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	545 b'linkrevs': linkrevs,
39638 d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	546 }
39640 039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	547
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	548
40327 55836a34f41b exchangev2: recognize narrow patterns when pulling Gregory Szorc <gregory.szorc@gmail.com> parents: 40179 diff changeset	549 def _derivefilesfrommanifests(repo, matcher, manifestnodes):
39640 039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	550 """Determine what file nodes are relevant given a set of manifest nodes.
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	551
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	552 Returns a dict mapping file paths to dicts of file node to first manifest
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	553 node.
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	554 """
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	555 ml = repo.manifestlog
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	556 fnodes = collections.defaultdict(dict)
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	557
40035 7a347d362a45 exchangev2: add progress bar around manifest scanning Gregory Szorc <gregory.szorc@gmail.com> parents: 39814 diff changeset	558 progress = repo.ui.makeprogress(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	559 _(b'scanning manifests'), total=len(manifestnodes)
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	560 )
40035 7a347d362a45 exchangev2: add progress bar around manifest scanning Gregory Szorc <gregory.szorc@gmail.com> parents: 39814 diff changeset	561
7a347d362a45 exchangev2: add progress bar around manifest scanning Gregory Szorc <gregory.szorc@gmail.com> parents: 39814 diff changeset	562 with progress:
7a347d362a45 exchangev2: add progress bar around manifest scanning Gregory Szorc <gregory.szorc@gmail.com> parents: 39814 diff changeset	563 for manifestnode in manifestnodes:
7a347d362a45 exchangev2: add progress bar around manifest scanning Gregory Szorc <gregory.szorc@gmail.com> parents: 39814 diff changeset	564 m = ml.get(b'', manifestnode)
39640 039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	565
40035 7a347d362a45 exchangev2: add progress bar around manifest scanning Gregory Szorc <gregory.szorc@gmail.com> parents: 39814 diff changeset	566 # TODO this will pull in unwanted nodes because it takes the storage
7a347d362a45 exchangev2: add progress bar around manifest scanning Gregory Szorc <gregory.szorc@gmail.com> parents: 39814 diff changeset	567 # delta into consideration. What we really want is something that
7a347d362a45 exchangev2: add progress bar around manifest scanning Gregory Szorc <gregory.szorc@gmail.com> parents: 39814 diff changeset	568 # takes the delta between the manifest's parents. And ideally we
7a347d362a45 exchangev2: add progress bar around manifest scanning Gregory Szorc <gregory.szorc@gmail.com> parents: 39814 diff changeset	569 # would ignore file nodes that are known locally. For now, ignore
7a347d362a45 exchangev2: add progress bar around manifest scanning Gregory Szorc <gregory.szorc@gmail.com> parents: 39814 diff changeset	570 # both these limitations. This will result in incremental fetches
7a347d362a45 exchangev2: add progress bar around manifest scanning Gregory Szorc <gregory.szorc@gmail.com> parents: 39814 diff changeset	571 # requesting data we already have. So this is far from ideal.
7a347d362a45 exchangev2: add progress bar around manifest scanning Gregory Szorc <gregory.szorc@gmail.com> parents: 39814 diff changeset	572 md = m.readfast()
39640 039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	573
40035 7a347d362a45 exchangev2: add progress bar around manifest scanning Gregory Szorc <gregory.szorc@gmail.com> parents: 39814 diff changeset	574 for path, fnode in md.items():
40327 55836a34f41b exchangev2: recognize narrow patterns when pulling Gregory Szorc <gregory.szorc@gmail.com> parents: 40179 diff changeset	575 if matcher(path):
55836a34f41b exchangev2: recognize narrow patterns when pulling Gregory Szorc <gregory.szorc@gmail.com> parents: 40179 diff changeset	576 fnodes[path].setdefault(fnode, manifestnode)
40035 7a347d362a45 exchangev2: add progress bar around manifest scanning Gregory Szorc <gregory.szorc@gmail.com> parents: 39814 diff changeset	577
7a347d362a45 exchangev2: add progress bar around manifest scanning Gregory Szorc <gregory.szorc@gmail.com> parents: 39814 diff changeset	578 progress.increment()
39640 039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	579
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	580 return fnodes
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	581
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	582
39640 039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	583 def _fetchfiles(repo, tr, remote, fnodes, linkrevs):
40179 b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	584 """Fetch file data from explicit file revisions."""
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	585
39640 039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	586 def iterrevisions(objs, progress):
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	587 for filerevision in objs:
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	588 node = filerevision[b'node']
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	589
39814 d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39641 diff changeset	590 extrafields = {}
d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39641 diff changeset	591
d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39641 diff changeset	592 for field, size in filerevision.get(b'fieldsfollowing', []):
d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39641 diff changeset	593 extrafields[field] = next(objs)
d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39641 diff changeset	594
d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39641 diff changeset	595 if b'delta' in extrafields:
39640 039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	596 basenode = filerevision[b'deltabasenode']
39814 d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39641 diff changeset	597 delta = extrafields[b'delta']
d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39641 diff changeset	598 elif b'revision' in extrafields:
39640 039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	599 basenode = nullid
39814 d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39641 diff changeset	600 revision = extrafields[b'revision']
39640 039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	601 delta = mdiff.trivialdiffheader(len(revision)) + revision
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	602 else:
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	603 continue
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	604
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	605 yield (
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	606 node,
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	607 filerevision[b'parents'][0],
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	608 filerevision[b'parents'][1],
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	609 node,
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	610 basenode,
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	611 delta,
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	612 # Flags not yet supported.
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	613 0,
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	614 )
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	615
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	616 progress.increment()
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	617
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	618 progress = repo.ui.makeprogress(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	619 _(b'files'),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	620 unit=_(b'chunks'),
43104 74802979dd9d py3: define and use pycompat.itervalues() Gregory Szorc <gregory.szorc@gmail.com> parents: 43077 diff changeset	621 total=sum(len(v) for v in pycompat.itervalues(fnodes)),
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	622 )
39640 039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	623
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	624 # TODO make batch size configurable
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	625 batchsize = 10000
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	626 fnodeslist = [x for x in sorted(fnodes.items())]
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	627
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	628 for i in pycompat.xrange(0, len(fnodeslist), batchsize):
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	629 batch = [x for x in fnodeslist[i : i + batchsize]]
39640 039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	630 if not batch:
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	631 continue
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	632
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	633 with remote.commandexecutor() as e:
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	634 fs = []
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	635 locallinkrevs = {}
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	636
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	637 for path, nodes in batch:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	638 fs.append(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	639 (
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	640 path,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	641 e.callcommand(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	642 b'filedata',
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	643 {
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	644 b'path': path,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	645 b'nodes': sorted(nodes),
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	646 b'fields': {b'parents', b'revision'},
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	647 b'haveparents': True,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	648 },
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	649 ),
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	650 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	651 )
39640 039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	652
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	653 locallinkrevs[path] = {
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	654 node: linkrevs[manifestnode]
43106 d783f945a701 py3: finish porting iteritems() to pycompat and remove source transformer Gregory Szorc <gregory.szorc@gmail.com> parents: 43104 diff changeset	655 for node, manifestnode in pycompat.iteritems(nodes)
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	656 }
39640 039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	657
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	658 for path, f in fs:
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	659 objs = f.result()
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	660
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	661 # Chomp off header objects.
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	662 next(objs)
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	663
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	664 store = repo.file(path)
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	665 store.addgroup(
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	666 iterrevisions(objs, progress),
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	667 locallinkrevs[path].__getitem__,
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	668 weakref.proxy(tr),
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	669 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	670
40179 b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	671
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	672 def _fetchfilesfromcsets(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	673 repo, tr, remote, pathfilter, fnodes, csets, manlinkrevs, shallow=False
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	674 ):
40179 b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	675 """Fetch file data from explicit changeset revisions."""
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	676
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	677 def iterrevisions(objs, remaining, progress):
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	678 while remaining:
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	679 filerevision = next(objs)
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	680
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	681 node = filerevision[b'node']
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	682
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	683 extrafields = {}
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	684
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	685 for field, size in filerevision.get(b'fieldsfollowing', []):
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	686 extrafields[field] = next(objs)
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	687
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	688 if b'delta' in extrafields:
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	689 basenode = filerevision[b'deltabasenode']
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	690 delta = extrafields[b'delta']
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	691 elif b'revision' in extrafields:
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	692 basenode = nullid
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	693 revision = extrafields[b'revision']
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	694 delta = mdiff.trivialdiffheader(len(revision)) + revision
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	695 else:
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	696 continue
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	697
40393 229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	698 if b'linknode' in filerevision:
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	699 linknode = filerevision[b'linknode']
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	700 else:
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	701 linknode = node
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	702
40179 b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	703 yield (
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	704 node,
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	705 filerevision[b'parents'][0],
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	706 filerevision[b'parents'][1],
40393 229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	707 linknode,
40179 b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	708 basenode,
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	709 delta,
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	710 # Flags not yet supported.
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	711 0,
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	712 )
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	713
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	714 progress.increment()
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	715 remaining -= 1
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	716
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	717 progress = repo.ui.makeprogress(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	718 _(b'files'),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	719 unit=_(b'chunks'),
43104 74802979dd9d py3: define and use pycompat.itervalues() Gregory Szorc <gregory.szorc@gmail.com> parents: 43077 diff changeset	720 total=sum(len(v) for v in pycompat.itervalues(fnodes)),
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	721 )
40179 b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	722
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	723 commandmeta = remote.apidescriptor[b'commands'][b'filesdata']
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	724 batchsize = commandmeta.get(b'recommendedbatchsize', 50000)
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	725
40393 229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	726 shallowfiles = repository.REPO_FEATURE_SHALLOW_FILE_STORAGE in repo.features
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	727 fields = {b'parents', b'revision'}
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	728 clrev = repo.changelog.rev
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	729
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	730 # There are no guarantees that we'll have ancestor revisions if
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	731 # a) this repo has shallow file storage b) shallow data fetching is enabled.
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	732 # Force remote to not delta against possibly unknown revisions when these
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	733 # conditions hold.
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	734 haveparents = not (shallowfiles or shallow)
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	735
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	736 # Similarly, we may not have calculated linkrevs for all incoming file
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	737 # revisions. Ask the remote to do work for us in this case.
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	738 if not haveparents:
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	739 fields.add(b'linknode')
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	740
40179 b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	741 for i in pycompat.xrange(0, len(csets), batchsize):
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	742 batch = [x for x in csets[i : i + batchsize]]
40179 b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	743 if not batch:
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	744 continue
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	745
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	746 with remote.commandexecutor() as e:
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	747 args = {
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	748 b'revisions': [
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	749 {b'type': b'changesetexplicit', b'nodes': batch,}
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	750 ],
40393 229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	751 b'fields': fields,
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	752 b'haveparents': haveparents,
40179 b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	753 }
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	754
40327 55836a34f41b exchangev2: recognize narrow patterns when pulling Gregory Szorc <gregory.szorc@gmail.com> parents: 40179 diff changeset	755 if pathfilter:
55836a34f41b exchangev2: recognize narrow patterns when pulling Gregory Szorc <gregory.szorc@gmail.com> parents: 40179 diff changeset	756 args[b'pathfilter'] = pathfilter
55836a34f41b exchangev2: recognize narrow patterns when pulling Gregory Szorc <gregory.szorc@gmail.com> parents: 40179 diff changeset	757
40179 b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	758 objs = e.callcommand(b'filesdata', args).result()
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	759
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	760 # First object is an overall header.
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	761 overall = next(objs)
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	762
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	763 # We have overall['totalpaths'] segments.
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	764 for i in pycompat.xrange(overall[b'totalpaths']):
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	765 header = next(objs)
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	766
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	767 path = header[b'path']
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	768 store = repo.file(path)
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	769
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	770 linkrevs = {
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	771 fnode: manlinkrevs[mnode]
43106 d783f945a701 py3: finish porting iteritems() to pycompat and remove source transformer Gregory Szorc <gregory.szorc@gmail.com> parents: 43104 diff changeset	772 for fnode, mnode in pycompat.iteritems(fnodes[path])
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	773 }
40179 b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	774
40393 229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	775 def getlinkrev(node):
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	776 if node in linkrevs:
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	777 return linkrevs[node]
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	778 else:
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	779 return clrev(node)
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	780
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	781 store.addgroup(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	782 iterrevisions(objs, header[b'totalitems'], progress),
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	783 getlinkrev,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	784 weakref.proxy(tr),
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	785 maybemissingparents=shallow,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	786 )

Mercurial > public > mercurial-scm > hg

annotate mercurial/exchangev2.py @ 45892:06b64fabf91c