mercurial-scm/hg: mercurial/streamclone.py annotate

annotate mercurial/streamclone.py @ 52997:e75ed9ae5fb9

stream-clone: adjust default value for worker and memory Increasing the memory usage does not seems to yield much speed up, so we can restrict it more aggressively. However, unlimited memory usage can overwhelm a system and result in massive slowdown if the process start swapping. So it is still possible to configure it, but it is not longer used in any default profile. Having too many workers can slow things a bit, but not too much. So we can try to use higher default to yield benefit on large system.

author	Pierre-Yves David <pierre-yves.david@octobus.net>
date	Wed, 19 Feb 2025 02:45:09 +0100
parents	5480647c2964
children	e705fec4a03f

rev	line source
26441 56527b886d1d streamclone: move applystreamclone() from localrepo.py Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1 # streamclone.py - producing and consuming streaming repository data
56527b886d1d streamclone: move applystreamclone() from localrepo.py Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	2 #
56527b886d1d streamclone: move applystreamclone() from localrepo.py Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	3 # Copyright 2015 Gregory Szorc <gregory.szorc@gmail.com>
56527b886d1d streamclone: move applystreamclone() from localrepo.py Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	4 #
56527b886d1d streamclone: move applystreamclone() from localrepo.py Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	5 # This software may be used and distributed according to the terms of the
56527b886d1d streamclone: move applystreamclone() from localrepo.py Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	6 # GNU General Public License version 2 or any later version.
56527b886d1d streamclone: move applystreamclone() from localrepo.py Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	7
51859 f4733654f144 typing: add `from __future__ import annotations` to most files Matt Harbison <matt_harbison@yahoo.com> parents: 51699 diff changeset	8 from __future__ import annotations
26441 56527b886d1d streamclone: move applystreamclone() from localrepo.py Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	9
52922 0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	10 import collections
35765 56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	11 import contextlib
52359 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	12 import errno
35765 56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	13 import os
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	14 import struct
52912 7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	15 import threading
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	16
52909 3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	17 from typing import (
52920 aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	18 Callable,
52909 3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	19 Iterable,
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	20 Iterator,
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	21 Optional,
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	22 Set,
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	23 Tuple,
52911 307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52910 diff changeset	24 Type,
52909 3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	25 )
52359 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	26
26442 ef8d27f53204 streamclone: move stream_in() from localrepo Gregory Szorc <gregory.szorc@gmail.com> parents: 26441 diff changeset	27 from .i18n import _
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	28 from .interfaces import repository
26441 56527b886d1d streamclone: move applystreamclone() from localrepo.py Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	29 from . import (
47447 377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	30 bookmarks,
50520 a6543983b8f4 stream-clone: check is a compatible protocol can be found Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50519 diff changeset	31 bundle2 as bundle2mod,
35767 5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	32 cacheutil,
26442 ef8d27f53204 streamclone: move stream_in() from localrepo Gregory Szorc <gregory.szorc@gmail.com> parents: 26441 diff changeset	33 error,
40339 f0e8f27768eb streamclone: pass narrowing related info in _walkstreamfiles() Pulkit Goyal <pulkit@yandex-team.ru> parents: 40338 diff changeset	34 narrowspec,
32744 33b7283a3828 streamclone: consider secret changesets (BC) (issue5589) Gregory Szorc <gregory.szorc@gmail.com> parents: 32259 diff changeset	35 phases,
38164 aac4be30e250 py3: wrap tempfile.mkstemp() to use bytes path Yuya Nishihara <yuya@tcha.org> parents: 37638 diff changeset	36 pycompat,
46626 ee91966aec0f requirements: add constant for revlog v1 requirement Rapha?l Gom?s <rgomes@octobus.net> parents: 45106 diff changeset	37 requirements as requirementsmod,
45106 a03c177a4679 scmutil: add writereporequirements() and route requires writing through it Pulkit Goyal <7895pulkit@gmail.com> parents: 43117 diff changeset	38 scmutil,
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	39 store,
50286 3d0b5760851c undo-files: move the undo cleanup code in the transaction module Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50279 diff changeset	40 transaction,
26442 ef8d27f53204 streamclone: move stream_in() from localrepo Gregory Szorc <gregory.szorc@gmail.com> parents: 26441 diff changeset	41 util,
26441 56527b886d1d streamclone: move applystreamclone() from localrepo.py Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	42 )
48693 de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48681 diff changeset	43 from .revlogutils import (
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48681 diff changeset	44 nodemap,
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48681 diff changeset	45 )
26441 56527b886d1d streamclone: move applystreamclone() from localrepo.py Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	46
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	47
52913 58baa86c7a02 stream-clone-v2: make the number of writer dependent of usage config Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52912 diff changeset	48 # Number arbitrarily picked, feel free to change them (but the LOW one)
58baa86c7a02 stream-clone-v2: make the number of writer dependent of usage config Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52912 diff changeset	49 #
58baa86c7a02 stream-clone-v2: make the number of writer dependent of usage config Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52912 diff changeset	50 # update the configuration documentation if you touch this.
58baa86c7a02 stream-clone-v2: make the number of writer dependent of usage config Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52912 diff changeset	51 DEFAULT_NUM_WRITER = {
58baa86c7a02 stream-clone-v2: make the number of writer dependent of usage config Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52912 diff changeset	52 scmutil.RESOURCE_LOW: 1,
52997 e75ed9ae5fb9 stream-clone: adjust default value for worker and memory Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52996 diff changeset	53 scmutil.RESOURCE_MEDIUM: 4,
e75ed9ae5fb9 stream-clone: adjust default value for worker and memory Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52996 diff changeset	54 scmutil.RESOURCE_HIGH: 8,
52913 58baa86c7a02 stream-clone-v2: make the number of writer dependent of usage config Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52912 diff changeset	55 }
58baa86c7a02 stream-clone-v2: make the number of writer dependent of usage config Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52912 diff changeset	56
58baa86c7a02 stream-clone-v2: make the number of writer dependent of usage config Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52912 diff changeset	57
52920 aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	58 # Number arbitrarily picked, feel free to adjust them. Do update the
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	59 # documentation if you do so
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	60 DEFAULT_MEMORY_TARGET = {
52997 e75ed9ae5fb9 stream-clone: adjust default value for worker and memory Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52996 diff changeset	61 scmutil.RESOURCE_LOW: 50 * (2**20), # 100 MB
e75ed9ae5fb9 stream-clone: adjust default value for worker and memory Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52996 diff changeset	62 scmutil.RESOURCE_MEDIUM: 500 * 2**20, # 500 MB
e75ed9ae5fb9 stream-clone: adjust default value for worker and memory Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52996 diff changeset	63 scmutil.RESOURCE_HIGH: 2 * 2**30, # 2 GB
52920 aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	64 }
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	65
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	66
52695 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	67 def new_stream_clone_requirements(
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	68 default_requirements: Iterable[bytes],
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	69 streamed_requirements: Iterable[bytes],
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	70 ) -> Set[bytes]:
48596 739f2ca3aa3f stream-clone: factor computation of new clone requirement out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47878 diff changeset	71 """determine the final set of requirement for a new stream clone
739f2ca3aa3f stream-clone: factor computation of new clone requirement out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47878 diff changeset	72
739f2ca3aa3f stream-clone: factor computation of new clone requirement out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47878 diff changeset	73 this method combine the "default" requirements that a new repository would
739f2ca3aa3f stream-clone: factor computation of new clone requirement out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47878 diff changeset	74 use with the constaint we get from the stream clone content. We keep local
739f2ca3aa3f stream-clone: factor computation of new clone requirement out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47878 diff changeset	75 configuration choice when possible.
739f2ca3aa3f stream-clone: factor computation of new clone requirement out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47878 diff changeset	76 """
739f2ca3aa3f stream-clone: factor computation of new clone requirement out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47878 diff changeset	77 requirements = set(default_requirements)
48601 baddab229b86 stream-clone: add a explicit set list requirements relevant to stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48600 diff changeset	78 requirements -= requirementsmod.STREAM_FIXED_REQUIREMENTS
48596 739f2ca3aa3f stream-clone: factor computation of new clone requirement out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47878 diff changeset	79 requirements.update(streamed_requirements)
739f2ca3aa3f stream-clone: factor computation of new clone requirement out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47878 diff changeset	80 return requirements
739f2ca3aa3f stream-clone: factor computation of new clone requirement out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47878 diff changeset	81
739f2ca3aa3f stream-clone: factor computation of new clone requirement out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47878 diff changeset	82
52695 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	83 def streamed_requirements(repo) -> Set[bytes]:
48597 8475a1364909 stream-clone: factor computation of requirement of a stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48596 diff changeset	84 """the set of requirement the new clone will have to support
8475a1364909 stream-clone: factor computation of requirement of a stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48596 diff changeset	85
8475a1364909 stream-clone: factor computation of requirement of a stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48596 diff changeset	86 This is used for advertising the stream options and to generate the actual
8475a1364909 stream-clone: factor computation of requirement of a stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48596 diff changeset	87 stream content."""
48601 baddab229b86 stream-clone: add a explicit set list requirements relevant to stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48600 diff changeset	88 requiredformats = (
baddab229b86 stream-clone: add a explicit set list requirements relevant to stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48600 diff changeset	89 repo.requirements & requirementsmod.STREAM_FIXED_REQUIREMENTS
baddab229b86 stream-clone: add a explicit set list requirements relevant to stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48600 diff changeset	90 )
48597 8475a1364909 stream-clone: factor computation of requirement of a stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48596 diff changeset	91 return requiredformats
8475a1364909 stream-clone: factor computation of requirement of a stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48596 diff changeset	92
8475a1364909 stream-clone: factor computation of requirement of a stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48596 diff changeset	93
52695 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	94 def canperformstreamclone(pullop, bundle2: bool = False):
26446 3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	95 """Whether it is possible to perform a streaming clone as part of pull.
26445 f134fb33c906 streamclone: move streaming clone logic from localrepo Gregory Szorc <gregory.szorc@gmail.com> parents: 26444 diff changeset	96
35757 bbf7abd09ff0 streamclone: rework canperformstreamclone Boris Feld <boris.feld@octobus.net> parents: 35756 diff changeset	97 ``bundle2`` will cause the function to consider stream clone through
bbf7abd09ff0 streamclone: rework canperformstreamclone Boris Feld <boris.feld@octobus.net> parents: 35756 diff changeset	98 bundle2 and only through bundle2.
26467 ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware Gregory Szorc <gregory.szorc@gmail.com> parents: 26466 diff changeset	99
26446 3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	100 Returns a tuple of (supported, requirements). ``supported`` is True if
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	101 streaming clone is supported and False otherwise. ``requirements`` is
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	102 a set of repo requirements from the remote, or ``None`` if stream clone
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	103 isn't supported.
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	104 """
26466 3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop Gregory Szorc <gregory.szorc@gmail.com> parents: 26462 diff changeset	105 repo = pullop.repo
3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop Gregory Szorc <gregory.szorc@gmail.com> parents: 26462 diff changeset	106 remote = pullop.remote
3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop Gregory Szorc <gregory.szorc@gmail.com> parents: 26462 diff changeset	107
50519 58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50518 diff changeset	108 # should we consider streaming clone at all ?
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50518 diff changeset	109 streamrequested = pullop.streamclonerequested
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50518 diff changeset	110 # If we don't have a preference, let the server decide for us. This
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50518 diff changeset	111 # likely only comes into play in LANs.
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50518 diff changeset	112 if streamrequested is None:
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50518 diff changeset	113 # The server can advertise whether to prefer streaming clone.
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50518 diff changeset	114 streamrequested = remote.capable(b'stream-preferred')
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50518 diff changeset	115 if not streamrequested:
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50518 diff changeset	116 return False, None
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50518 diff changeset	117
50517 0558866957fa stream-clone: bail-out earlier if destination repo is not empty Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50513 diff changeset	118 # Streaming clone only works on an empty destination repository
0558866957fa stream-clone: bail-out earlier if destination repo is not empty Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50513 diff changeset	119 if len(repo):
0558866957fa stream-clone: bail-out earlier if destination repo is not empty Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50513 diff changeset	120 return False, None
0558866957fa stream-clone: bail-out earlier if destination repo is not empty Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50513 diff changeset	121
50518 f697af015683 stream-clone: bail-out earlier if pull is partial Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50517 diff changeset	122 # Streaming clone only works if all data is being requested.
f697af015683 stream-clone: bail-out earlier if pull is partial Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50517 diff changeset	123 if pullop.heads:
f697af015683 stream-clone: bail-out earlier if pull is partial Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50517 diff changeset	124 return False, None
f697af015683 stream-clone: bail-out earlier if pull is partial Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50517 diff changeset	125
26467 ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware Gregory Szorc <gregory.szorc@gmail.com> parents: 26466 diff changeset	126 bundle2supported = False
ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware Gregory Szorc <gregory.szorc@gmail.com> parents: 26466 diff changeset	127 if pullop.canusebundle2:
50520 a6543983b8f4 stream-clone: check is a compatible protocol can be found Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50519 diff changeset	128 local_caps = bundle2mod.getrepocaps(repo, role=b'client')
a6543983b8f4 stream-clone: check is a compatible protocol can be found Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50519 diff changeset	129 local_supported = set(local_caps.get(b'stream', []))
a6543983b8f4 stream-clone: check is a compatible protocol can be found Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50519 diff changeset	130 remote_supported = set(pullop.remotebundle2caps.get(b'stream', []))
a6543983b8f4 stream-clone: check is a compatible protocol can be found Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50519 diff changeset	131 bundle2supported = bool(local_supported & remote_supported)
26467 ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware Gregory Szorc <gregory.szorc@gmail.com> parents: 26466 diff changeset	132 # else
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	133 # Server doesn't support bundle2 stream clone or doesn't support
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	134 # the versions we support. Fall back and possibly allow legacy.
26467 ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware Gregory Szorc <gregory.szorc@gmail.com> parents: 26466 diff changeset	135
ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware Gregory Szorc <gregory.szorc@gmail.com> parents: 26466 diff changeset	136 # Ensures legacy code path uses available bundle2.
35757 bbf7abd09ff0 streamclone: rework canperformstreamclone Boris Feld <boris.feld@octobus.net> parents: 35756 diff changeset	137 if bundle2supported and not bundle2:
26467 ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware Gregory Szorc <gregory.szorc@gmail.com> parents: 26466 diff changeset	138 return False, None
ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware Gregory Szorc <gregory.szorc@gmail.com> parents: 26466 diff changeset	139 # Ensures bundle2 doesn't try to do a stream clone if it isn't supported.
35757 bbf7abd09ff0 streamclone: rework canperformstreamclone Boris Feld <boris.feld@octobus.net> parents: 35756 diff changeset	140 elif bundle2 and not bundle2supported:
bbf7abd09ff0 streamclone: rework canperformstreamclone Boris Feld <boris.feld@octobus.net> parents: 35756 diff changeset	141 return False, None
26467 ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware Gregory Szorc <gregory.szorc@gmail.com> parents: 26466 diff changeset	142
26446 3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	143 # In order for stream clone to work, the client has to support all the
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	144 # requirements advertised by the server.
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	145 #
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	146 # The server advertises its requirements via the "stream" and "streamreqs"
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	147 # capability. "stream" (a value-less capability) is advertised if and only
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	148 # if the only requirement is "revlogv1." Else, the "streamreqs" capability
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	149 # is advertised and contains a comma-delimited list of requirements.
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	150 requirements = set()
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	151 if remote.capable(b'stream'):
46626 ee91966aec0f requirements: add constant for revlog v1 requirement Rapha?l Gom?s <rgomes@octobus.net> parents: 45106 diff changeset	152 requirements.add(requirementsmod.REVLOGV1_REQUIREMENT)
26446 3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	153 else:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	154 streamreqs = remote.capable(b'streamreqs')
26446 3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	155 # This is weird and shouldn't happen with modern servers.
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	156 if not streamreqs:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	157 pullop.repo.ui.warn(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	158 _(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	159 b'warning: stream clone requested but server has them '
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	160 b'disabled\n'
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	161 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	162 )
26446 3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	163 return False, None
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	164
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	165 streamreqs = set(streamreqs.split(b','))
26446 3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	166 # Server requires something we don't support. Bail.
48670 6d2ddea0721a stream-clone: filter possible missing requirements using all supported one Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47878 diff changeset	167 missingreqs = streamreqs - repo.supported
32259 076f1ff43f0f clone: warn when streaming was requested but couldn't be performed Siddharth Agarwal <sid0@fb.com> parents: 30975 diff changeset	168 if missingreqs:
076f1ff43f0f clone: warn when streaming was requested but couldn't be performed Siddharth Agarwal <sid0@fb.com> parents: 30975 diff changeset	169 pullop.repo.ui.warn(
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	170 _(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	171 b'warning: stream clone requested but client is missing '
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	172 b'requirements: %s\n'
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	173 )
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	174 % b', '.join(sorted(missingreqs))
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	175 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	176 pullop.repo.ui.warn(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	177 _(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	178 b'(see https://www.mercurial-scm.org/wiki/MissingRequirement '
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	179 b'for more information)\n'
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	180 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	181 )
26446 3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	182 return False, None
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	183 requirements = streamreqs
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	184
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	185 return True, requirements
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	186
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	187
52695 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	188 def maybeperformlegacystreamclone(pullop) -> None:
26462 3b0ec09192ae streamclone: rename and document maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26461 diff changeset	189 """Possibly perform a legacy stream clone operation.
3b0ec09192ae streamclone: rename and document maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26461 diff changeset	190
3b0ec09192ae streamclone: rename and document maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26461 diff changeset	191 Legacy stream clones are performed as part of pull but before all other
3b0ec09192ae streamclone: rename and document maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26461 diff changeset	192 operations.
3b0ec09192ae streamclone: rename and document maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26461 diff changeset	193
3b0ec09192ae streamclone: rename and document maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26461 diff changeset	194 A legacy stream clone will not be performed if a bundle2 stream clone is
3b0ec09192ae streamclone: rename and document maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26461 diff changeset	195 supported.
3b0ec09192ae streamclone: rename and document maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26461 diff changeset	196 """
39700 b10d145837bc localrepo: extract resolving of opener options to standalone functions Gregory Szorc <gregory.szorc@gmail.com> parents: 38783 diff changeset	197 from . import localrepo
b10d145837bc localrepo: extract resolving of opener options to standalone functions Gregory Szorc <gregory.szorc@gmail.com> parents: 38783 diff changeset	198
26466 3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop Gregory Szorc <gregory.szorc@gmail.com> parents: 26462 diff changeset	199 supported, requirements = canperformstreamclone(pullop)
26458 362793295640 streamclone: refactor maybeperformstreamclone to take a pullop Gregory Szorc <gregory.szorc@gmail.com> parents: 26447 diff changeset	200
26446 3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	201 if not supported:
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	202 return
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	203
26466 3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop Gregory Szorc <gregory.szorc@gmail.com> parents: 26462 diff changeset	204 repo = pullop.repo
3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop Gregory Szorc <gregory.szorc@gmail.com> parents: 26462 diff changeset	205 remote = pullop.remote
3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop Gregory Szorc <gregory.szorc@gmail.com> parents: 26462 diff changeset	206
26459 3b28ffde133a streamclone: move streamin() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26458 diff changeset	207 # Save remote branchmap. We will use it later to speed up branchcache
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26458 diff changeset	208 # creation.
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26458 diff changeset	209 rbranchmap = None
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	210 if remote.capable(b'branchmap'):
37638 65b86ee69383 streamclone: use command executor for wire protocol commands Gregory Szorc <gregory.szorc@gmail.com> parents: 35804 diff changeset	211 with remote.commandexecutor() as e:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	212 rbranchmap = e.callcommand(b'branchmap', {}).result()
26459 3b28ffde133a streamclone: move streamin() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26458 diff changeset	213
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	214 repo.ui.status(_(b'streaming all changes\n'))
26470 4b5647d9ee13 streamclone: move "streaming all changes" message location Gregory Szorc <gregory.szorc@gmail.com> parents: 26469 diff changeset	215
37638 65b86ee69383 streamclone: use command executor for wire protocol commands Gregory Szorc <gregory.szorc@gmail.com> parents: 35804 diff changeset	216 with remote.commandexecutor() as e:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	217 fp = e.callcommand(b'stream_out', {}).result()
37638 65b86ee69383 streamclone: use command executor for wire protocol commands Gregory Szorc <gregory.szorc@gmail.com> parents: 35804 diff changeset	218
65b86ee69383 streamclone: use command executor for wire protocol commands Gregory Szorc <gregory.szorc@gmail.com> parents: 35804 diff changeset	219 # TODO strictly speaking, this code should all be inside the context
65b86ee69383 streamclone: use command executor for wire protocol commands Gregory Szorc <gregory.szorc@gmail.com> parents: 35804 diff changeset	220 # manager because the context manager is supposed to ensure all wire state
65b86ee69383 streamclone: use command executor for wire protocol commands Gregory Szorc <gregory.szorc@gmail.com> parents: 35804 diff changeset	221 # is flushed when exiting. But the legacy peers don't do this, so it
65b86ee69383 streamclone: use command executor for wire protocol commands Gregory Szorc <gregory.szorc@gmail.com> parents: 35804 diff changeset	222 # doesn't matter.
26459 3b28ffde133a streamclone: move streamin() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26458 diff changeset	223 l = fp.readline()
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26458 diff changeset	224 try:
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26458 diff changeset	225 resp = int(l)
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26458 diff changeset	226 except ValueError:
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26458 diff changeset	227 raise error.ResponseError(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	228 _(b'unexpected response from remote server:'), l
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	229 )
26459 3b28ffde133a streamclone: move streamin() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26458 diff changeset	230 if resp == 1:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	231 raise error.Abort(_(b'operation forbidden by server'))
26459 3b28ffde133a streamclone: move streamin() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26458 diff changeset	232 elif resp == 2:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	233 raise error.Abort(_(b'locking the remote repository failed'))
26459 3b28ffde133a streamclone: move streamin() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26458 diff changeset	234 elif resp != 0:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	235 raise error.Abort(_(b'the server sent an unknown error code'))
26459 3b28ffde133a streamclone: move streamin() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26458 diff changeset	236
26468 19bbd53af46d streamclone: move payload header line consumption Gregory Szorc <gregory.szorc@gmail.com> parents: 26467 diff changeset	237 l = fp.readline()
19bbd53af46d streamclone: move payload header line consumption Gregory Szorc <gregory.szorc@gmail.com> parents: 26467 diff changeset	238 try:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	239 filecount, bytecount = map(int, l.split(b' ', 1))
26468 19bbd53af46d streamclone: move payload header line consumption Gregory Szorc <gregory.szorc@gmail.com> parents: 26467 diff changeset	240 except (ValueError, TypeError):
19bbd53af46d streamclone: move payload header line consumption Gregory Szorc <gregory.szorc@gmail.com> parents: 26467 diff changeset	241 raise error.ResponseError(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	242 _(b'unexpected response from remote server:'), l
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	243 )
26468 19bbd53af46d streamclone: move payload header line consumption Gregory Szorc <gregory.szorc@gmail.com> parents: 26467 diff changeset	244
27850 49cfddbf54ba with: use context manager in maybeperformlegacystreamclone Bryan O'Sullivan <bryano@fb.com> parents: 27845 diff changeset	245 with repo.lock():
26468 19bbd53af46d streamclone: move payload header line consumption Gregory Szorc <gregory.szorc@gmail.com> parents: 26467 diff changeset	246 consumev1(repo, fp, filecount, bytecount)
48596 739f2ca3aa3f stream-clone: factor computation of new clone requirement out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47878 diff changeset	247 repo.requirements = new_stream_clone_requirements(
739f2ca3aa3f stream-clone: factor computation of new clone requirement out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47878 diff changeset	248 repo.requirements,
739f2ca3aa3f stream-clone: factor computation of new clone requirement out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47878 diff changeset	249 requirements,
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	250 )
39700 b10d145837bc localrepo: extract resolving of opener options to standalone functions Gregory Szorc <gregory.szorc@gmail.com> parents: 38783 diff changeset	251 repo.svfs.options = localrepo.resolvestorevfsoptions(
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	252 repo.ui, repo.requirements, repo.features
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	253 )
45106 a03c177a4679 scmutil: add writereporequirements() and route requires writing through it Pulkit Goyal <7895pulkit@gmail.com> parents: 43117 diff changeset	254 scmutil.writereporequirements(repo)
48693 de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48681 diff changeset	255 nodemap.post_stream_cleanup(repo)
26461 09cc3c2e9ece streamclone: move applyremotedata() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26460 diff changeset	256
09cc3c2e9ece streamclone: move applyremotedata() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26460 diff changeset	257 if rbranchmap:
41615 328ca3b9e545 branchmap: encapsulate cache updating in the map itself Martijn Pieters <mj@octobus.net> parents: 40494 diff changeset	258 repo._branchcaches.replace(repo, rbranchmap)
26461 09cc3c2e9ece streamclone: move applyremotedata() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26460 diff changeset	259
09cc3c2e9ece streamclone: move applyremotedata() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26460 diff changeset	260 repo.invalidate()
26445 f134fb33c906 streamclone: move streaming clone logic from localrepo Gregory Szorc <gregory.szorc@gmail.com> parents: 26444 diff changeset	261
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	262
52695 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	263 def allowservergeneration(repo) -> bool:
26444 623743010133 streamclone: move _allowstream() from wireproto Gregory Szorc <gregory.szorc@gmail.com> parents: 26443 diff changeset	264 """Whether streaming clones are allowed from the server."""
40028 51f10e6d66c7 streamclone: don't support stream clone unless repo feature present Gregory Szorc <gregory.szorc@gmail.com> parents: 39850 diff changeset	265 if repository.REPO_FEATURE_STREAM_CLONE not in repo.features:
51f10e6d66c7 streamclone: don't support stream clone unless repo feature present Gregory Szorc <gregory.szorc@gmail.com> parents: 39850 diff changeset	266 return False
51f10e6d66c7 streamclone: don't support stream clone unless repo feature present Gregory Szorc <gregory.szorc@gmail.com> parents: 39850 diff changeset	267
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	268 if not repo.ui.configbool(b'server', b'uncompressed', untrusted=True):
32744 33b7283a3828 streamclone: consider secret changesets (BC) (issue5589) Gregory Szorc <gregory.szorc@gmail.com> parents: 32259 diff changeset	269 return False
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589) Gregory Szorc <gregory.szorc@gmail.com> parents: 32259 diff changeset	270
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589) Gregory Szorc <gregory.szorc@gmail.com> parents: 32259 diff changeset	271 # The way stream clone works makes it impossible to hide secret changesets.
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589) Gregory Szorc <gregory.szorc@gmail.com> parents: 32259 diff changeset	272 # So don't allow this by default.
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589) Gregory Szorc <gregory.szorc@gmail.com> parents: 32259 diff changeset	273 secret = phases.hassecret(repo)
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589) Gregory Szorc <gregory.szorc@gmail.com> parents: 32259 diff changeset	274 if secret:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	275 return repo.ui.configbool(b'server', b'uncompressedallowsecret')
32744 33b7283a3828 streamclone: consider secret changesets (BC) (issue5589) Gregory Szorc <gregory.szorc@gmail.com> parents: 32259 diff changeset	276
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589) Gregory Szorc <gregory.szorc@gmail.com> parents: 32259 diff changeset	277 return True
26444 623743010133 streamclone: move _allowstream() from wireproto Gregory Szorc <gregory.szorc@gmail.com> parents: 26443 diff changeset	278
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	279
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	280 # This is it's own function so extensions can override it.
52695 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	281 def _walkstreamfiles(
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	282 repo, matcher=None, phase: bool = False, obsolescence: bool = False
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	283 ):
50511 0925eaf09c8b store: make `walk` return an entry for obsolescence if requested so Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50509 diff changeset	284 return repo.store.walk(matcher, phase=phase, obsolescence=obsolescence)
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	285
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	286
52702 ca17d31624ac stream: report number of file written then information is available Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52701 diff changeset	287 def _report_transferred(
ca17d31624ac stream: report number of file written then information is available Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52701 diff changeset	288 repo, start_time: float, file_count: int, byte_count: int
ca17d31624ac stream: report number of file written then information is available Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52701 diff changeset	289 ):
ca17d31624ac stream: report number of file written then information is available Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52701 diff changeset	290 """common utility to report time it took to apply the stream bundle"""
52700 061bfd699a56 stream: use an utility function to report transferred bytes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52696 diff changeset	291 elapsed = util.timer() - start_time
061bfd699a56 stream: use an utility function to report transferred bytes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52696 diff changeset	292 if elapsed <= 0:
061bfd699a56 stream: use an utility function to report transferred bytes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52696 diff changeset	293 elapsed = 0.001
52702 ca17d31624ac stream: report number of file written then information is available Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52701 diff changeset	294 m = _(b'stream-cloned %d files / %s in %.1f seconds (%s/sec)\n')
52700 061bfd699a56 stream: use an utility function to report transferred bytes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52696 diff changeset	295 m %= (
52702 ca17d31624ac stream: report number of file written then information is available Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52701 diff changeset	296 file_count,
52700 061bfd699a56 stream: use an utility function to report transferred bytes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52696 diff changeset	297 util.bytecount(byte_count),
061bfd699a56 stream: use an utility function to report transferred bytes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52696 diff changeset	298 elapsed,
061bfd699a56 stream: use an utility function to report transferred bytes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52696 diff changeset	299 util.bytecount(byte_count / elapsed),
061bfd699a56 stream: use an utility function to report transferred bytes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52696 diff changeset	300 )
061bfd699a56 stream: use an utility function to report transferred bytes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52696 diff changeset	301 repo.ui.status(m)
061bfd699a56 stream: use an utility function to report transferred bytes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52696 diff changeset	302
061bfd699a56 stream: use an utility function to report transferred bytes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52696 diff changeset	303
52695 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	304 def generatev1(repo) -> tuple[int, int, Iterator[bytes]]:
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	305 """Emit content for version 1 of a streaming clone.
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	306
26469 fb743268510e streamclone: move payload header generation into own function Gregory Szorc <gregory.szorc@gmail.com> parents: 26468 diff changeset	307 This returns a 3-tuple of (file count, byte size, data iterator).
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	308
26469 fb743268510e streamclone: move payload header generation into own function Gregory Szorc <gregory.szorc@gmail.com> parents: 26468 diff changeset	309 The data iterator consists of N entries for each file being transferred.
fb743268510e streamclone: move payload header generation into own function Gregory Szorc <gregory.szorc@gmail.com> parents: 26468 diff changeset	310 Each file entry starts as a line with the file name and integer size
fb743268510e streamclone: move payload header generation into own function Gregory Szorc <gregory.szorc@gmail.com> parents: 26468 diff changeset	311 delimited by a null byte.
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	312
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	313 The raw file data follows. Following the raw file data is the next file
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	314 entry, or EOF.
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	315
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	316 When used on the wire protocol, an additional line indicating protocol
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	317 success will be prepended to the stream. This function is not responsible
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	318 for adding it.
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	319
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	320 This function will obtain a repository lock to ensure a consistent view of
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	321 the store is captured. It therefore may raise LockError.
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	322 """
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	323 entries = []
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	324 total_bytes = 0
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	325 # Get consistent snapshot of repo, lock during scan.
27845 7417e1c10253 with: use context manager in streamclone generatev1 Bryan O'Sullivan <bryano@fb.com> parents: 27794 diff changeset	326 with repo.lock():
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	327 repo.ui.debug(b'scanning\n')
52359 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	328 _test_sync_point_walk_1_2(repo)
50468 521fec115dad store: use a StoreEntry object instead of tuple for store files Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50288 diff changeset	329 for entry in _walkstreamfiles(repo):
50469 9fdc28e21b68 store: introduce a EntryFile object to actually access file info Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50468 diff changeset	330 for f in entry.files():
50474 4cbdfab6f812 store: lazily get file size on demand for the fncache case Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50469 diff changeset	331 file_size = f.file_size(repo.store.vfs)
4cbdfab6f812 store: lazily get file size on demand for the fncache case Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50469 diff changeset	332 if file_size:
4cbdfab6f812 store: lazily get file size on demand for the fncache case Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50469 diff changeset	333 entries.append((f.unencoded_path, file_size))
4cbdfab6f812 store: lazily get file size on demand for the fncache case Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50469 diff changeset	334 total_bytes += file_size
52358 11484a19cd77 stream: rename all test hook point one number up Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52357 diff changeset	335 _test_sync_point_walk_3(repo)
11484a19cd77 stream: rename all test hook point one number up Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52357 diff changeset	336 _test_sync_point_walk_4(repo)
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	337
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	338 repo.ui.debug(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	339 b'%d files, %d bytes to transfer\n' % (len(entries), total_bytes)
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	340 )
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	341
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	342 svfs = repo.svfs
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	343 debugflag = repo.ui.debugflag
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	344
52695 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	345 def emitrevlogdata() -> Iterator[bytes]:
33258 761ccfeff8b1 streamclone: stop using 'vfs.mustaudit = False' Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33257 diff changeset	346 for name, size in entries:
761ccfeff8b1 streamclone: stop using 'vfs.mustaudit = False' Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33257 diff changeset	347 if debugflag:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	348 repo.ui.debug(b'sending %s (%d bytes)\n' % (name, size))
33258 761ccfeff8b1 streamclone: stop using 'vfs.mustaudit = False' Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33257 diff changeset	349 # partially encode name over the wire for backwards compat
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	350 yield b'%s\0%d\n' % (store.encodedir(name), size)
33411 50b49bb0fff3 streamclone: comment why path auditing is disabled in generatev1() Yuya Nishihara <yuya@tcha.org> parents: 33410 diff changeset	351 # auditing at this stage is both pointless (paths are already
50b49bb0fff3 streamclone: comment why path auditing is disabled in generatev1() Yuya Nishihara <yuya@tcha.org> parents: 33410 diff changeset	352 # trusted by the local repo) and expensive
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	353 with svfs(name, b'rb', auditpath=False) as fp:
33410 c784308305c6 streamclone: close large revlog files explicitly in generatev1() Yuya Nishihara <yuya@tcha.org> parents: 33258 diff changeset	354 if size <= 65536:
33258 761ccfeff8b1 streamclone: stop using 'vfs.mustaudit = False' Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33257 diff changeset	355 yield fp.read(size)
33410 c784308305c6 streamclone: close large revlog files explicitly in generatev1() Yuya Nishihara <yuya@tcha.org> parents: 33258 diff changeset	356 else:
52644 e627cc25b6f3 pyupgrade: rewrite `yield` statements in a loop to `yield from` Matt Harbison <matt_harbison@yahoo.com> parents: 52640 diff changeset	357 yield from util.filechunkiter(fp, limit=size)
26469 fb743268510e streamclone: move payload header generation into own function Gregory Szorc <gregory.szorc@gmail.com> parents: 26468 diff changeset	358
fb743268510e streamclone: move payload header generation into own function Gregory Szorc <gregory.szorc@gmail.com> parents: 26468 diff changeset	359 return len(entries), total_bytes, emitrevlogdata()
fb743268510e streamclone: move payload header generation into own function Gregory Szorc <gregory.szorc@gmail.com> parents: 26468 diff changeset	360
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	361
52691 279e217d6041 typing: lock in the new type annotations detected with the pyupgrade changes Matt Harbison <matt_harbison@yahoo.com> parents: 52644 diff changeset	362 def generatev1wireproto(repo) -> Iterator[bytes]:
26469 fb743268510e streamclone: move payload header generation into own function Gregory Szorc <gregory.szorc@gmail.com> parents: 26468 diff changeset	363 """Emit content for version 1 of streaming clone suitable for the wire.
fb743268510e streamclone: move payload header generation into own function Gregory Szorc <gregory.szorc@gmail.com> parents: 26468 diff changeset	364
35491 ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	365 This is the data output from ``generatev1()`` with 2 header lines. The
ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	366 first line indicates overall success. The 2nd contains the file count and
ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	367 byte size of payload.
ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	368
ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	369 The success line contains "0" for success, "1" for stream generation not
ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	370 allowed, and "2" for error locking the repository (possibly indicating
ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	371 a permissions error for the server process).
26469 fb743268510e streamclone: move payload header generation into own function Gregory Szorc <gregory.szorc@gmail.com> parents: 26468 diff changeset	372 """
35491 ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	373 if not allowservergeneration(repo):
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	374 yield b'1\n'
35491 ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	375 return
ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	376
ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	377 try:
ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	378 filecount, bytecount, it = generatev1(repo)
ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	379 except error.LockError:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	380 yield b'2\n'
35491 ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	381 return
ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	382
ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	383 # Indicates successful response.
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	384 yield b'0\n'
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	385 yield b'%d %d\n' % (filecount, bytecount)
52644 e627cc25b6f3 pyupgrade: rewrite `yield` statements in a loop to `yield from` Matt Harbison <matt_harbison@yahoo.com> parents: 52640 diff changeset	386 yield from it
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	387
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	388
52695 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	389 def generatebundlev1(
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	390 repo, compression: bytes = b'UN'
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	391 ) -> tuple[Set[bytes], Iterator[bytes]]:
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	392 """Emit content for version 1 of a stream clone bundle.
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	393
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	394 The first 4 bytes of the output ("HGS1") denote this as stream clone
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	395 bundle version 1.
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	396
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	397 The next 2 bytes indicate the compression type. Only "UN" is currently
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	398 supported.
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	399
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	400 The next 16 bytes are two 64-bit big endian unsigned integers indicating
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	401 file count and byte count, respectively.
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	402
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	403 The next 2 bytes is a 16-bit big endian unsigned short declaring the length
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	404 of the requirements string, including a trailing \0. The following N bytes
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	405 are the requirements string, which is ASCII containing a comma-delimited
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	406 list of repo requirements that are needed to support the data.
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	407
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	408 The remaining content is the output of ``generatev1()`` (which may be
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	409 compressed in the future).
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	410
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	411 Returns a tuple of (requirements, data generator).
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	412 """
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	413 if compression != b'UN':
52696 10e7adbffa8c streamclone: unbyteify string args to builtin Error classes Matt Harbison <matt_harbison@yahoo.com> parents: 52695 diff changeset	414 raise ValueError('we do not support the compression argument yet')
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	415
48597 8475a1364909 stream-clone: factor computation of requirement of a stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48596 diff changeset	416 requirements = streamed_requirements(repo)
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	417 requires = b','.join(sorted(requirements))
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	418
52695 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	419 def gen() -> Iterator[bytes]:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	420 yield b'HGS1'
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	421 yield compression
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	422
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	423 filecount, bytecount, it = generatev1(repo)
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	424 repo.ui.status(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	425 _(b'writing %d bytes for %d files\n') % (bytecount, filecount)
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	426 )
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	427
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	428 yield struct.pack(b'>QQ', filecount, bytecount)
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	429 yield struct.pack(b'>H', len(requires) + 1)
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	430 yield requires + b'\0'
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	431
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	432 # This is where we'll add compression in the future.
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	433 assert compression == b'UN'
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	434
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	435 progress = repo.ui.makeprogress(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	436 _(b'bundle'), total=bytecount, unit=_(b'bytes')
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	437 )
38349 e59eaf51cc0d streamclone: use progress helper Martin von Zweigbergk <martinvonz@google.com> parents: 38164 diff changeset	438 progress.update(0)
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	439
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	440 for chunk in it:
38349 e59eaf51cc0d streamclone: use progress helper Martin von Zweigbergk <martinvonz@google.com> parents: 38164 diff changeset	441 progress.increment(step=len(chunk))
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	442 yield chunk
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	443
38373 ef692614e601 progress: hide update(None) in a new complete() method Martin von Zweigbergk <martinvonz@google.com> parents: 38360 diff changeset	444 progress.complete()
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	445
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	446 return requirements, gen()
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	447
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	448
52695 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	449 def consumev1(repo, fp, filecount: int, bytecount: int) -> None:
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	450 """Apply the contents from version 1 of a streaming clone file handle.
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	451
30332 318a24b52eeb spelling: fixes of non-dictionary words Mads Kiilerich <madski@unity3d.com> parents: 29919 diff changeset	452 This takes the output from "stream_out" and applies it to the specified
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	453 repository.
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	454
30332 318a24b52eeb spelling: fixes of non-dictionary words Mads Kiilerich <madski@unity3d.com> parents: 29919 diff changeset	455 Like "stream_out," the status line added by the wire protocol is not
318a24b52eeb spelling: fixes of non-dictionary words Mads Kiilerich <madski@unity3d.com> parents: 29919 diff changeset	456 handled by this function.
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	457 """
27859 f55a5ace8e69 with: use context manager in streamclone consumev1 Bryan O'Sullivan <bryano@fb.com> parents: 27850 diff changeset	458 with repo.lock():
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	459 repo.ui.status(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	460 _(b'%d files to transfer, %s of data\n')
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	461 % (filecount, util.bytecount(bytecount))
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	462 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	463 progress = repo.ui.makeprogress(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	464 _(b'clone'), total=bytecount, unit=_(b'bytes')
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	465 )
38349 e59eaf51cc0d streamclone: use progress helper Martin von Zweigbergk <martinvonz@google.com> parents: 38164 diff changeset	466 progress.update(0)
30975 22fbca1d11ed mercurial: switch to util.timer for all interval timings Simon Farnsworth <simonfar@fb.com> parents: 30332 diff changeset	467 start = util.timer()
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	468
29919 519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29917 diff changeset	469 # TODO: get rid of (potential) inconsistency
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29917 diff changeset	470 #
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29917 diff changeset	471 # If transaction is started and any @filecache property is
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29917 diff changeset	472 # changed at this point, it causes inconsistency between
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29917 diff changeset	473 # in-memory cached property and streamclone-ed file on the
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29917 diff changeset	474 # disk. Nested transaction prevents transaction scope "clone"
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29917 diff changeset	475 # below from writing in-memory changes out at the end of it,
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29917 diff changeset	476 # even though in-memory changes are discarded at the end of it
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29917 diff changeset	477 # regardless of transaction nesting.
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29917 diff changeset	478 #
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29917 diff changeset	479 # But transaction nesting can't be simply prohibited, because
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29917 diff changeset	480 # nesting occurs also in ordinary case (e.g. enabling
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29917 diff changeset	481 # clonebundles).
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29917 diff changeset	482
52702 ca17d31624ac stream: report number of file written then information is available Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52701 diff changeset	483 total_file_count = 0
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	484 with repo.transaction(b'clone'):
27897 2fdbf22a1b63 streamclone: use backgroundfilecloser (issue4889) Gregory Szorc <gregory.szorc@gmail.com> parents: 27896 diff changeset	485 with repo.svfs.backgroundclosing(repo.ui, expectedcount=filecount):
49284 d44e3c45f0e4 py3: replace `pycompat.xrange` by `range` Manuel Jacob <me@manueljacob.de> parents: 49190 diff changeset	486 for i in range(filecount):
27896 1d29893240cc streamclone: indent code Gregory Szorc <gregory.szorc@gmail.com> parents: 27882 diff changeset	487 # XXX doesn't support '\n' or '\r' in filenames
50925 d718eddf01d9 safehasattr: drop usage in favor of hasattr Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50703 diff changeset	488 if hasattr(fp, 'readline'):
50663 60f9602b413e clonebundles: add support for inline (streaming) clonebundles Mathias De Mare <mathias.de_mare@nokia.com> parents: 50637 diff changeset	489 l = fp.readline()
60f9602b413e clonebundles: add support for inline (streaming) clonebundles Mathias De Mare <mathias.de_mare@nokia.com> parents: 50637 diff changeset	490 else:
60f9602b413e clonebundles: add support for inline (streaming) clonebundles Mathias De Mare <mathias.de_mare@nokia.com> parents: 50637 diff changeset	491 # inline clonebundles use a chunkbuffer, so no readline
60f9602b413e clonebundles: add support for inline (streaming) clonebundles Mathias De Mare <mathias.de_mare@nokia.com> parents: 50637 diff changeset	492 # --> this should be small anyway, the first line
60f9602b413e clonebundles: add support for inline (streaming) clonebundles Mathias De Mare <mathias.de_mare@nokia.com> parents: 50637 diff changeset	493 # only contains the size of the bundle
60f9602b413e clonebundles: add support for inline (streaming) clonebundles Mathias De Mare <mathias.de_mare@nokia.com> parents: 50637 diff changeset	494 l_buf = []
60f9602b413e clonebundles: add support for inline (streaming) clonebundles Mathias De Mare <mathias.de_mare@nokia.com> parents: 50637 diff changeset	495 while not (l_buf and l_buf[-1] == b'\n'):
60f9602b413e clonebundles: add support for inline (streaming) clonebundles Mathias De Mare <mathias.de_mare@nokia.com> parents: 50637 diff changeset	496 l_buf.append(fp.read(1))
60f9602b413e clonebundles: add support for inline (streaming) clonebundles Mathias De Mare <mathias.de_mare@nokia.com> parents: 50637 diff changeset	497 l = b''.join(l_buf)
27896 1d29893240cc streamclone: indent code Gregory Szorc <gregory.szorc@gmail.com> parents: 27882 diff changeset	498 try:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	499 name, size = l.split(b'\0', 1)
27896 1d29893240cc streamclone: indent code Gregory Szorc <gregory.szorc@gmail.com> parents: 27882 diff changeset	500 size = int(size)
1d29893240cc streamclone: indent code Gregory Szorc <gregory.szorc@gmail.com> parents: 27882 diff changeset	501 except (ValueError, TypeError):
1d29893240cc streamclone: indent code Gregory Szorc <gregory.szorc@gmail.com> parents: 27882 diff changeset	502 raise error.ResponseError(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	503 _(b'unexpected response from remote server:'), l
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	504 )
27896 1d29893240cc streamclone: indent code Gregory Szorc <gregory.szorc@gmail.com> parents: 27882 diff changeset	505 if repo.ui.debugflag:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	506 repo.ui.debug(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	507 b'adding %s (%s)\n' % (name, util.bytecount(size))
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	508 )
27896 1d29893240cc streamclone: indent code Gregory Szorc <gregory.szorc@gmail.com> parents: 27882 diff changeset	509 # for backwards compat, name was partially encoded
27897 2fdbf22a1b63 streamclone: use backgroundfilecloser (issue4889) Gregory Szorc <gregory.szorc@gmail.com> parents: 27896 diff changeset	510 path = store.decodedir(name)
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	511 with repo.svfs(path, b'w', backgroundclose=True) as ofp:
52702 ca17d31624ac stream: report number of file written then information is available Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52701 diff changeset	512 total_file_count += 1
27896 1d29893240cc streamclone: indent code Gregory Szorc <gregory.szorc@gmail.com> parents: 27882 diff changeset	513 for chunk in util.filechunkiter(fp, limit=size):
38349 e59eaf51cc0d streamclone: use progress helper Martin von Zweigbergk <martinvonz@google.com> parents: 38164 diff changeset	514 progress.increment(step=len(chunk))
27896 1d29893240cc streamclone: indent code Gregory Szorc <gregory.szorc@gmail.com> parents: 27882 diff changeset	515 ofp.write(chunk)
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	516
29919 519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29917 diff changeset	517 # force @filecache properties to be reloaded from
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29917 diff changeset	518 # streamclone-ed file at next access
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29917 diff changeset	519 repo.invalidate(clearfilecache=True)
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	520
38373 ef692614e601 progress: hide update(None) in a new complete() method Martin von Zweigbergk <martinvonz@google.com> parents: 38360 diff changeset	521 progress.complete()
52702 ca17d31624ac stream: report number of file written then information is available Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52701 diff changeset	522 _report_transferred(repo, start, total_file_count, bytecount)
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	523
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	524
52695 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	525 def readbundle1header(fp) -> tuple[int, int, Set[bytes]]:
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	526 compression = fp.read(2)
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	527 if compression != b'UN':
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	528 raise error.Abort(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	529 _(
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	530 b'only uncompressed stream clone bundles are '
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	531 b'supported; got %s'
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	532 )
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	533 % compression
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	534 )
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	535
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	536 filecount, bytecount = struct.unpack(b'>QQ', fp.read(16))
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	537 requireslen = struct.unpack(b'>H', fp.read(2))[0]
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	538 requires = fp.read(requireslen)
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	539
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	540 if not requires.endswith(b'\0'):
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	541 raise error.Abort(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	542 _(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	543 b'malformed stream clone bundle: '
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	544 b'requirements not properly encoded'
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	545 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	546 )
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	547
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	548 requirements = set(requires.rstrip(b'\0').split(b','))
27882 319b0bf6ecc9 streamclone: extract code for reading header fields Gregory Szorc <gregory.szorc@gmail.com> parents: 27870 diff changeset	549
319b0bf6ecc9 streamclone: extract code for reading header fields Gregory Szorc <gregory.szorc@gmail.com> parents: 27870 diff changeset	550 return filecount, bytecount, requirements
319b0bf6ecc9 streamclone: extract code for reading header fields Gregory Szorc <gregory.szorc@gmail.com> parents: 27870 diff changeset	551
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	552
52695 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	553 def applybundlev1(repo, fp) -> None:
27882 319b0bf6ecc9 streamclone: extract code for reading header fields Gregory Szorc <gregory.szorc@gmail.com> parents: 27870 diff changeset	554 """Apply the content from a stream clone bundle version 1.
319b0bf6ecc9 streamclone: extract code for reading header fields Gregory Szorc <gregory.szorc@gmail.com> parents: 27870 diff changeset	555
319b0bf6ecc9 streamclone: extract code for reading header fields Gregory Szorc <gregory.szorc@gmail.com> parents: 27870 diff changeset	556 We assume the 4 byte header has been read and validated and the file handle
319b0bf6ecc9 streamclone: extract code for reading header fields Gregory Szorc <gregory.szorc@gmail.com> parents: 27870 diff changeset	557 is at the 2 byte compression identifier.
319b0bf6ecc9 streamclone: extract code for reading header fields Gregory Szorc <gregory.szorc@gmail.com> parents: 27870 diff changeset	558 """
319b0bf6ecc9 streamclone: extract code for reading header fields Gregory Szorc <gregory.szorc@gmail.com> parents: 27870 diff changeset	559 if len(repo):
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	560 raise error.Abort(
43117 8ff1ecfadcd1 cleanup: join string literals that are already on one line Martin von Zweigbergk <martinvonz@google.com> parents: 43085 diff changeset	561 _(b'cannot apply stream clone bundle on non-empty repo')
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	562 )
27882 319b0bf6ecc9 streamclone: extract code for reading header fields Gregory Szorc <gregory.szorc@gmail.com> parents: 27870 diff changeset	563
319b0bf6ecc9 streamclone: extract code for reading header fields Gregory Szorc <gregory.szorc@gmail.com> parents: 27870 diff changeset	564 filecount, bytecount, requirements = readbundle1header(fp)
48670 6d2ddea0721a stream-clone: filter possible missing requirements using all supported one Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47878 diff changeset	565 missingreqs = requirements - repo.supported
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	566 if missingreqs:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	567 raise error.Abort(
43117 8ff1ecfadcd1 cleanup: join string literals that are already on one line Martin von Zweigbergk <martinvonz@google.com> parents: 43085 diff changeset	568 _(b'unable to apply stream clone: unsupported format: %s')
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	569 % b', '.join(sorted(missingreqs))
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	570 )
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	571
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	572 consumev1(repo, fp, filecount, bytecount)
48693 de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48681 diff changeset	573 nodemap.post_stream_cleanup(repo)
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	574
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	575
48946 642e31cb55f0 py3: use class X: instead of class X(object): Gregory Szorc <gregory.szorc@gmail.com> parents: 48875 diff changeset	576 class streamcloneapplier:
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	577 """Class to manage applying streaming clone bundles.
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	578
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	579 We need to wrap ``applybundlev1()`` in a dedicated type to enable bundle
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	580 readers to perform bundle type-specific functionality.
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	581 """
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	582
52695 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	583 def __init__(self, fh) -> None:
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	584 self._fh = fh
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	585
52695 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	586 def apply(self, repo) -> None:
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	587 return applybundlev1(repo, self._fh)
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	588
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	589
35765 56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	590 # type of file to stream
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	591 _fileappend = 0 # append only file
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	592 _filefull = 1 # full snapshot file
35765 56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	593
35767 5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	594 # Source of the file
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	595 _srcstore = b's' # store (svfs)
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	596 _srccache = b'c' # cache (cache)
35767 5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	597
51699 ca7bde5dbafb black: format the codebase with 23.3.0 Rapha?l Gom?s <rgomes@octobus.net> parents: 51696 diff changeset	598
35765 56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	599 # This is it's own function so extensions can override it.
52695 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	600 def _walkstreamfullstorefiles(repo) -> list[bytes]:
35765 56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	601 """list snapshot file from the store"""
56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	602 fnames = []
56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	603 if not repo.publishing():
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	604 fnames.append(b'phaseroots')
35765 56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	605 return fnames
56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	606
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	607
35767 5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	608 def _filterfull(entry, copy, vfsmap):
35765 56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	609 """actually copy the snapshot files"""
35767 5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	610 src, name, ftype, data = entry
35765 56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	611 if ftype != _filefull:
56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	612 return entry
35767 5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	613 return (src, name, ftype, copy(vfsmap[src].join(name)))
35765 56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	614
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	615
52012 e308439339e2 stream: rename TempCopyManager to VolatileManager Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51859 diff changeset	616 class VolatileManager:
52015 a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	617 """Manage temporary backups of volatile files during stream clone.
50627 8c7b04e69894 stream-clone: introduce a richer TempCopyManager object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50521 diff changeset	618
52015 a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	619 This class will keep open file handles for the volatile files, writing the
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	620 smaller ones on disk if the number of open file handles grow too much.
50627 8c7b04e69894 stream-clone: introduce a richer TempCopyManager object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50521 diff changeset	621
52015 a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	622 This should be used as a Python context, the file handles and copies will
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	623 be discarded when exiting the context.
49190 4ff4e23de7df clone: use better names for temp files Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 48946 diff changeset	624
52015 a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	625 The preservation can be done by calling the object on the real path
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	626 (encoded full path).
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	627
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	628 Valid filehandles for any file should be retrieved by calling `open(path)`.
50627 8c7b04e69894 stream-clone: introduce a richer TempCopyManager object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50521 diff changeset	629 """
8c7b04e69894 stream-clone: introduce a richer TempCopyManager object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50521 diff changeset	630
52015 a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	631 # arbitrarily picked as "it seemed fine" and much higher than the current
52184 e4b242f9d4d9 streamclone: disable the volatile file open handle optimization on Windows Matt Harbison <matt_harbison@yahoo.com> parents: 52015 diff changeset	632 # usage. The Windows value of 2 is actually 1 file open at a time, due to
e4b242f9d4d9 streamclone: disable the volatile file open handle optimization on Windows Matt Harbison <matt_harbison@yahoo.com> parents: 52015 diff changeset	633 # the `flush_count = self.MAX_OPEN // 2` and `self.MAX_OPEN - 1` threshold
e4b242f9d4d9 streamclone: disable the volatile file open handle optimization on Windows Matt Harbison <matt_harbison@yahoo.com> parents: 52015 diff changeset	634 # for flushing to disk in __call__().
e4b242f9d4d9 streamclone: disable the volatile file open handle optimization on Windows Matt Harbison <matt_harbison@yahoo.com> parents: 52015 diff changeset	635 MAX_OPEN = 2 if pycompat.iswindows else 100
52015 a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	636
52695 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	637 def __init__(self) -> None:
52015 a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	638 self._counter = 0
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	639 self._volatile_fps = None
50627 8c7b04e69894 stream-clone: introduce a richer TempCopyManager object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50521 diff changeset	640 self._copies = None
8c7b04e69894 stream-clone: introduce a richer TempCopyManager object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50521 diff changeset	641 self._dst_dir = None
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	642
50627 8c7b04e69894 stream-clone: introduce a richer TempCopyManager object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50521 diff changeset	643 def __enter__(self):
52015 a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	644 if self._counter == 0:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	645 assert self._volatile_fps is None
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	646 self._volatile_fps = {}
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	647 self._counter += 1
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	648 return self
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	649
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	650 def __exit__(self, args, *kwars):
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	651 """discard all backups"""
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	652 self._counter -= 1
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	653 if self._counter == 0:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	654 for _size, fp in self._volatile_fps.values():
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	655 fp.close()
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	656 self._volatile_fps = None
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	657 if self._copies is not None:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	658 for tmp in self._copies.values():
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	659 util.tryunlink(tmp)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	660 util.tryrmdir(self._dst_dir)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	661 self._copies = None
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	662 self._dst_dir = None
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	663 assert self._volatile_fps is None
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	664 assert self._copies is None
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	665 assert self._dst_dir is None
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	666
52695 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	667 def _init_tmp_copies(self) -> None:
52015 a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	668 """prepare a temporary directory to save volatile files
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	669
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	670 This will be used as backup if we have too many files open"""
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	671 assert 0 < self._counter
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	672 assert self._copies is None
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	673 assert self._dst_dir is None
50627 8c7b04e69894 stream-clone: introduce a richer TempCopyManager object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50521 diff changeset	674 self._copies = {}
8c7b04e69894 stream-clone: introduce a richer TempCopyManager object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50521 diff changeset	675 self._dst_dir = pycompat.mkdtemp(prefix=b'hg-clone-')
52015 a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	676
52695 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	677 def _flush_some_on_disk(self) -> None:
52015 a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	678 """move some of the open files to tempory files on disk"""
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	679 if self._copies is None:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	680 self._init_tmp_copies()
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	681 flush_count = self.MAX_OPEN // 2
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	682 for src, (size, fp) in sorted(self._volatile_fps.items())[:flush_count]:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	683 prefix = os.path.basename(src)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	684 fd, dst = pycompat.mkstemp(prefix=prefix, dir=self._dst_dir)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	685 self._copies[src] = dst
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	686 os.close(fd)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	687 # we no longer hardlink, but on the other hand we rarely do this,
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	688 # and we do it for the smallest file only and not at all in the
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	689 # common case.
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	690 with open(dst, 'wb') as bck:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	691 fp.seek(0)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	692 bck.write(fp.read())
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	693 del self._volatile_fps[src]
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	694 fp.close()
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	695
52695 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	696 def _keep_one(self, src: bytes) -> int:
52015 a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	697 """preserve an open file handle for a given path"""
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	698 # store the file quickly to ensure we close it if any error happens
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	699 _, fp = self._volatile_fps[src] = (None, open(src, 'rb'))
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	700 fp.seek(0, os.SEEK_END)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	701 size = fp.tell()
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	702 self._volatile_fps[src] = (size, fp)
52359 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	703 return size
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	704
52695 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	705 def __call__(self, src: bytes) -> None:
52015 a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	706 """preserve the volatile file at src"""
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	707 assert 0 < self._counter
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	708 if len(self._volatile_fps) >= (self.MAX_OPEN - 1):
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	709 self._flush_some_on_disk()
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	710 self._keep_one(src)
50627 8c7b04e69894 stream-clone: introduce a richer TempCopyManager object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50521 diff changeset	711
52695 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	712 def try_keep(self, src: bytes) -> Optional[int]:
52359 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	713 """record a volatile file and returns it size
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	714
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	715 return None if the file does not exists.
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	716
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	717 Used for cache file that are not lock protected.
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	718 """
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	719 assert 0 < self._counter
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	720 if len(self._volatile_fps) >= (self.MAX_OPEN - 1):
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	721 self._flush_some_on_disk()
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	722 try:
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	723 return self._keep_one(src)
52640 24ee91ba9aa8 pyupgrade: drop usage of py3 aliases for `OSError` Matt Harbison <matt_harbison@yahoo.com> parents: 52359 diff changeset	724 except OSError as err:
52359 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	725 if err.errno not in (errno.ENOENT, errno.EPERM):
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	726 raise
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	727 return None
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	728
52013 0ad269e24075 stream: open volatile file through the manager Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52012 diff changeset	729 @contextlib.contextmanager
0ad269e24075 stream: open volatile file through the manager Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52012 diff changeset	730 def open(self, src):
52015 a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	731 assert 0 < self._counter
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	732 entry = self._volatile_fps.get(src)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	733 if entry is not None:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	734 _size, fp = entry
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	735 fp.seek(0)
52013 0ad269e24075 stream: open volatile file through the manager Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52012 diff changeset	736 yield fp
52015 a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	737 else:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	738 if self._copies is None:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	739 actual_path = src
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	740 else:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	741 actual_path = self._copies.get(src, src)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	742 with open(actual_path, 'rb') as fp:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	743 yield fp
35765 56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	744
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	745
35767 5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	746 def _makemap(repo):
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	747 """make a (src -> vfs) map for the repo"""
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	748 vfsmap = {
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	749 _srcstore: repo.svfs,
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	750 _srccache: repo.cachevfs,
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	751 }
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	752 # we keep repo.vfs out of the on purpose, ther are too many danger there
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	753 # (eg: .hg/hgrc)
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	754 assert repo.vfs not in vfsmap.values()
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	755
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	756 return vfsmap
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	757
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	758
50632 3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2 Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50631 diff changeset	759 def _emit2(repo, entries):
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	760 """actually emit the stream bundle"""
35767 5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	761 vfsmap = _makemap(repo)
47502 65c519661991 stream: double check that self.vfs is not in the vfsmap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	762 # we keep repo.vfs out of the on purpose, ther are too many danger there
65c519661991 stream: double check that self.vfs is not in the vfsmap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	763 # (eg: .hg/hgrc),
65c519661991 stream: double check that self.vfs is not in the vfsmap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	764 #
65c519661991 stream: double check that self.vfs is not in the vfsmap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	765 # this assert is duplicated (from _makemap) as author might think this is
65c519661991 stream: double check that self.vfs is not in the vfsmap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	766 # fine, while this is really not fine.
65c519661991 stream: double check that self.vfs is not in the vfsmap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	767 if repo.vfs in vfsmap.values():
65c519661991 stream: double check that self.vfs is not in the vfsmap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	768 raise error.ProgrammingError(
52696 10e7adbffa8c streamclone: unbyteify string args to builtin Error classes Matt Harbison <matt_harbison@yahoo.com> parents: 52695 diff changeset	769 'repo.vfs must not be added to vfsmap for security reasons'
47502 65c519661991 stream: double check that self.vfs is not in the vfsmap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	770 )
65c519661991 stream: double check that self.vfs is not in the vfsmap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	771
50632 3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2 Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50631 diff changeset	772 # translate the vfs one
3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2 Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50631 diff changeset	773 entries = [(vfs_key, vfsmap[vfs_key], e) for (vfs_key, e) in entries]
52359 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	774 _test_sync_point_walk_1_2(repo)
50632 3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2 Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50631 diff changeset	775
50637 9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50636 diff changeset	776 max_linkrev = len(repo)
50632 3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2 Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50631 diff changeset	777 file_count = totalfilesize = 0
52356 46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	778 with VolatileManager() as volatiles:
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	779 # make sure we preserve volatile files
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	780 with util.nogc():
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	781 # record the expected size of every file
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	782 for k, vfs, e in entries:
52357 a260d326458f stream: create a `e.preserve_volatiles` method directly on StoreEntry Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52356 diff changeset	783 e.preserve_volatiles(vfs, volatiles)
52356 46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	784 for f in e.files():
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	785 file_count += 1
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	786 totalfilesize += f.file_size(vfs)
50632 3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2 Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50631 diff changeset	787
52356 46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	788 progress = repo.ui.makeprogress(
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	789 _(b'bundle'), total=totalfilesize, unit=_(b'bytes')
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	790 )
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	791 progress.update(0)
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	792 with progress:
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	793 # the first yield release the lock on the repository
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	794 yield file_count, totalfilesize
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	795 totalbytecount = 0
35765 56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	796
52356 46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	797 for src, vfs, e in entries:
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	798 entry_streams = e.get_streams(
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	799 repo=repo,
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	800 vfs=vfs,
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	801 volatiles=volatiles,
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	802 max_changeset=max_linkrev,
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	803 preserve_file_count=True,
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	804 )
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	805 for name, stream, size in entry_streams:
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	806 yield src
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	807 yield util.uvarintencode(len(name))
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	808 yield util.uvarintencode(size)
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	809 yield name
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	810 bytecount = 0
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	811 for chunk in stream:
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	812 bytecount += len(chunk)
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	813 totalbytecount += len(chunk)
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	814 progress.update(totalbytecount)
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	815 yield chunk
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	816 if bytecount != size:
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	817 # Would most likely be caused by a race due to `hg
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	818 # strip` or a revlog split
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	819 msg = _(
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	820 b'clone could only read %d bytes from %s, but '
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	821 b'expected %d bytes'
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	822 )
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	823 raise error.Abort(msg % (bytecount, name, size))
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	824
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	825
52695 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	826 def _emit3(repo, entries) -> Iterator[bytes \| None]:
50703 0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	827 """actually emit the stream bundle (v3)"""
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	828 vfsmap = _makemap(repo)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	829 # we keep repo.vfs out of the map on purpose, ther are too many dangers
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	830 # there (eg: .hg/hgrc),
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	831 #
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	832 # this assert is duplicated (from _makemap) as authors might think this is
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	833 # fine, while this is really not fine.
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	834 if repo.vfs in vfsmap.values():
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	835 raise error.ProgrammingError(
52696 10e7adbffa8c streamclone: unbyteify string args to builtin Error classes Matt Harbison <matt_harbison@yahoo.com> parents: 52695 diff changeset	836 'repo.vfs must not be added to vfsmap for security reasons'
50703 0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	837 )
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	838
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	839 # translate the vfs once
52359 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	840 # we only turn this into a list for the `_test_sync`, this is not ideal
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	841 base_entries = list(entries)
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	842 _test_sync_point_walk_1_2(repo)
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	843 entries = []
52356 46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	844 with VolatileManager() as volatiles:
52012 e308439339e2 stream: rename TempCopyManager to VolatileManager Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51859 diff changeset	845 # make sure we preserve volatile files
52359 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	846 for vfs_key, e in base_entries:
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	847 vfs = vfsmap[vfs_key]
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	848 any_files = True
51549 49faa72b994e streamclone: stop listing files for entries that have no volatile files Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51548 diff changeset	849 if e.maybe_volatile:
52359 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	850 any_files = False
52357 a260d326458f stream: create a `e.preserve_volatiles` method directly on StoreEntry Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52356 diff changeset	851 e.preserve_volatiles(vfs, volatiles)
51549 49faa72b994e streamclone: stop listing files for entries that have no volatile files Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51548 diff changeset	852 for f in e.files():
49faa72b994e streamclone: stop listing files for entries that have no volatile files Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51548 diff changeset	853 if f.is_volatile:
49faa72b994e streamclone: stop listing files for entries that have no volatile files Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51548 diff changeset	854 # record the expected size under lock
49faa72b994e streamclone: stop listing files for entries that have no volatile files Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51548 diff changeset	855 f.file_size(vfs)
52359 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	856 any_files = True
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	857 if any_files:
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	858 entries.append((vfs_key, vfsmap[vfs_key], e))
52356 46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	859
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	860 total_entry_count = len(entries)
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	861
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	862 max_linkrev = len(repo)
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	863 progress = repo.ui.makeprogress(
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	864 _(b'bundle'),
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	865 total=total_entry_count,
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	866 unit=_(b'entry'),
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	867 )
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	868 progress.update(0)
50703 0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	869 # the first yield release the lock on the repository
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	870 yield None
52356 46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	871 with progress:
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	872 yield util.uvarintencode(total_entry_count)
50703 0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	873
52356 46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	874 for src, vfs, e in entries:
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	875 entry_streams = e.get_streams(
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	876 repo=repo,
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	877 vfs=vfs,
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	878 volatiles=volatiles,
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	879 max_changeset=max_linkrev,
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	880 )
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	881 yield util.uvarintencode(len(entry_streams))
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	882 for name, stream, size in entry_streams:
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	883 yield src
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	884 yield util.uvarintencode(len(name))
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	885 yield util.uvarintencode(size)
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	886 yield name
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	887 yield from stream
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	888 progress.increment()
50703 0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	889
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	890
52359 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	891 def _test_sync_point_walk_1_2(repo):
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	892 """a function for synchronisation during tests
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	893
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	894 Triggered after gather entry, but before starting to process/preserve them
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	895 under lock.
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	896
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	897 (on v1 is triggered before the actual walk start)
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	898 """
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	899
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	900
52358 11484a19cd77 stream: rename all test hook point one number up Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52357 diff changeset	901 def _test_sync_point_walk_3(repo):
52359 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	902 """a function for synchronisation during tests
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	903
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	904 Triggered right before releasing the lock, but after computing what need
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	905 needed to compute under lock.
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	906 """
46986 faa43f09ad98 streamclone: remove sleep based "synchronisation" in tests Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46895 diff changeset	907
faa43f09ad98 streamclone: remove sleep based "synchronisation" in tests Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46895 diff changeset	908
52358 11484a19cd77 stream: rename all test hook point one number up Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52357 diff changeset	909 def _test_sync_point_walk_4(repo):
52359 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	910 """a function for synchronisation during tests
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	911
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	912 Triggered right after releasing the lock.
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	913 """
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	914
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	915
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	916 # not really a StoreEntry, but close enough
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	917 class CacheEntry(store.SimpleStoreEntry):
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	918 """Represent an entry for Cache files
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	919
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	920 It has special logic to preserve cache file early and accept optional
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	921 presence.
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	922
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	923
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	924 (Yes... this is not really a StoreEntry, but close enough. We could have a
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	925 BaseEntry base class, bbut the store one would be identical)
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	926 """
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	927
52695 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	928 def __init__(self, entry_path) -> None:
52359 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	929 super().__init__(
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	930 entry_path,
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	931 # we will directly deal with that in `setup_cache_file`
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	932 is_volatile=True,
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	933 )
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	934
52695 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	935 def preserve_volatiles(self, vfs, volatiles) -> None:
52359 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	936 self._file_size = volatiles.try_keep(vfs.join(self._entry_path))
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	937 if self._file_size is None:
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	938 self._files = []
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	939 else:
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	940 assert self._is_volatile
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	941 self._files = [
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	942 CacheFile(
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	943 unencoded_path=self._entry_path,
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	944 file_size=self._file_size,
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	945 is_volatile=self._is_volatile,
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	946 )
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	947 ]
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	948
52695 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	949 def files(self) -> list[store.StoreFile]:
52359 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	950 if self._files is None:
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	951 self._files = [
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	952 CacheFile(
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	953 unencoded_path=self._entry_path,
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	954 is_volatile=self._is_volatile,
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	955 )
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	956 ]
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	957 return super().files()
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	958
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	959
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	960 class CacheFile(store.StoreFile):
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	961 # inform the "copy/hardlink" version that this file might be missing
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	962 # without consequences.
52695 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	963 optional: bool = True
46986 faa43f09ad98 streamclone: remove sleep based "synchronisation" in tests Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46895 diff changeset	964
faa43f09ad98 streamclone: remove sleep based "synchronisation" in tests Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46895 diff changeset	965
52695 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	966 def _entries_walk(repo, includes, excludes, includeobsmarkers: bool):
50512 06d580b8f432 stream-clone: introduce a _entries_walk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50511 diff changeset	967 """emit a seris of files information useful to clone a repo
06d580b8f432 stream-clone: introduce a _entries_walk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50511 diff changeset	968
06d580b8f432 stream-clone: introduce a _entries_walk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50511 diff changeset	969 return (vfs-key, entry) iterator
06d580b8f432 stream-clone: introduce a _entries_walk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50511 diff changeset	970
06d580b8f432 stream-clone: introduce a _entries_walk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50511 diff changeset	971 Where `entry` is StoreEntry. (used even for cache entries)
06d580b8f432 stream-clone: introduce a _entries_walk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50511 diff changeset	972 """
06d580b8f432 stream-clone: introduce a _entries_walk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50511 diff changeset	973 assert repo._currentlock(repo._lockref) is not None
06d580b8f432 stream-clone: introduce a _entries_walk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50511 diff changeset	974
06d580b8f432 stream-clone: introduce a _entries_walk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50511 diff changeset	975 matcher = None
06d580b8f432 stream-clone: introduce a _entries_walk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50511 diff changeset	976 if includes or excludes:
06d580b8f432 stream-clone: introduce a _entries_walk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50511 diff changeset	977 matcher = narrowspec.match(repo.root, includes, excludes)
06d580b8f432 stream-clone: introduce a _entries_walk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50511 diff changeset	978
06d580b8f432 stream-clone: introduce a _entries_walk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50511 diff changeset	979 phase = not repo.publishing()
51547 463e63aa547c stream-clone: disable gc for `_entries_walk` duration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51545 diff changeset	980 # Python is getting crazy at all the small container we creates, disabling
463e63aa547c stream-clone: disable gc for `_entries_walk` duration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51545 diff changeset	981 # the gc while we do so helps performance a lot.
463e63aa547c stream-clone: disable gc for `_entries_walk` duration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51545 diff changeset	982 with util.nogc():
463e63aa547c stream-clone: disable gc for `_entries_walk` duration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51545 diff changeset	983 entries = _walkstreamfiles(
463e63aa547c stream-clone: disable gc for `_entries_walk` duration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51545 diff changeset	984 repo,
463e63aa547c stream-clone: disable gc for `_entries_walk` duration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51545 diff changeset	985 matcher,
463e63aa547c stream-clone: disable gc for `_entries_walk` duration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51545 diff changeset	986 phase=phase,
463e63aa547c stream-clone: disable gc for `_entries_walk` duration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51545 diff changeset	987 obsolescence=includeobsmarkers,
463e63aa547c stream-clone: disable gc for `_entries_walk` duration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51545 diff changeset	988 )
463e63aa547c stream-clone: disable gc for `_entries_walk` duration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51545 diff changeset	989 for entry in entries:
463e63aa547c stream-clone: disable gc for `_entries_walk` duration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51545 diff changeset	990 yield (_srcstore, entry)
50512 06d580b8f432 stream-clone: introduce a _entries_walk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50511 diff changeset	991
51547 463e63aa547c stream-clone: disable gc for `_entries_walk` duration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51545 diff changeset	992 for name in cacheutil.cachetocopy(repo):
463e63aa547c stream-clone: disable gc for `_entries_walk` duration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51545 diff changeset	993 if repo.cachevfs.exists(name):
463e63aa547c stream-clone: disable gc for `_entries_walk` duration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51545 diff changeset	994 # not really a StoreEntry, but close enough
52359 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	995 yield (_srccache, CacheEntry(entry_path=name))
50513 43ed1f12b00a stream-clone: yield cache entry in `_entries_walk` too Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50512 diff changeset	996
50512 06d580b8f432 stream-clone: introduce a _entries_walk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50511 diff changeset	997
52695 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	998 def generatev2(repo, includes, excludes, includeobsmarkers: bool):
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	999 """Emit content for version 2 of a streaming clone.
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1000
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1001 the data stream consists the following entries:
35767 5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	1002 1) A char representing the file destination (eg: store or cache)
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	1003 2) A varint containing the length of the filename
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	1004 3) A varint containing the length of file data
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	1005 4) N bytes containing the filename (the internal, store-agnostic form)
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	1006 5) N bytes containing the file data
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1007
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1008 Returns a 3-tuple of (file count, file size, data iterator).
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1009 """
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1010
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1011 with repo.lock():
47444 2f4ca4807033 streamingclone: extract the scanning part from the generation part Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	1012 repo.ui.debug(b'scanning\n')
40339 f0e8f27768eb streamclone: pass narrowing related info in _walkstreamfiles() Pulkit Goyal <pulkit@yandex-team.ru> parents: 40338 diff changeset	1013
50632 3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2 Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50631 diff changeset	1014 entries = _entries_walk(
47444 2f4ca4807033 streamingclone: extract the scanning part from the generation part Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	1015 repo,
2f4ca4807033 streamingclone: extract the scanning part from the generation part Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	1016 includes=includes,
2f4ca4807033 streamingclone: extract the scanning part from the generation part Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	1017 excludes=excludes,
2f4ca4807033 streamingclone: extract the scanning part from the generation part Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	1018 includeobsmarkers=includeobsmarkers,
2f4ca4807033 streamingclone: extract the scanning part from the generation part Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	1019 )
50632 3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2 Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50631 diff changeset	1020 chunks = _emit2(repo, entries)
35765 56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	1021 first = next(chunks)
50632 3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2 Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50631 diff changeset	1022 file_count, total_file_size = first
52358 11484a19cd77 stream: rename all test hook point one number up Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52357 diff changeset	1023 _test_sync_point_walk_3(repo)
11484a19cd77 stream: rename all test hook point one number up Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52357 diff changeset	1024 _test_sync_point_walk_4(repo)
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1025
50632 3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2 Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50631 diff changeset	1026 return file_count, total_file_size, chunks
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1027
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	1028
52695 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	1029 def generatev3(
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	1030 repo, includes, excludes, includeobsmarkers: bool
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	1031 ) -> Iterator[bytes \| None]:
50703 0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1032 """Emit content for version 3 of a streaming clone.
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1033
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1034 the data stream consists the following:
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1035 1) A varint E containing the number of entries (can be 0), then E entries follow
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1036 2) For each entry:
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1037 2.1) The number of files in this entry (can be 0, but typically 1 or 2)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1038 2.2) For each file:
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1039 2.2.1) A char representing the file destination (eg: store or cache)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1040 2.2.2) A varint N containing the length of the filename
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1041 2.2.3) A varint M containing the length of file data
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1042 2.2.4) N bytes containing the filename (the internal, store-agnostic form)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1043 2.2.5) M bytes containing the file data
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1044
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1045 Returns the data iterator.
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1046
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1047 XXX This format is experimental and subject to change. Here is a
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1048 XXX non-exhaustive list of things this format could do or change:
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1049
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1050 - making it easier to write files in parallel
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1051 - holding the lock for a shorter time
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1052 - improving progress information
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1053 - ways to adjust the number of expected entries/files ?
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1054 """
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1055
51548 6e4c8366c5ce stream-clone: disable gc for the initial section for the v3 format Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51547 diff changeset	1056 # Python is getting crazy at all the small container we creates while
6e4c8366c5ce stream-clone: disable gc for the initial section for the v3 format Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51547 diff changeset	1057 # considering the files to preserve, disabling the gc while we do so helps
6e4c8366c5ce stream-clone: disable gc for the initial section for the v3 format Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51547 diff changeset	1058 # performance a lot.
6e4c8366c5ce stream-clone: disable gc for the initial section for the v3 format Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51547 diff changeset	1059 with repo.lock(), util.nogc():
50703 0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1060 repo.ui.debug(b'scanning\n')
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1061
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1062 entries = _entries_walk(
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1063 repo,
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1064 includes=includes,
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1065 excludes=excludes,
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1066 includeobsmarkers=includeobsmarkers,
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1067 )
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1068 chunks = _emit3(repo, list(entries))
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1069 first = next(chunks)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1070 assert first is None
52358 11484a19cd77 stream: rename all test hook point one number up Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52357 diff changeset	1071 _test_sync_point_walk_3(repo)
11484a19cd77 stream: rename all test hook point one number up Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52357 diff changeset	1072 _test_sync_point_walk_4(repo)
50703 0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1073
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1074 return chunks
50521 58adcabc295f stream-clone: introduce the notion of an experimental "v3" version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50520 diff changeset	1075
58adcabc295f stream-clone: introduce the notion of an experimental "v3" version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50520 diff changeset	1076
35767 5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	1077 @contextlib.contextmanager
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	1078 def nested(*ctxs):
39757 97f2992c26f6 streamclone: reimplement nested context manager Augie Fackler <augie@google.com> parents: 39700 diff changeset	1079 this = ctxs[0]
97f2992c26f6 streamclone: reimplement nested context manager Augie Fackler <augie@google.com> parents: 39700 diff changeset	1080 rest = ctxs[1:]
97f2992c26f6 streamclone: reimplement nested context manager Augie Fackler <augie@google.com> parents: 39700 diff changeset	1081 with this:
97f2992c26f6 streamclone: reimplement nested context manager Augie Fackler <augie@google.com> parents: 39700 diff changeset	1082 if rest:
97f2992c26f6 streamclone: reimplement nested context manager Augie Fackler <augie@google.com> parents: 39700 diff changeset	1083 with nested(*rest):
97f2992c26f6 streamclone: reimplement nested context manager Augie Fackler <augie@google.com> parents: 39700 diff changeset	1084 yield
97f2992c26f6 streamclone: reimplement nested context manager Augie Fackler <augie@google.com> parents: 39700 diff changeset	1085 else:
35767 5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	1086 yield
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	1087
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	1088
52909 3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1089 class V2Report:
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1090 """a small class to track the data we saw within the stream"""
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1091
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1092 def __init__(self):
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1093 self.byte_count = 0
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1094
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1095
52695 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	1096 def consumev2(repo, fp, filecount: int, filesize: int) -> None:
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1097 """Apply the contents from a version 2 streaming clone.
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1098
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1099 Data is read from an object that only needs to provide a ``read(size)``
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1100 method.
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1101 """
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1102 with repo.lock():
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	1103 repo.ui.status(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	1104 _(b'%d files to transfer, %s of data\n')
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	1105 % (filecount, util.bytecount(filesize))
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	1106 )
52909 3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1107 progress = repo.ui.makeprogress(
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1108 _(b'clone'),
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1109 total=filesize,
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1110 unit=_(b'bytes'),
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1111 )
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1112 start = util.timer()
52909 3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1113 report = V2Report()
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1114
35767 5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	1115 vfsmap = _makemap(repo)
47502 65c519661991 stream: double check that self.vfs is not in the vfsmap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	1116 # we keep repo.vfs out of the on purpose, ther are too many danger
65c519661991 stream: double check that self.vfs is not in the vfsmap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	1117 # there (eg: .hg/hgrc),
65c519661991 stream: double check that self.vfs is not in the vfsmap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	1118 #
65c519661991 stream: double check that self.vfs is not in the vfsmap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	1119 # this assert is duplicated (from _makemap) as author might think this
65c519661991 stream: double check that self.vfs is not in the vfsmap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	1120 # is fine, while this is really not fine.
65c519661991 stream: double check that self.vfs is not in the vfsmap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	1121 if repo.vfs in vfsmap.values():
65c519661991 stream: double check that self.vfs is not in the vfsmap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	1122 raise error.ProgrammingError(
52696 10e7adbffa8c streamclone: unbyteify string args to builtin Error classes Matt Harbison <matt_harbison@yahoo.com> parents: 52695 diff changeset	1123 'repo.vfs must not be added to vfsmap for security reasons'
47502 65c519661991 stream: double check that self.vfs is not in the vfsmap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	1124 )
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1125
52913 58baa86c7a02 stream-clone-v2: make the number of writer dependent of usage config Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52912 diff changeset	1126 cpu_profile = scmutil.get_resource_profile(repo.ui, b'cpu')
52920 aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1127 mem_profile = scmutil.get_resource_profile(repo.ui, b'memory')
52912 7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1128 threaded = repo.ui.configbool(
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1129 b"worker", b"parallel-stream-bundle-processing"
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1130 )
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1131 num_writer = repo.ui.configint(
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1132 b"worker",
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1133 b"parallel-stream-bundle-processing.num-writer",
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1134 )
52913 58baa86c7a02 stream-clone-v2: make the number of writer dependent of usage config Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52912 diff changeset	1135 if num_writer <= 0:
58baa86c7a02 stream-clone-v2: make the number of writer dependent of usage config Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52912 diff changeset	1136 num_writer = DEFAULT_NUM_WRITER[cpu_profile]
52920 aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1137 memory_target = repo.ui.configbytes(
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1138 b"worker",
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1139 b"parallel-stream-bundle-processing.memory-target",
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1140 )
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1141 if memory_target < 0:
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1142 memory_target = None
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1143 elif memory_target == 0:
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1144 memory_target = DEFAULT_MEMORY_TARGET[mem_profile]
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	1145 with repo.transaction(b'clone'):
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	1146 ctxs = (vfs.backgroundclosing(repo.ui) for vfs in vfsmap.values())
35767 5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	1147 with nested(*ctxs):
52914 d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1148 workers = []
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1149 info_queue = None
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1150 data_queue = None
52920 aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1151 mark_used = None
52914 d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1152 try:
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1153 if not threaded:
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1154 fc = _FileChunker
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1155 raw_data = fp
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1156 else:
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1157 fc = _ThreadSafeFileChunker
52920 aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1158 data_queue = _DataQueue(memory_target=memory_target)
52996 5480647c2964 stream-clone: skip the memory_target logic when usage is unlimited Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52971 diff changeset	1159 if memory_target is not None:
5480647c2964 stream-clone: skip the memory_target logic when usage is unlimited Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52971 diff changeset	1160 mark_used = data_queue.mark_used
52914 d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1161 raw_data = util.chunkbuffer(data_queue)
52912 7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1162
52914 d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1163 w = threading.Thread(
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1164 target=data_queue.fill_from,
52917 9abf173a958b stream-clone-v2: use the pass through for threaded stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52914 diff changeset	1165 args=(fp,),
52914 d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1166 )
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1167 workers.append(w)
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1168 w.start()
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1169 files = _v2_parse_files(
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1170 repo,
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1171 raw_data,
52929 5b8f6e198a6e stream-clone-v2: centralize preparation for streamed files Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52928 diff changeset	1172 vfsmap,
52914 d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1173 filecount,
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1174 progress,
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1175 report,
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1176 file_chunker=fc,
52920 aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1177 mark_used=mark_used,
52914 d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1178 )
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1179 if not threaded:
52930 22e264ac7f60 stream-clone-v2: bypass the vfs to write the file on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52929 diff changeset	1180 _write_files(files)
52914 d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1181 else:
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1182 info_queue = _FileInfoQueue(files)
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1183
52912 7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1184 for __ in range(num_writer):
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1185 w = threading.Thread(
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1186 target=_write_files,
52930 22e264ac7f60 stream-clone-v2: bypass the vfs to write the file on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52929 diff changeset	1187 args=(info_queue,),
52912 7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1188 )
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1189 workers.append(w)
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1190 w.start()
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1191 info_queue.fill()
52914 d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1192 except: # re-raises
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1193 if data_queue is not None:
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1194 data_queue.abort()
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1195 raise
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1196 finally:
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1197 # shut down all the workers
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1198 if info_queue is not None:
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1199 # this is strictly speaking one too many worker for
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1200 # this queu, but closing too many is not a problem.
52912 7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1201 info_queue.close(len(workers))
52914 d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1202 for w in workers:
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1203 w.join()
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1204
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1205 # force @filecache properties to be reloaded from
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1206 # streamclone-ed file at next access
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1207 repo.invalidate(clearfilecache=True)
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1208
38373 ef692614e601 progress: hide update(None) in a new complete() method Martin von Zweigbergk <martinvonz@google.com> parents: 38360 diff changeset	1209 progress.complete()
52703 2e82bd50978c stream: acknowledge the end of the bundle part earlier Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52702 diff changeset	1210 # acknowledge the end of the bundle2 part, this help aligning
2e82bd50978c stream: acknowledge the end of the bundle part earlier Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52702 diff changeset	1211 # sequential and parallel behavior.
2e82bd50978c stream: acknowledge the end of the bundle part earlier Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52702 diff changeset	1212 remains = fp.read(1)
2e82bd50978c stream: acknowledge the end of the bundle part earlier Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52702 diff changeset	1213 assert not remains
52909 3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1214 _report_transferred(repo, start, filecount, report.byte_count)
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1215
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1216
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1217 # iterator of chunk of bytes that constitute a file content.
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1218 FileChunksT = Iterator[bytes]
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1219 # Contains the information necessary to write stream file on disk
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1220 FileInfoT = Tuple[
52930 22e264ac7f60 stream-clone-v2: bypass the vfs to write the file on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52929 diff changeset	1221 bytes, # real fs path
22e264ac7f60 stream-clone-v2: bypass the vfs to write the file on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52929 diff changeset	1222 Optional[int], # permission to give to chmod
52909 3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1223 FileChunksT, # content
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1224 ]
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1225
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1226
52922 0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1227 class _Queue:
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1228 """a reimplementation of queue.Queue which doesn't use thread.Condition"""
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1229
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1230 def __init__(self):
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1231 self._queue = collections.deque()
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1232
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1233 # the "_lock" protect manipulation of the "_queue" deque
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1234 # the "_wait" is used to have the "get" thread waits for the
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1235 # "put" thread when the queue is empty.
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1236 #
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1237 # This is similar to the "threading.Condition", but without the absurd
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1238 # slowness of the stdlib implementation.
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1239 #
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1240 # the "_wait" is always released while holding the "_lock".
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1241 self._lock = threading.Lock()
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1242 self._wait = threading.Lock()
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1243
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1244 def put(self, item):
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1245 with self._lock:
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1246 self._queue.append(item)
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1247 # if anyone is waiting on item, unblock it.
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1248 if self._wait.locked():
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1249 self._wait.release()
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1250
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1251 def get(self):
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1252 with self._lock:
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1253 while len(self._queue) == 0:
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1254 # "arm" the waiting lock
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1255 self._wait.acquire(blocking=False)
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1256 # release the lock to let other touch the queue
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1257 # (especially the put call we wait on)
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1258 self._lock.release()
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1259 # wait for for a `put` call to release the lock
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1260 self._wait.acquire()
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1261 # grab the lock to look at a possible available value
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1262 self._lock.acquire()
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1263 # disarm the lock if necessary.
52923 f1ac5117459b stream-clone-v2: avoid waking further thread if only 1 item is available Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52922 diff changeset	1264 #
f1ac5117459b stream-clone-v2: avoid waking further thread if only 1 item is available Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52922 diff changeset	1265 # If the queue only constains one item, keep the _wait lock
f1ac5117459b stream-clone-v2: avoid waking further thread if only 1 item is available Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52922 diff changeset	1266 # armed, as there is no need to wake another waiter anyway.
f1ac5117459b stream-clone-v2: avoid waking further thread if only 1 item is available Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52922 diff changeset	1267 if self._wait.locked() and len(self._queue) > 1:
52922 0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1268 self._wait.release()
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1269 return self._queue.popleft()
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1270
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1271
52914 d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1272 class _DataQueue:
52920 aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1273 """A queue passing data from the bundle stream to other thread
52914 d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1274
52996 5480647c2964 stream-clone: skip the memory_target logic when usage is unlimited Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52971 diff changeset	1275 It has a "memory_target" optional parameter to avoid buffering too much
52920 aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1276 information. The implementation is not exact and the memory target might be
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1277 exceed for a time in some situation.
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1278 """
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1279
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1280 def __init__(self, memory_target=None):
52922 0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1281 self._q = _Queue()
52914 d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1282 self._abort = False
52920 aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1283 self._memory_target = memory_target
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1284 if self._memory_target is not None and self._memory_target <= 0:
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1285 raise error.ProgrammingError("memory target should be > 0")
52921 363914ba328d stream-clone-v2: no longer use the stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52920 diff changeset	1286
363914ba328d stream-clone-v2: no longer use the stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52920 diff changeset	1287 # the "_lock" protect manipulation of the _current_used" variable
363914ba328d stream-clone-v2: no longer use the stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52920 diff changeset	1288 # the "_wait" is used to have the "reading" thread waits for the
363914ba328d stream-clone-v2: no longer use the stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52920 diff changeset	1289 # "using" thread when the buffer is full.
363914ba328d stream-clone-v2: no longer use the stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52920 diff changeset	1290 #
363914ba328d stream-clone-v2: no longer use the stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52920 diff changeset	1291 # This is similar to the "threading.Condition", but without the absurd
363914ba328d stream-clone-v2: no longer use the stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52920 diff changeset	1292 # slowness of the stdlib implementation.
363914ba328d stream-clone-v2: no longer use the stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52920 diff changeset	1293 #
363914ba328d stream-clone-v2: no longer use the stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52920 diff changeset	1294 # the "_wait" is always released while holding the "_lock".
363914ba328d stream-clone-v2: no longer use the stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52920 diff changeset	1295 self._lock = threading.Lock()
363914ba328d stream-clone-v2: no longer use the stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52920 diff changeset	1296 self._wait = threading.Lock()
52920 aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1297 # only the stream reader touch this, it is find to touch without the lock
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1298 self._current_read = 0
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1299 # do not touch this without the lock
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1300 self._current_used = 0
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1301
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1302 def _has_free_space(self):
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1303 """True if more data can be read without further exceeding memory target
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1304
52921 363914ba328d stream-clone-v2: no longer use the stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52920 diff changeset	1305 Must be called under the lock.
52920 aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1306 """
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1307 if self._memory_target is None:
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1308 # Ideally we should not even get into the locking business in that
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1309 # case, but we keep the implementation simple for now.
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1310 return True
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1311 return (self._current_read - self._current_used) < self._memory_target
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1312
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1313 def mark_used(self, offset):
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1314 """Notify we have used the buffer up to "offset"
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1315
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1316 This is meant to be used from another thread than the one filler the queue.
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1317 """
52996 5480647c2964 stream-clone: skip the memory_target logic when usage is unlimited Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52971 diff changeset	1318 if self._memory_target is not None:
5480647c2964 stream-clone: skip the memory_target logic when usage is unlimited Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52971 diff changeset	1319 with self._lock:
5480647c2964 stream-clone: skip the memory_target logic when usage is unlimited Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52971 diff changeset	1320 if offset > self._current_used:
5480647c2964 stream-clone: skip the memory_target logic when usage is unlimited Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52971 diff changeset	1321 self._current_used = offset
5480647c2964 stream-clone: skip the memory_target logic when usage is unlimited Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52971 diff changeset	1322 # If the reader is waiting for room, unblock it.
5480647c2964 stream-clone: skip the memory_target logic when usage is unlimited Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52971 diff changeset	1323 if self._wait.locked() and self._has_free_space():
5480647c2964 stream-clone: skip the memory_target logic when usage is unlimited Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52971 diff changeset	1324 self._wait.release()
52914 d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1325
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1326 def fill_from(self, data):
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1327 """fill the data queue from a bundle2 part object
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1328
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1329 This is meant to be called by the data reading thread
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1330 """
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1331 q = self._q
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1332 try:
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1333 for item in data:
52920 aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1334 self._current_read += len(item)
52914 d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1335 q.put(item)
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1336 if self._abort:
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1337 break
52996 5480647c2964 stream-clone: skip the memory_target logic when usage is unlimited Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52971 diff changeset	1338 if self._memory_target is not None:
5480647c2964 stream-clone: skip the memory_target logic when usage is unlimited Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52971 diff changeset	1339 with self._lock:
5480647c2964 stream-clone: skip the memory_target logic when usage is unlimited Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52971 diff changeset	1340 while not self._has_free_space():
5480647c2964 stream-clone: skip the memory_target logic when usage is unlimited Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52971 diff changeset	1341 # make sure the _wait lock is locked
5480647c2964 stream-clone: skip the memory_target logic when usage is unlimited Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52971 diff changeset	1342 # this is done under lock, so there case be no race with the release logic
5480647c2964 stream-clone: skip the memory_target logic when usage is unlimited Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52971 diff changeset	1343 self._wait.acquire(blocking=False)
5480647c2964 stream-clone: skip the memory_target logic when usage is unlimited Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52971 diff changeset	1344 self._lock.release()
5480647c2964 stream-clone: skip the memory_target logic when usage is unlimited Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52971 diff changeset	1345 # acquiring the lock will block until some other thread release it.
5480647c2964 stream-clone: skip the memory_target logic when usage is unlimited Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52971 diff changeset	1346 self._wait.acquire()
5480647c2964 stream-clone: skip the memory_target logic when usage is unlimited Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52971 diff changeset	1347 # lets dive into the locked section again
5480647c2964 stream-clone: skip the memory_target logic when usage is unlimited Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52971 diff changeset	1348 self._lock.acquire()
5480647c2964 stream-clone: skip the memory_target logic when usage is unlimited Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52971 diff changeset	1349 # make sure we release the lock we just grabed if
5480647c2964 stream-clone: skip the memory_target logic when usage is unlimited Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52971 diff changeset	1350 # needed.
5480647c2964 stream-clone: skip the memory_target logic when usage is unlimited Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52971 diff changeset	1351 if self._wait.locked():
5480647c2964 stream-clone: skip the memory_target logic when usage is unlimited Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52971 diff changeset	1352 self._wait.release()
52914 d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1353 finally:
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1354 q.put(None)
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1355
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1356 def __iter__(self):
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1357 """Iterate over the bundle chunkgs
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1358
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1359 This is meant to be called by the data parsing thread."""
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1360 q = self._q
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1361 while (i := q.get()) is not None:
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1362 yield i
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1363 if self._abort:
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1364 break
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1365
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1366 def abort(self):
52920 aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1367 """stop the data-reading thread and interrupt the comsuming iteration
52914 d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1368
52920 aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1369 This is meant to be called on errors.
52914 d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1370 """
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1371 self._abort = True
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1372 self._q.put(None)
52996 5480647c2964 stream-clone: skip the memory_target logic when usage is unlimited Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52971 diff changeset	1373 if self._memory_target is not None:
5480647c2964 stream-clone: skip the memory_target logic when usage is unlimited Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52971 diff changeset	1374 with self._lock:
5480647c2964 stream-clone: skip the memory_target logic when usage is unlimited Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52971 diff changeset	1375 # make sure we unstuck the reader thread.
5480647c2964 stream-clone: skip the memory_target logic when usage is unlimited Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52971 diff changeset	1376 if self._wait.locked():
5480647c2964 stream-clone: skip the memory_target logic when usage is unlimited Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52971 diff changeset	1377 self._wait.release()
52914 d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1378
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1379
52912 7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1380 class _FileInfoQueue:
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1381 """A thread-safe queue to passer parsed file information to the writers"""
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1382
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1383 def __init__(self, info: Iterable[FileInfoT]):
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1384 self._info = info
52922 0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1385 self._q = _Queue()
52912 7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1386
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1387 def fill(self):
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1388 """iterate over the parsed information to file the queue
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1389
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1390 This is meant to be call from the thread parsing the stream information.
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1391 """
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1392 q = self._q
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1393 for i in self._info:
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1394 q.put(i)
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1395
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1396 def close(self, number_worker):
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1397 """signal all the workers that we no longer have any file info coming
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1398
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1399 Called from the thread parsing the stream information (and/or the main
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1400 thread if different).
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1401 """
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1402 for __ in range(number_worker):
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1403 self._q.put(None)
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1404
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1405 def __iter__(self):
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1406 """iterate over the available file info
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1407
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1408 This is meant to be called from the writer threads.
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1409 """
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1410 q = self._q
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1411 while (i := q.get()) is not None:
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1412 yield i
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1413
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1414
52911 307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52910 diff changeset	1415 class _FileChunker:
52909 3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1416 """yield the chunk that constitute a file
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1417
52912 7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1418 This class exists as the counterpart of the threaded version and
52909 3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1419 would not be very useful on its own.
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1420 """
52911 307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52910 diff changeset	1421
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52910 diff changeset	1422 def __init__(
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52910 diff changeset	1423 self,
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52910 diff changeset	1424 fp: bundle2mod.unbundlepart,
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52910 diff changeset	1425 data_len: int,
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52910 diff changeset	1426 progress: scmutil.progress,
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52910 diff changeset	1427 report: V2Report,
52920 aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1428 mark_used: Optional[Callable[[int], None]] = None,
52911 307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52910 diff changeset	1429 ):
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52910 diff changeset	1430 self.report = report
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52910 diff changeset	1431 self.progress = progress
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52910 diff changeset	1432 self._chunks = util.filechunkiter(fp, limit=data_len)
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52910 diff changeset	1433
52912 7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1434 def fill(self) -> None:
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1435 """Do nothing in non-threading context"""
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1436
52911 307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52910 diff changeset	1437 def __iter__(self) -> FileChunksT:
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52910 diff changeset	1438 for chunk in self._chunks:
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52910 diff changeset	1439 self.report.byte_count += len(chunk)
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52910 diff changeset	1440 self.progress.increment(step=len(chunk))
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52910 diff changeset	1441 yield chunk
52909 3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1442
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1443
52912 7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1444 class _ThreadSafeFileChunker(_FileChunker):
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1445 """yield the chunk that constitute a file
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1446
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1447 Make sure you call the "fill" function in the main thread to read the
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1448 right data at the right time.
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1449 """
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1450
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1451 def __init__(
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1452 self,
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1453 fp: bundle2mod.unbundlepart,
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1454 data_len: int,
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1455 progress: scmutil.progress,
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1456 report: V2Report,
52920 aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1457 mark_used: Optional[Callable[[int], None]] = None,
52912 7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1458 ):
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1459 super().__init__(fp, data_len, progress, report)
52920 aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1460 self._fp = fp
52922 0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1461 self._queue = _Queue()
52920 aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1462 self._mark_used = mark_used
52912 7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1463
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1464 def fill(self) -> None:
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1465 """fill the file chunker queue with data read from the stream
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1466
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1467 This is meant to be called from the thread parsing information (and
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1468 consuming the stream data).
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1469 """
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1470 try:
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1471 for chunk in super().__iter__():
52920 aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1472 offset = self._fp.tell()
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1473 self._queue.put((chunk, offset))
52912 7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1474 finally:
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1475 self._queue.put(None)
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1476
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1477 def __iter__(self) -> FileChunksT:
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1478 """Iterate over all the file chunk
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1479
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1480 This is meant to be called from the writer threads.
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1481 """
52920 aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1482 while (info := self._queue.get()) is not None:
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1483 chunk, offset = info
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1484 if self._mark_used is not None:
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1485 self._mark_used(offset)
52912 7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1486 yield chunk
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1487
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1488
52925 119cddd02b8c stream-clone-v2: simplify the handling of small file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52923 diff changeset	1489 def _trivial_file(
119cddd02b8c stream-clone-v2: simplify the handling of small file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52923 diff changeset	1490 chunk: bytes,
119cddd02b8c stream-clone-v2: simplify the handling of small file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52923 diff changeset	1491 mark_used: Optional[Callable[[int], None]],
119cddd02b8c stream-clone-v2: simplify the handling of small file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52923 diff changeset	1492 offset: int,
119cddd02b8c stream-clone-v2: simplify the handling of small file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52923 diff changeset	1493 ) -> FileChunksT:
119cddd02b8c stream-clone-v2: simplify the handling of small file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52923 diff changeset	1494 """used for single chunk file,"""
119cddd02b8c stream-clone-v2: simplify the handling of small file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52923 diff changeset	1495 if mark_used is not None:
119cddd02b8c stream-clone-v2: simplify the handling of small file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52923 diff changeset	1496 mark_used(offset)
119cddd02b8c stream-clone-v2: simplify the handling of small file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52923 diff changeset	1497 yield chunk
119cddd02b8c stream-clone-v2: simplify the handling of small file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52923 diff changeset	1498
119cddd02b8c stream-clone-v2: simplify the handling of small file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52923 diff changeset	1499
52909 3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1500 def _v2_parse_files(
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1501 repo,
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1502 fp: bundle2mod.unbundlepart,
52929 5b8f6e198a6e stream-clone-v2: centralize preparation for streamed files Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52928 diff changeset	1503 vfs_map,
52909 3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1504 file_count: int,
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1505 progress: scmutil.progress,
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1506 report: V2Report,
52911 307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52910 diff changeset	1507 file_chunker: Type[_FileChunker] = _FileChunker,
52920 aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1508 mark_used: Optional[Callable[[int], None]] = None,
52909 3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1509 ) -> Iterator[FileInfoT]:
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1510 """do the "stream-parsing" part of stream v2
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1511
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1512 The parsed information are yield result for consumption by the "writer"
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1513 """
52929 5b8f6e198a6e stream-clone-v2: centralize preparation for streamed files Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52928 diff changeset	1514 known_dirs = set() # set of directory that we know to exists
52909 3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1515 progress.update(0)
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1516 for i in range(file_count):
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1517 src = util.readexactly(fp, 1)
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1518 namelen = util.uvarintdecodestream(fp)
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1519 datalen = util.uvarintdecodestream(fp)
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1520
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1521 name = util.readexactly(fp, namelen)
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1522
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1523 if repo.ui.debugflag:
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1524 repo.ui.debug(
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1525 b'adding [%s] %s (%s)\n' % (src, name, util.bytecount(datalen))
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1526 )
52929 5b8f6e198a6e stream-clone-v2: centralize preparation for streamed files Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52928 diff changeset	1527 vfs = vfs_map[src]
52930 22e264ac7f60 stream-clone-v2: bypass the vfs to write the file on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52929 diff changeset	1528 path, mode = vfs.prepare_streamed_file(name, known_dirs)
52925 119cddd02b8c stream-clone-v2: simplify the handling of small file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52923 diff changeset	1529 if datalen <= util.DEFAULT_FILE_CHUNK:
119cddd02b8c stream-clone-v2: simplify the handling of small file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52923 diff changeset	1530 c = fp.read(datalen)
119cddd02b8c stream-clone-v2: simplify the handling of small file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52923 diff changeset	1531 offset = fp.tell()
119cddd02b8c stream-clone-v2: simplify the handling of small file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52923 diff changeset	1532 report.byte_count += len(c)
119cddd02b8c stream-clone-v2: simplify the handling of small file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52923 diff changeset	1533 progress.increment(step=len(c))
119cddd02b8c stream-clone-v2: simplify the handling of small file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52923 diff changeset	1534 chunks = _trivial_file(c, mark_used, offset)
52930 22e264ac7f60 stream-clone-v2: bypass the vfs to write the file on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52929 diff changeset	1535 yield (path, mode, iter(chunks))
52925 119cddd02b8c stream-clone-v2: simplify the handling of small file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52923 diff changeset	1536 else:
119cddd02b8c stream-clone-v2: simplify the handling of small file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52923 diff changeset	1537 chunks = file_chunker(
119cddd02b8c stream-clone-v2: simplify the handling of small file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52923 diff changeset	1538 fp,
119cddd02b8c stream-clone-v2: simplify the handling of small file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52923 diff changeset	1539 datalen,
119cddd02b8c stream-clone-v2: simplify the handling of small file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52923 diff changeset	1540 progress,
119cddd02b8c stream-clone-v2: simplify the handling of small file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52923 diff changeset	1541 report,
119cddd02b8c stream-clone-v2: simplify the handling of small file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52923 diff changeset	1542 mark_used=mark_used,
119cddd02b8c stream-clone-v2: simplify the handling of small file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52923 diff changeset	1543 )
52930 22e264ac7f60 stream-clone-v2: bypass the vfs to write the file on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52929 diff changeset	1544 yield (path, mode, iter(chunks))
52925 119cddd02b8c stream-clone-v2: simplify the handling of small file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52923 diff changeset	1545 # make sure we read all the chunk before moving to the next file
119cddd02b8c stream-clone-v2: simplify the handling of small file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52923 diff changeset	1546 chunks.fill()
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1547
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	1548
52930 22e264ac7f60 stream-clone-v2: bypass the vfs to write the file on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52929 diff changeset	1549 def _write_files(info: Iterable[FileInfoT]):
52910 70306aefa52b stream-clone-v2: extract the file writing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52909 diff changeset	1550 """write files from parsed data"""
52971 c636f2835c95 stream-bundle: make sure to open the file in binary mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52970 diff changeset	1551 io_flags = os.O_WRONLY \| os.O_CREAT
c636f2835c95 stream-bundle: make sure to open the file in binary mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52970 diff changeset	1552 if pycompat.iswindows:
c636f2835c95 stream-bundle: make sure to open the file in binary mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52970 diff changeset	1553 io_flags \|= os.O_BINARY
52930 22e264ac7f60 stream-clone-v2: bypass the vfs to write the file on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52929 diff changeset	1554 for path, mode, data in info:
52970 e4ff37b5317c stream-bundle: simple use of `mode` argument of `os.open` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52931 diff changeset	1555 if mode is None:
52971 c636f2835c95 stream-bundle: make sure to open the file in binary mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52970 diff changeset	1556 fd = os.open(path, io_flags)
52970 e4ff37b5317c stream-bundle: simple use of `mode` argument of `os.open` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52931 diff changeset	1557 else:
52971 c636f2835c95 stream-bundle: make sure to open the file in binary mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52970 diff changeset	1558 fd = os.open(path, io_flags, mode=mode)
52931 c124308e3cd4 stream-clone-v2: directly use the os module for file operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52930 diff changeset	1559 try:
52910 70306aefa52b stream-clone-v2: extract the file writing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52909 diff changeset	1560 for chunk in data:
52931 c124308e3cd4 stream-clone-v2: directly use the os module for file operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52930 diff changeset	1561 written = os.write(fd, chunk)
52928 7fc882f7fada stream-clone-v2: disable buffering when writing the files Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52926 diff changeset	1562 # write missing pieces if the write was interrupted
7fc882f7fada stream-clone-v2: disable buffering when writing the files Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52926 diff changeset	1563 while written < len(chunk):
52931 c124308e3cd4 stream-clone-v2: directly use the os module for file operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52930 diff changeset	1564 written = os.write(fd, chunk[written:])
c124308e3cd4 stream-clone-v2: directly use the os module for file operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52930 diff changeset	1565 finally:
c124308e3cd4 stream-clone-v2: directly use the os module for file operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52930 diff changeset	1566 os.close(fd)
52910 70306aefa52b stream-clone-v2: extract the file writing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52909 diff changeset	1567
70306aefa52b stream-clone-v2: extract the file writing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52909 diff changeset	1568
52695 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	1569 def consumev3(repo, fp) -> None:
50703 0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1570 """Apply the contents from a version 3 streaming clone.
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1571
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1572 Data is read from an object that only needs to provide a ``read(size)``
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1573 method.
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1574 """
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1575 with repo.lock():
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1576 start = util.timer()
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1577
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1578 entrycount = util.uvarintdecodestream(fp)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1579 repo.ui.status(_(b'%d entries to transfer\n') % (entrycount))
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1580
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1581 progress = repo.ui.makeprogress(
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1582 _(b'clone'),
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1583 total=entrycount,
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1584 unit=_(b'entries'),
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1585 )
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1586 progress.update(0)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1587 bytes_transferred = 0
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1588
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1589 vfsmap = _makemap(repo)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1590 # we keep repo.vfs out of the on purpose, there are too many dangers
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1591 # there (eg: .hg/hgrc),
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1592 #
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1593 # this assert is duplicated (from _makemap) as authors might think this
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1594 # is fine, while this is really not fine.
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1595 if repo.vfs in vfsmap.values():
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1596 raise error.ProgrammingError(
52696 10e7adbffa8c streamclone: unbyteify string args to builtin Error classes Matt Harbison <matt_harbison@yahoo.com> parents: 52695 diff changeset	1597 'repo.vfs must not be added to vfsmap for security reasons'
50703 0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1598 )
52702 ca17d31624ac stream: report number of file written then information is available Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52701 diff changeset	1599 total_file_count = 0
50703 0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1600 with repo.transaction(b'clone'):
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1601 ctxs = (vfs.backgroundclosing(repo.ui) for vfs in vfsmap.values())
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1602 with nested(*ctxs):
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1603 for i in range(entrycount):
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1604 filecount = util.uvarintdecodestream(fp)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1605 if filecount == 0:
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1606 if repo.ui.debugflag:
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1607 repo.ui.debug(b'entry with no files [%d]\n' % (i))
52702 ca17d31624ac stream: report number of file written then information is available Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52701 diff changeset	1608 total_file_count += filecount
50703 0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1609 for i in range(filecount):
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1610 src = util.readexactly(fp, 1)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1611 vfs = vfsmap[src]
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1612 namelen = util.uvarintdecodestream(fp)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1613 datalen = util.uvarintdecodestream(fp)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1614
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1615 name = util.readexactly(fp, namelen)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1616
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1617 if repo.ui.debugflag:
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1618 msg = b'adding [%s] %s (%s)\n'
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1619 msg %= (src, name, util.bytecount(datalen))
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1620 repo.ui.debug(msg)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1621 bytes_transferred += datalen
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1622
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1623 with vfs(name, b'w') as ofp:
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1624 for chunk in util.filechunkiter(fp, limit=datalen):
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1625 ofp.write(chunk)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1626 progress.increment(step=1)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1627
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1628 # force @filecache properties to be reloaded from
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1629 # streamclone-ed file at next access
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1630 repo.invalidate(clearfilecache=True)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1631
52701 6feb3b3029b5 stream: consistently close progress before reporting time Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52700 diff changeset	1632 progress.complete()
52702 ca17d31624ac stream: report number of file written then information is available Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52701 diff changeset	1633 _report_transferred(repo, start, total_file_count, bytes_transferred)
50703 0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1634
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1635
52695 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	1636 def applybundlev2(
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	1637 repo, fp, filecount: int, filesize: int, requirements: Iterable[bytes]
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	1638 ) -> None:
39700 b10d145837bc localrepo: extract resolving of opener options to standalone functions Gregory Szorc <gregory.szorc@gmail.com> parents: 38783 diff changeset	1639 from . import localrepo
b10d145837bc localrepo: extract resolving of opener options to standalone functions Gregory Szorc <gregory.szorc@gmail.com> parents: 38783 diff changeset	1640
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1641 missingreqs = [r for r in requirements if r not in repo.supported]
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1642 if missingreqs:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	1643 raise error.Abort(
43117 8ff1ecfadcd1 cleanup: join string literals that are already on one line Martin von Zweigbergk <martinvonz@google.com> parents: 43085 diff changeset	1644 _(b'unable to apply stream clone: unsupported format: %s')
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	1645 % b', '.join(sorted(missingreqs))
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	1646 )
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1647
52926 34fa51c25112 stream-clone-v2: disable the garbage collector during consumption Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52925 diff changeset	1648 with util.nogc():
34fa51c25112 stream-clone-v2: disable the garbage collector during consumption Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52925 diff changeset	1649 consumev2(repo, fp, filecount, filesize)
35804 2d3e486d09d0 streamclone: move requirement update into consumev2 Boris Feld <boris.feld@octobus.net> parents: 35803 diff changeset	1650
48596 739f2ca3aa3f stream-clone: factor computation of new clone requirement out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47878 diff changeset	1651 repo.requirements = new_stream_clone_requirements(
739f2ca3aa3f stream-clone: factor computation of new clone requirement out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47878 diff changeset	1652 repo.requirements,
739f2ca3aa3f stream-clone: factor computation of new clone requirement out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47878 diff changeset	1653 requirements,
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	1654 )
39700 b10d145837bc localrepo: extract resolving of opener options to standalone functions Gregory Szorc <gregory.szorc@gmail.com> parents: 38783 diff changeset	1655 repo.svfs.options = localrepo.resolvestorevfsoptions(
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	1656 repo.ui, repo.requirements, repo.features
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	1657 )
45106 a03c177a4679 scmutil: add writereporequirements() and route requires writing through it Pulkit Goyal <7895pulkit@gmail.com> parents: 43117 diff changeset	1658 scmutil.writereporequirements(repo)
48693 de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48681 diff changeset	1659 nodemap.post_stream_cleanup(repo)
47447 377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1660
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1661
52695 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	1662 def applybundlev3(repo, fp, requirements: Iterable[bytes]) -> None:
50703 0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1663 from . import localrepo
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1664
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1665 missingreqs = [r for r in requirements if r not in repo.supported]
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1666 if missingreqs:
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1667 msg = _(b'unable to apply stream clone: unsupported format: %s')
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1668 msg %= b', '.join(sorted(missingreqs))
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1669 raise error.Abort(msg)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1670
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1671 consumev3(repo, fp)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1672
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1673 repo.requirements = new_stream_clone_requirements(
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1674 repo.requirements,
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1675 requirements,
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1676 )
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1677 repo.svfs.options = localrepo.resolvestorevfsoptions(
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1678 repo.ui, repo.requirements, repo.features
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1679 )
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1680 scmutil.writereporequirements(repo)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1681 nodemap.post_stream_cleanup(repo)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1682
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1683
52695 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	1684 def _copy_files(src_vfs_map, dst_vfs_map, entries, progress) -> bool:
47447 377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1685 hardlink = [True]
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1686
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1687 def copy_used():
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1688 hardlink[0] = False
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1689 progress.topic = _(b'copying')
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1690
52359 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	1691 for k, path, optional in entries:
47447 377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1692 src_vfs = src_vfs_map[k]
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1693 dst_vfs = dst_vfs_map[k]
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1694 src_path = src_vfs.join(path)
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1695 dst_path = dst_vfs.join(path)
47871 132525ead0db clone: properly create target directories during local clone (issue6581) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47788 diff changeset	1696 # We cannot use dirname and makedirs of dst_vfs here because the store
132525ead0db clone: properly create target directories during local clone (issue6581) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47788 diff changeset	1697 # encoding confuses them. See issue 6581 for details.
132525ead0db clone: properly create target directories during local clone (issue6581) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47788 diff changeset	1698 dirname = os.path.dirname(dst_path)
132525ead0db clone: properly create target directories during local clone (issue6581) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47788 diff changeset	1699 if not os.path.exists(dirname):
132525ead0db clone: properly create target directories during local clone (issue6581) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47788 diff changeset	1700 util.makedirs(dirname)
47447 377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1701 dst_vfs.register_file(path)
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1702 # XXX we could use the #nb_bytes argument.
52359 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	1703 try:
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	1704 util.copyfile(
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	1705 src_path,
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	1706 dst_path,
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	1707 hardlink=hardlink[0],
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	1708 no_hardlink_cb=copy_used,
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	1709 check_fs_hardlink=False,
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	1710 )
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	1711 except FileNotFoundError:
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	1712 if not optional:
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	1713 raise
47447 377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1714 progress.increment()
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1715 return hardlink[0]
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1716
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1717
52695 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	1718 def local_copy(src_repo, dest_repo) -> None:
47447 377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1719 """copy all content from one local repository to another
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1720
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1721 This is useful for local clone"""
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1722 src_store_requirements = {
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1723 r
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1724 for r in src_repo.requirements
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1725 if r not in requirementsmod.WORKING_DIR_REQUIREMENTS
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1726 }
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1727 dest_store_requirements = {
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1728 r
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1729 for r in dest_repo.requirements
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1730 if r not in requirementsmod.WORKING_DIR_REQUIREMENTS
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1731 }
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1732 assert src_store_requirements == dest_store_requirements
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1733
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1734 with dest_repo.lock():
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1735 with src_repo.lock():
47448 d370256636fe clone: also report the bookmark file as copied Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47447 diff changeset	1736 # bookmark is not integrated to the streaming as it might use the
d370256636fe clone: also report the bookmark file as copied Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47447 diff changeset	1737 # `repo.vfs` and they are too many sentitive data accessible
d370256636fe clone: also report the bookmark file as copied Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47447 diff changeset	1738 # through `repo.vfs` to expose it to streaming clone.
d370256636fe clone: also report the bookmark file as copied Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47447 diff changeset	1739 src_book_vfs = bookmarks.bookmarksvfs(src_repo)
d370256636fe clone: also report the bookmark file as copied Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47447 diff changeset	1740 srcbookmarks = src_book_vfs.join(b'bookmarks')
d370256636fe clone: also report the bookmark file as copied Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47447 diff changeset	1741 bm_count = 0
d370256636fe clone: also report the bookmark file as copied Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47447 diff changeset	1742 if os.path.exists(srcbookmarks):
d370256636fe clone: also report the bookmark file as copied Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47447 diff changeset	1743 bm_count = 1
d370256636fe clone: also report the bookmark file as copied Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47447 diff changeset	1744
50630 f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50627 diff changeset	1745 entries = _entries_walk(
47447 377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1746 src_repo,
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1747 includes=None,
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1748 excludes=None,
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1749 includeobsmarkers=True,
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1750 )
50630 f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50627 diff changeset	1751 entries = list(entries)
47447 377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1752 src_vfs_map = _makemap(src_repo)
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1753 dest_vfs_map = _makemap(dest_repo)
50630 f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50627 diff changeset	1754 total_files = sum(len(e[1].files()) for e in entries) + bm_count
47447 377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1755 progress = src_repo.ui.makeprogress(
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1756 topic=_(b'linking'),
50630 f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50627 diff changeset	1757 total=total_files,
47447 377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1758 unit=_(b'files'),
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1759 )
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1760 # copy files
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1761 #
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1762 # We could copy the full file while the source repository is locked
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1763 # and the other one without the lock. However, in the linking case,
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1764 # this would also requires checks that nobody is appending any data
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1765 # to the files while we do the clone, so this is not done yet. We
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1766 # could do this blindly when copying files.
50630 f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50627 diff changeset	1767 files = [
52359 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	1768 (vfs_key, f.unencoded_path, f.optional)
50630 f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50627 diff changeset	1769 for vfs_key, e in entries
f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50627 diff changeset	1770 for f in e.files()
f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50627 diff changeset	1771 ]
47447 377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1772 hardlink = _copy_files(src_vfs_map, dest_vfs_map, files, progress)
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1773
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1774 # copy bookmarks over
47448 d370256636fe clone: also report the bookmark file as copied Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47447 diff changeset	1775 if bm_count:
d370256636fe clone: also report the bookmark file as copied Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47447 diff changeset	1776 dst_book_vfs = bookmarks.bookmarksvfs(dest_repo)
d370256636fe clone: also report the bookmark file as copied Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47447 diff changeset	1777 dstbookmarks = dst_book_vfs.join(b'bookmarks')
47447 377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1778 util.copyfile(srcbookmarks, dstbookmarks)
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1779 progress.complete()
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1780 if hardlink:
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1781 msg = b'linked %d files\n'
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1782 else:
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1783 msg = b'copied %d files\n'
50630 f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50627 diff changeset	1784 src_repo.ui.debug(msg % total_files)
47447 377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1785
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1786 with dest_repo.transaction(b"localclone") as tr:
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1787 dest_repo.store.write(tr)
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1788
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1789 # clean up transaction file as they do not make sense
50288 d89eecf9605e undo-files: no longer pass the `repo` to `cleanup_undo_files` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50286 diff changeset	1790 transaction.cleanup_undo_files(dest_repo.ui.warn, dest_repo.vfs_map)

Mercurial > public > mercurial-scm > hg

annotate mercurial/streamclone.py @ 52997:e75ed9ae5fb9