mercurial-scm/hg: mercurial/streamclone.py annotate

annotate mercurial/streamclone.py @ 52644:e627cc25b6f3

pyupgrade: rewrite `yield` statements in a loop to `yield from` This is the `legacy` fixer in `pyupgrade`, with the `yield` statement yielding loop commented back in. This seems to help pytype in some cases, and hurt it in others. But that can be manually fixed later. Note that it's possibly buggy in that it aggressively changed `import-checker.py` to `yield from 'fcntl', 'grp', 'pwd', 'select', 'termios': # Unix only`, which is invalid syntax. Possibly it needed help from the token fixer that I've disabled locally (because that wants to make a bunch of unrelated changes). Just change those few places to yield from a list, to avoid having to constantly revert that.

author	Matt Harbison <matt_harbison@yahoo.com>
date	Sun, 05 Jan 2025 22:26:16 -0500
parents	24ee91ba9aa8
children	279e217d6041

rev	line source
26441 56527b886d1d streamclone: move applystreamclone() from localrepo.py Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1 # streamclone.py - producing and consuming streaming repository data
56527b886d1d streamclone: move applystreamclone() from localrepo.py Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	2 #
56527b886d1d streamclone: move applystreamclone() from localrepo.py Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	3 # Copyright 2015 Gregory Szorc <gregory.szorc@gmail.com>
56527b886d1d streamclone: move applystreamclone() from localrepo.py Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	4 #
56527b886d1d streamclone: move applystreamclone() from localrepo.py Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	5 # This software may be used and distributed according to the terms of the
56527b886d1d streamclone: move applystreamclone() from localrepo.py Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	6 # GNU General Public License version 2 or any later version.
56527b886d1d streamclone: move applystreamclone() from localrepo.py Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	7
51859 f4733654f144 typing: add `from __future__ import annotations` to most files Matt Harbison <matt_harbison@yahoo.com> parents: 51699 diff changeset	8 from __future__ import annotations
26441 56527b886d1d streamclone: move applystreamclone() from localrepo.py Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	9
35765 56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	10 import contextlib
52359 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	11 import errno
35765 56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	12 import os
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	13 import struct
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	14
52359 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	15 from typing import Optional
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	16
26442 ef8d27f53204 streamclone: move stream_in() from localrepo Gregory Szorc <gregory.szorc@gmail.com> parents: 26441 diff changeset	17 from .i18n import _
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	18 from .interfaces import repository
26441 56527b886d1d streamclone: move applystreamclone() from localrepo.py Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	19 from . import (
47447 377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	20 bookmarks,
50520 a6543983b8f4 stream-clone: check is a compatible protocol can be found Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50519 diff changeset	21 bundle2 as bundle2mod,
35767 5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	22 cacheutil,
26442 ef8d27f53204 streamclone: move stream_in() from localrepo Gregory Szorc <gregory.szorc@gmail.com> parents: 26441 diff changeset	23 error,
40339 f0e8f27768eb streamclone: pass narrowing related info in _walkstreamfiles() Pulkit Goyal <pulkit@yandex-team.ru> parents: 40338 diff changeset	24 narrowspec,
32744 33b7283a3828 streamclone: consider secret changesets (BC) (issue5589) Gregory Szorc <gregory.szorc@gmail.com> parents: 32259 diff changeset	25 phases,
38164 aac4be30e250 py3: wrap tempfile.mkstemp() to use bytes path Yuya Nishihara <yuya@tcha.org> parents: 37638 diff changeset	26 pycompat,
46626 ee91966aec0f requirements: add constant for revlog v1 requirement Rapha?l Gom?s <rgomes@octobus.net> parents: 45106 diff changeset	27 requirements as requirementsmod,
45106 a03c177a4679 scmutil: add writereporequirements() and route requires writing through it Pulkit Goyal <7895pulkit@gmail.com> parents: 43117 diff changeset	28 scmutil,
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	29 store,
50286 3d0b5760851c undo-files: move the undo cleanup code in the transaction module Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50279 diff changeset	30 transaction,
26442 ef8d27f53204 streamclone: move stream_in() from localrepo Gregory Szorc <gregory.szorc@gmail.com> parents: 26441 diff changeset	31 util,
26441 56527b886d1d streamclone: move applystreamclone() from localrepo.py Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	32 )
48693 de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48681 diff changeset	33 from .revlogutils import (
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48681 diff changeset	34 nodemap,
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48681 diff changeset	35 )
26441 56527b886d1d streamclone: move applystreamclone() from localrepo.py Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	36
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	37
48601 baddab229b86 stream-clone: add a explicit set list requirements relevant to stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48600 diff changeset	38 def new_stream_clone_requirements(default_requirements, streamed_requirements):
48596 739f2ca3aa3f stream-clone: factor computation of new clone requirement out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47878 diff changeset	39 """determine the final set of requirement for a new stream clone
739f2ca3aa3f stream-clone: factor computation of new clone requirement out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47878 diff changeset	40
739f2ca3aa3f stream-clone: factor computation of new clone requirement out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47878 diff changeset	41 this method combine the "default" requirements that a new repository would
739f2ca3aa3f stream-clone: factor computation of new clone requirement out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47878 diff changeset	42 use with the constaint we get from the stream clone content. We keep local
739f2ca3aa3f stream-clone: factor computation of new clone requirement out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47878 diff changeset	43 configuration choice when possible.
739f2ca3aa3f stream-clone: factor computation of new clone requirement out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47878 diff changeset	44 """
739f2ca3aa3f stream-clone: factor computation of new clone requirement out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47878 diff changeset	45 requirements = set(default_requirements)
48601 baddab229b86 stream-clone: add a explicit set list requirements relevant to stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48600 diff changeset	46 requirements -= requirementsmod.STREAM_FIXED_REQUIREMENTS
48596 739f2ca3aa3f stream-clone: factor computation of new clone requirement out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47878 diff changeset	47 requirements.update(streamed_requirements)
739f2ca3aa3f stream-clone: factor computation of new clone requirement out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47878 diff changeset	48 return requirements
739f2ca3aa3f stream-clone: factor computation of new clone requirement out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47878 diff changeset	49
739f2ca3aa3f stream-clone: factor computation of new clone requirement out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47878 diff changeset	50
48597 8475a1364909 stream-clone: factor computation of requirement of a stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48596 diff changeset	51 def streamed_requirements(repo):
8475a1364909 stream-clone: factor computation of requirement of a stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48596 diff changeset	52 """the set of requirement the new clone will have to support
8475a1364909 stream-clone: factor computation of requirement of a stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48596 diff changeset	53
8475a1364909 stream-clone: factor computation of requirement of a stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48596 diff changeset	54 This is used for advertising the stream options and to generate the actual
8475a1364909 stream-clone: factor computation of requirement of a stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48596 diff changeset	55 stream content."""
48601 baddab229b86 stream-clone: add a explicit set list requirements relevant to stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48600 diff changeset	56 requiredformats = (
baddab229b86 stream-clone: add a explicit set list requirements relevant to stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48600 diff changeset	57 repo.requirements & requirementsmod.STREAM_FIXED_REQUIREMENTS
baddab229b86 stream-clone: add a explicit set list requirements relevant to stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48600 diff changeset	58 )
48597 8475a1364909 stream-clone: factor computation of requirement of a stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48596 diff changeset	59 return requiredformats
8475a1364909 stream-clone: factor computation of requirement of a stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48596 diff changeset	60
8475a1364909 stream-clone: factor computation of requirement of a stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48596 diff changeset	61
35757 bbf7abd09ff0 streamclone: rework canperformstreamclone Boris Feld <boris.feld@octobus.net> parents: 35756 diff changeset	62 def canperformstreamclone(pullop, bundle2=False):
26446 3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	63 """Whether it is possible to perform a streaming clone as part of pull.
26445 f134fb33c906 streamclone: move streaming clone logic from localrepo Gregory Szorc <gregory.szorc@gmail.com> parents: 26444 diff changeset	64
35757 bbf7abd09ff0 streamclone: rework canperformstreamclone Boris Feld <boris.feld@octobus.net> parents: 35756 diff changeset	65 ``bundle2`` will cause the function to consider stream clone through
bbf7abd09ff0 streamclone: rework canperformstreamclone Boris Feld <boris.feld@octobus.net> parents: 35756 diff changeset	66 bundle2 and only through bundle2.
26467 ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware Gregory Szorc <gregory.szorc@gmail.com> parents: 26466 diff changeset	67
26446 3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	68 Returns a tuple of (supported, requirements). ``supported`` is True if
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	69 streaming clone is supported and False otherwise. ``requirements`` is
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	70 a set of repo requirements from the remote, or ``None`` if stream clone
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	71 isn't supported.
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	72 """
26466 3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop Gregory Szorc <gregory.szorc@gmail.com> parents: 26462 diff changeset	73 repo = pullop.repo
3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop Gregory Szorc <gregory.szorc@gmail.com> parents: 26462 diff changeset	74 remote = pullop.remote
3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop Gregory Szorc <gregory.szorc@gmail.com> parents: 26462 diff changeset	75
50519 58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50518 diff changeset	76 # should we consider streaming clone at all ?
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50518 diff changeset	77 streamrequested = pullop.streamclonerequested
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50518 diff changeset	78 # If we don't have a preference, let the server decide for us. This
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50518 diff changeset	79 # likely only comes into play in LANs.
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50518 diff changeset	80 if streamrequested is None:
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50518 diff changeset	81 # The server can advertise whether to prefer streaming clone.
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50518 diff changeset	82 streamrequested = remote.capable(b'stream-preferred')
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50518 diff changeset	83 if not streamrequested:
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50518 diff changeset	84 return False, None
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50518 diff changeset	85
50517 0558866957fa stream-clone: bail-out earlier if destination repo is not empty Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50513 diff changeset	86 # Streaming clone only works on an empty destination repository
0558866957fa stream-clone: bail-out earlier if destination repo is not empty Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50513 diff changeset	87 if len(repo):
0558866957fa stream-clone: bail-out earlier if destination repo is not empty Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50513 diff changeset	88 return False, None
0558866957fa stream-clone: bail-out earlier if destination repo is not empty Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50513 diff changeset	89
50518 f697af015683 stream-clone: bail-out earlier if pull is partial Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50517 diff changeset	90 # Streaming clone only works if all data is being requested.
f697af015683 stream-clone: bail-out earlier if pull is partial Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50517 diff changeset	91 if pullop.heads:
f697af015683 stream-clone: bail-out earlier if pull is partial Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50517 diff changeset	92 return False, None
f697af015683 stream-clone: bail-out earlier if pull is partial Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50517 diff changeset	93
26467 ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware Gregory Szorc <gregory.szorc@gmail.com> parents: 26466 diff changeset	94 bundle2supported = False
ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware Gregory Szorc <gregory.szorc@gmail.com> parents: 26466 diff changeset	95 if pullop.canusebundle2:
50520 a6543983b8f4 stream-clone: check is a compatible protocol can be found Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50519 diff changeset	96 local_caps = bundle2mod.getrepocaps(repo, role=b'client')
a6543983b8f4 stream-clone: check is a compatible protocol can be found Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50519 diff changeset	97 local_supported = set(local_caps.get(b'stream', []))
a6543983b8f4 stream-clone: check is a compatible protocol can be found Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50519 diff changeset	98 remote_supported = set(pullop.remotebundle2caps.get(b'stream', []))
a6543983b8f4 stream-clone: check is a compatible protocol can be found Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50519 diff changeset	99 bundle2supported = bool(local_supported & remote_supported)
26467 ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware Gregory Szorc <gregory.szorc@gmail.com> parents: 26466 diff changeset	100 # else
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	101 # Server doesn't support bundle2 stream clone or doesn't support
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	102 # the versions we support. Fall back and possibly allow legacy.
26467 ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware Gregory Szorc <gregory.szorc@gmail.com> parents: 26466 diff changeset	103
ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware Gregory Szorc <gregory.szorc@gmail.com> parents: 26466 diff changeset	104 # Ensures legacy code path uses available bundle2.
35757 bbf7abd09ff0 streamclone: rework canperformstreamclone Boris Feld <boris.feld@octobus.net> parents: 35756 diff changeset	105 if bundle2supported and not bundle2:
26467 ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware Gregory Szorc <gregory.szorc@gmail.com> parents: 26466 diff changeset	106 return False, None
ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware Gregory Szorc <gregory.szorc@gmail.com> parents: 26466 diff changeset	107 # Ensures bundle2 doesn't try to do a stream clone if it isn't supported.
35757 bbf7abd09ff0 streamclone: rework canperformstreamclone Boris Feld <boris.feld@octobus.net> parents: 35756 diff changeset	108 elif bundle2 and not bundle2supported:
bbf7abd09ff0 streamclone: rework canperformstreamclone Boris Feld <boris.feld@octobus.net> parents: 35756 diff changeset	109 return False, None
26467 ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware Gregory Szorc <gregory.szorc@gmail.com> parents: 26466 diff changeset	110
26446 3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	111 # In order for stream clone to work, the client has to support all the
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	112 # requirements advertised by the server.
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	113 #
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	114 # The server advertises its requirements via the "stream" and "streamreqs"
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	115 # capability. "stream" (a value-less capability) is advertised if and only
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	116 # if the only requirement is "revlogv1." Else, the "streamreqs" capability
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	117 # is advertised and contains a comma-delimited list of requirements.
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	118 requirements = set()
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	119 if remote.capable(b'stream'):
46626 ee91966aec0f requirements: add constant for revlog v1 requirement Rapha?l Gom?s <rgomes@octobus.net> parents: 45106 diff changeset	120 requirements.add(requirementsmod.REVLOGV1_REQUIREMENT)
26446 3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	121 else:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	122 streamreqs = remote.capable(b'streamreqs')
26446 3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	123 # This is weird and shouldn't happen with modern servers.
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	124 if not streamreqs:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	125 pullop.repo.ui.warn(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	126 _(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	127 b'warning: stream clone requested but server has them '
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	128 b'disabled\n'
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	129 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	130 )
26446 3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	131 return False, None
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	132
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	133 streamreqs = set(streamreqs.split(b','))
26446 3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	134 # Server requires something we don't support. Bail.
48670 6d2ddea0721a stream-clone: filter possible missing requirements using all supported one Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47878 diff changeset	135 missingreqs = streamreqs - repo.supported
32259 076f1ff43f0f clone: warn when streaming was requested but couldn't be performed Siddharth Agarwal <sid0@fb.com> parents: 30975 diff changeset	136 if missingreqs:
076f1ff43f0f clone: warn when streaming was requested but couldn't be performed Siddharth Agarwal <sid0@fb.com> parents: 30975 diff changeset	137 pullop.repo.ui.warn(
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	138 _(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	139 b'warning: stream clone requested but client is missing '
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	140 b'requirements: %s\n'
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	141 )
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	142 % b', '.join(sorted(missingreqs))
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	143 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	144 pullop.repo.ui.warn(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	145 _(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	146 b'(see https://www.mercurial-scm.org/wiki/MissingRequirement '
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	147 b'for more information)\n'
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	148 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	149 )
26446 3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	150 return False, None
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	151 requirements = streamreqs
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	152
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	153 return True, requirements
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	154
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	155
26462 3b0ec09192ae streamclone: rename and document maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26461 diff changeset	156 def maybeperformlegacystreamclone(pullop):
3b0ec09192ae streamclone: rename and document maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26461 diff changeset	157 """Possibly perform a legacy stream clone operation.
3b0ec09192ae streamclone: rename and document maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26461 diff changeset	158
3b0ec09192ae streamclone: rename and document maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26461 diff changeset	159 Legacy stream clones are performed as part of pull but before all other
3b0ec09192ae streamclone: rename and document maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26461 diff changeset	160 operations.
3b0ec09192ae streamclone: rename and document maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26461 diff changeset	161
3b0ec09192ae streamclone: rename and document maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26461 diff changeset	162 A legacy stream clone will not be performed if a bundle2 stream clone is
3b0ec09192ae streamclone: rename and document maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26461 diff changeset	163 supported.
3b0ec09192ae streamclone: rename and document maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26461 diff changeset	164 """
39700 b10d145837bc localrepo: extract resolving of opener options to standalone functions Gregory Szorc <gregory.szorc@gmail.com> parents: 38783 diff changeset	165 from . import localrepo
b10d145837bc localrepo: extract resolving of opener options to standalone functions Gregory Szorc <gregory.szorc@gmail.com> parents: 38783 diff changeset	166
26466 3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop Gregory Szorc <gregory.szorc@gmail.com> parents: 26462 diff changeset	167 supported, requirements = canperformstreamclone(pullop)
26458 362793295640 streamclone: refactor maybeperformstreamclone to take a pullop Gregory Szorc <gregory.szorc@gmail.com> parents: 26447 diff changeset	168
26446 3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	169 if not supported:
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	170 return
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	171
26466 3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop Gregory Szorc <gregory.szorc@gmail.com> parents: 26462 diff changeset	172 repo = pullop.repo
3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop Gregory Szorc <gregory.szorc@gmail.com> parents: 26462 diff changeset	173 remote = pullop.remote
3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop Gregory Szorc <gregory.szorc@gmail.com> parents: 26462 diff changeset	174
26459 3b28ffde133a streamclone: move streamin() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26458 diff changeset	175 # Save remote branchmap. We will use it later to speed up branchcache
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26458 diff changeset	176 # creation.
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26458 diff changeset	177 rbranchmap = None
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	178 if remote.capable(b'branchmap'):
37638 65b86ee69383 streamclone: use command executor for wire protocol commands Gregory Szorc <gregory.szorc@gmail.com> parents: 35804 diff changeset	179 with remote.commandexecutor() as e:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	180 rbranchmap = e.callcommand(b'branchmap', {}).result()
26459 3b28ffde133a streamclone: move streamin() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26458 diff changeset	181
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	182 repo.ui.status(_(b'streaming all changes\n'))
26470 4b5647d9ee13 streamclone: move "streaming all changes" message location Gregory Szorc <gregory.szorc@gmail.com> parents: 26469 diff changeset	183
37638 65b86ee69383 streamclone: use command executor for wire protocol commands Gregory Szorc <gregory.szorc@gmail.com> parents: 35804 diff changeset	184 with remote.commandexecutor() as e:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	185 fp = e.callcommand(b'stream_out', {}).result()
37638 65b86ee69383 streamclone: use command executor for wire protocol commands Gregory Szorc <gregory.szorc@gmail.com> parents: 35804 diff changeset	186
65b86ee69383 streamclone: use command executor for wire protocol commands Gregory Szorc <gregory.szorc@gmail.com> parents: 35804 diff changeset	187 # TODO strictly speaking, this code should all be inside the context
65b86ee69383 streamclone: use command executor for wire protocol commands Gregory Szorc <gregory.szorc@gmail.com> parents: 35804 diff changeset	188 # manager because the context manager is supposed to ensure all wire state
65b86ee69383 streamclone: use command executor for wire protocol commands Gregory Szorc <gregory.szorc@gmail.com> parents: 35804 diff changeset	189 # is flushed when exiting. But the legacy peers don't do this, so it
65b86ee69383 streamclone: use command executor for wire protocol commands Gregory Szorc <gregory.szorc@gmail.com> parents: 35804 diff changeset	190 # doesn't matter.
26459 3b28ffde133a streamclone: move streamin() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26458 diff changeset	191 l = fp.readline()
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26458 diff changeset	192 try:
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26458 diff changeset	193 resp = int(l)
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26458 diff changeset	194 except ValueError:
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26458 diff changeset	195 raise error.ResponseError(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	196 _(b'unexpected response from remote server:'), l
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	197 )
26459 3b28ffde133a streamclone: move streamin() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26458 diff changeset	198 if resp == 1:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	199 raise error.Abort(_(b'operation forbidden by server'))
26459 3b28ffde133a streamclone: move streamin() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26458 diff changeset	200 elif resp == 2:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	201 raise error.Abort(_(b'locking the remote repository failed'))
26459 3b28ffde133a streamclone: move streamin() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26458 diff changeset	202 elif resp != 0:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	203 raise error.Abort(_(b'the server sent an unknown error code'))
26459 3b28ffde133a streamclone: move streamin() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26458 diff changeset	204
26468 19bbd53af46d streamclone: move payload header line consumption Gregory Szorc <gregory.szorc@gmail.com> parents: 26467 diff changeset	205 l = fp.readline()
19bbd53af46d streamclone: move payload header line consumption Gregory Szorc <gregory.szorc@gmail.com> parents: 26467 diff changeset	206 try:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	207 filecount, bytecount = map(int, l.split(b' ', 1))
26468 19bbd53af46d streamclone: move payload header line consumption Gregory Szorc <gregory.szorc@gmail.com> parents: 26467 diff changeset	208 except (ValueError, TypeError):
19bbd53af46d streamclone: move payload header line consumption Gregory Szorc <gregory.szorc@gmail.com> parents: 26467 diff changeset	209 raise error.ResponseError(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	210 _(b'unexpected response from remote server:'), l
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	211 )
26468 19bbd53af46d streamclone: move payload header line consumption Gregory Szorc <gregory.szorc@gmail.com> parents: 26467 diff changeset	212
27850 49cfddbf54ba with: use context manager in maybeperformlegacystreamclone Bryan O'Sullivan <bryano@fb.com> parents: 27845 diff changeset	213 with repo.lock():
26468 19bbd53af46d streamclone: move payload header line consumption Gregory Szorc <gregory.szorc@gmail.com> parents: 26467 diff changeset	214 consumev1(repo, fp, filecount, bytecount)
48596 739f2ca3aa3f stream-clone: factor computation of new clone requirement out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47878 diff changeset	215 repo.requirements = new_stream_clone_requirements(
739f2ca3aa3f stream-clone: factor computation of new clone requirement out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47878 diff changeset	216 repo.requirements,
739f2ca3aa3f stream-clone: factor computation of new clone requirement out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47878 diff changeset	217 requirements,
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	218 )
39700 b10d145837bc localrepo: extract resolving of opener options to standalone functions Gregory Szorc <gregory.szorc@gmail.com> parents: 38783 diff changeset	219 repo.svfs.options = localrepo.resolvestorevfsoptions(
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	220 repo.ui, repo.requirements, repo.features
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	221 )
45106 a03c177a4679 scmutil: add writereporequirements() and route requires writing through it Pulkit Goyal <7895pulkit@gmail.com> parents: 43117 diff changeset	222 scmutil.writereporequirements(repo)
48693 de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48681 diff changeset	223 nodemap.post_stream_cleanup(repo)
26461 09cc3c2e9ece streamclone: move applyremotedata() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26460 diff changeset	224
09cc3c2e9ece streamclone: move applyremotedata() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26460 diff changeset	225 if rbranchmap:
41615 328ca3b9e545 branchmap: encapsulate cache updating in the map itself Martijn Pieters <mj@octobus.net> parents: 40494 diff changeset	226 repo._branchcaches.replace(repo, rbranchmap)
26461 09cc3c2e9ece streamclone: move applyremotedata() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26460 diff changeset	227
09cc3c2e9ece streamclone: move applyremotedata() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26460 diff changeset	228 repo.invalidate()
26445 f134fb33c906 streamclone: move streaming clone logic from localrepo Gregory Szorc <gregory.szorc@gmail.com> parents: 26444 diff changeset	229
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	230
32744 33b7283a3828 streamclone: consider secret changesets (BC) (issue5589) Gregory Szorc <gregory.szorc@gmail.com> parents: 32259 diff changeset	231 def allowservergeneration(repo):
26444 623743010133 streamclone: move _allowstream() from wireproto Gregory Szorc <gregory.szorc@gmail.com> parents: 26443 diff changeset	232 """Whether streaming clones are allowed from the server."""
40028 51f10e6d66c7 streamclone: don't support stream clone unless repo feature present Gregory Szorc <gregory.szorc@gmail.com> parents: 39850 diff changeset	233 if repository.REPO_FEATURE_STREAM_CLONE not in repo.features:
51f10e6d66c7 streamclone: don't support stream clone unless repo feature present Gregory Szorc <gregory.szorc@gmail.com> parents: 39850 diff changeset	234 return False
51f10e6d66c7 streamclone: don't support stream clone unless repo feature present Gregory Szorc <gregory.szorc@gmail.com> parents: 39850 diff changeset	235
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	236 if not repo.ui.configbool(b'server', b'uncompressed', untrusted=True):
32744 33b7283a3828 streamclone: consider secret changesets (BC) (issue5589) Gregory Szorc <gregory.szorc@gmail.com> parents: 32259 diff changeset	237 return False
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589) Gregory Szorc <gregory.szorc@gmail.com> parents: 32259 diff changeset	238
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589) Gregory Szorc <gregory.szorc@gmail.com> parents: 32259 diff changeset	239 # The way stream clone works makes it impossible to hide secret changesets.
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589) Gregory Szorc <gregory.szorc@gmail.com> parents: 32259 diff changeset	240 # So don't allow this by default.
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589) Gregory Szorc <gregory.szorc@gmail.com> parents: 32259 diff changeset	241 secret = phases.hassecret(repo)
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589) Gregory Szorc <gregory.szorc@gmail.com> parents: 32259 diff changeset	242 if secret:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	243 return repo.ui.configbool(b'server', b'uncompressedallowsecret')
32744 33b7283a3828 streamclone: consider secret changesets (BC) (issue5589) Gregory Szorc <gregory.szorc@gmail.com> parents: 32259 diff changeset	244
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589) Gregory Szorc <gregory.szorc@gmail.com> parents: 32259 diff changeset	245 return True
26444 623743010133 streamclone: move _allowstream() from wireproto Gregory Szorc <gregory.szorc@gmail.com> parents: 26443 diff changeset	246
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	247
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	248 # This is it's own function so extensions can override it.
50511 0925eaf09c8b store: make `walk` return an entry for obsolescence if requested so Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50509 diff changeset	249 def _walkstreamfiles(repo, matcher=None, phase=False, obsolescence=False):
0925eaf09c8b store: make `walk` return an entry for obsolescence if requested so Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50509 diff changeset	250 return repo.store.walk(matcher, phase=phase, obsolescence=obsolescence)
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	251
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	252
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	253 def generatev1(repo):
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	254 """Emit content for version 1 of a streaming clone.
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	255
26469 fb743268510e streamclone: move payload header generation into own function Gregory Szorc <gregory.szorc@gmail.com> parents: 26468 diff changeset	256 This returns a 3-tuple of (file count, byte size, data iterator).
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	257
26469 fb743268510e streamclone: move payload header generation into own function Gregory Szorc <gregory.szorc@gmail.com> parents: 26468 diff changeset	258 The data iterator consists of N entries for each file being transferred.
fb743268510e streamclone: move payload header generation into own function Gregory Szorc <gregory.szorc@gmail.com> parents: 26468 diff changeset	259 Each file entry starts as a line with the file name and integer size
fb743268510e streamclone: move payload header generation into own function Gregory Szorc <gregory.szorc@gmail.com> parents: 26468 diff changeset	260 delimited by a null byte.
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	261
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	262 The raw file data follows. Following the raw file data is the next file
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	263 entry, or EOF.
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	264
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	265 When used on the wire protocol, an additional line indicating protocol
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	266 success will be prepended to the stream. This function is not responsible
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	267 for adding it.
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	268
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	269 This function will obtain a repository lock to ensure a consistent view of
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	270 the store is captured. It therefore may raise LockError.
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	271 """
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	272 entries = []
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	273 total_bytes = 0
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	274 # Get consistent snapshot of repo, lock during scan.
27845 7417e1c10253 with: use context manager in streamclone generatev1 Bryan O'Sullivan <bryano@fb.com> parents: 27794 diff changeset	275 with repo.lock():
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	276 repo.ui.debug(b'scanning\n')
52359 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	277 _test_sync_point_walk_1_2(repo)
50468 521fec115dad store: use a StoreEntry object instead of tuple for store files Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50288 diff changeset	278 for entry in _walkstreamfiles(repo):
50469 9fdc28e21b68 store: introduce a EntryFile object to actually access file info Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50468 diff changeset	279 for f in entry.files():
50474 4cbdfab6f812 store: lazily get file size on demand for the fncache case Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50469 diff changeset	280 file_size = f.file_size(repo.store.vfs)
4cbdfab6f812 store: lazily get file size on demand for the fncache case Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50469 diff changeset	281 if file_size:
4cbdfab6f812 store: lazily get file size on demand for the fncache case Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50469 diff changeset	282 entries.append((f.unencoded_path, file_size))
4cbdfab6f812 store: lazily get file size on demand for the fncache case Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50469 diff changeset	283 total_bytes += file_size
52358 11484a19cd77 stream: rename all test hook point one number up Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52357 diff changeset	284 _test_sync_point_walk_3(repo)
11484a19cd77 stream: rename all test hook point one number up Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52357 diff changeset	285 _test_sync_point_walk_4(repo)
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	286
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	287 repo.ui.debug(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	288 b'%d files, %d bytes to transfer\n' % (len(entries), total_bytes)
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	289 )
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	290
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	291 svfs = repo.svfs
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	292 debugflag = repo.ui.debugflag
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	293
26469 fb743268510e streamclone: move payload header generation into own function Gregory Szorc <gregory.szorc@gmail.com> parents: 26468 diff changeset	294 def emitrevlogdata():
33258 761ccfeff8b1 streamclone: stop using 'vfs.mustaudit = False' Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33257 diff changeset	295 for name, size in entries:
761ccfeff8b1 streamclone: stop using 'vfs.mustaudit = False' Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33257 diff changeset	296 if debugflag:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	297 repo.ui.debug(b'sending %s (%d bytes)\n' % (name, size))
33258 761ccfeff8b1 streamclone: stop using 'vfs.mustaudit = False' Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33257 diff changeset	298 # partially encode name over the wire for backwards compat
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	299 yield b'%s\0%d\n' % (store.encodedir(name), size)
33411 50b49bb0fff3 streamclone: comment why path auditing is disabled in generatev1() Yuya Nishihara <yuya@tcha.org> parents: 33410 diff changeset	300 # auditing at this stage is both pointless (paths are already
50b49bb0fff3 streamclone: comment why path auditing is disabled in generatev1() Yuya Nishihara <yuya@tcha.org> parents: 33410 diff changeset	301 # trusted by the local repo) and expensive
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	302 with svfs(name, b'rb', auditpath=False) as fp:
33410 c784308305c6 streamclone: close large revlog files explicitly in generatev1() Yuya Nishihara <yuya@tcha.org> parents: 33258 diff changeset	303 if size <= 65536:
33258 761ccfeff8b1 streamclone: stop using 'vfs.mustaudit = False' Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33257 diff changeset	304 yield fp.read(size)
33410 c784308305c6 streamclone: close large revlog files explicitly in generatev1() Yuya Nishihara <yuya@tcha.org> parents: 33258 diff changeset	305 else:
52644 e627cc25b6f3 pyupgrade: rewrite `yield` statements in a loop to `yield from` Matt Harbison <matt_harbison@yahoo.com> parents: 52640 diff changeset	306 yield from util.filechunkiter(fp, limit=size)
26469 fb743268510e streamclone: move payload header generation into own function Gregory Szorc <gregory.szorc@gmail.com> parents: 26468 diff changeset	307
fb743268510e streamclone: move payload header generation into own function Gregory Szorc <gregory.szorc@gmail.com> parents: 26468 diff changeset	308 return len(entries), total_bytes, emitrevlogdata()
fb743268510e streamclone: move payload header generation into own function Gregory Szorc <gregory.szorc@gmail.com> parents: 26468 diff changeset	309
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	310
26469 fb743268510e streamclone: move payload header generation into own function Gregory Szorc <gregory.szorc@gmail.com> parents: 26468 diff changeset	311 def generatev1wireproto(repo):
fb743268510e streamclone: move payload header generation into own function Gregory Szorc <gregory.szorc@gmail.com> parents: 26468 diff changeset	312 """Emit content for version 1 of streaming clone suitable for the wire.
fb743268510e streamclone: move payload header generation into own function Gregory Szorc <gregory.szorc@gmail.com> parents: 26468 diff changeset	313
35491 ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	314 This is the data output from ``generatev1()`` with 2 header lines. The
ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	315 first line indicates overall success. The 2nd contains the file count and
ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	316 byte size of payload.
ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	317
ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	318 The success line contains "0" for success, "1" for stream generation not
ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	319 allowed, and "2" for error locking the repository (possibly indicating
ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	320 a permissions error for the server process).
26469 fb743268510e streamclone: move payload header generation into own function Gregory Szorc <gregory.szorc@gmail.com> parents: 26468 diff changeset	321 """
35491 ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	322 if not allowservergeneration(repo):
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	323 yield b'1\n'
35491 ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	324 return
ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	325
ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	326 try:
ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	327 filecount, bytecount, it = generatev1(repo)
ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	328 except error.LockError:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	329 yield b'2\n'
35491 ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	330 return
ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	331
ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	332 # Indicates successful response.
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	333 yield b'0\n'
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	334 yield b'%d %d\n' % (filecount, bytecount)
52644 e627cc25b6f3 pyupgrade: rewrite `yield` statements in a loop to `yield from` Matt Harbison <matt_harbison@yahoo.com> parents: 52640 diff changeset	335 yield from it
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	336
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	337
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	338 def generatebundlev1(repo, compression=b'UN'):
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	339 """Emit content for version 1 of a stream clone bundle.
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	340
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	341 The first 4 bytes of the output ("HGS1") denote this as stream clone
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	342 bundle version 1.
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	343
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	344 The next 2 bytes indicate the compression type. Only "UN" is currently
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	345 supported.
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	346
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	347 The next 16 bytes are two 64-bit big endian unsigned integers indicating
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	348 file count and byte count, respectively.
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	349
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	350 The next 2 bytes is a 16-bit big endian unsigned short declaring the length
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	351 of the requirements string, including a trailing \0. The following N bytes
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	352 are the requirements string, which is ASCII containing a comma-delimited
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	353 list of repo requirements that are needed to support the data.
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	354
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	355 The remaining content is the output of ``generatev1()`` (which may be
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	356 compressed in the future).
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	357
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	358 Returns a tuple of (requirements, data generator).
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	359 """
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	360 if compression != b'UN':
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	361 raise ValueError(b'we do not support the compression argument yet')
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	362
48597 8475a1364909 stream-clone: factor computation of requirement of a stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48596 diff changeset	363 requirements = streamed_requirements(repo)
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	364 requires = b','.join(sorted(requirements))
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	365
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	366 def gen():
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	367 yield b'HGS1'
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	368 yield compression
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	369
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	370 filecount, bytecount, it = generatev1(repo)
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	371 repo.ui.status(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	372 _(b'writing %d bytes for %d files\n') % (bytecount, filecount)
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	373 )
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	374
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	375 yield struct.pack(b'>QQ', filecount, bytecount)
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	376 yield struct.pack(b'>H', len(requires) + 1)
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	377 yield requires + b'\0'
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	378
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	379 # This is where we'll add compression in the future.
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	380 assert compression == b'UN'
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	381
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	382 progress = repo.ui.makeprogress(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	383 _(b'bundle'), total=bytecount, unit=_(b'bytes')
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	384 )
38349 e59eaf51cc0d streamclone: use progress helper Martin von Zweigbergk <martinvonz@google.com> parents: 38164 diff changeset	385 progress.update(0)
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	386
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	387 for chunk in it:
38349 e59eaf51cc0d streamclone: use progress helper Martin von Zweigbergk <martinvonz@google.com> parents: 38164 diff changeset	388 progress.increment(step=len(chunk))
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	389 yield chunk
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	390
38373 ef692614e601 progress: hide update(None) in a new complete() method Martin von Zweigbergk <martinvonz@google.com> parents: 38360 diff changeset	391 progress.complete()
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	392
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	393 return requirements, gen()
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	394
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	395
26468 19bbd53af46d streamclone: move payload header line consumption Gregory Szorc <gregory.szorc@gmail.com> parents: 26467 diff changeset	396 def consumev1(repo, fp, filecount, bytecount):
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	397 """Apply the contents from version 1 of a streaming clone file handle.
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	398
30332 318a24b52eeb spelling: fixes of non-dictionary words Mads Kiilerich <madski@unity3d.com> parents: 29919 diff changeset	399 This takes the output from "stream_out" and applies it to the specified
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	400 repository.
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	401
30332 318a24b52eeb spelling: fixes of non-dictionary words Mads Kiilerich <madski@unity3d.com> parents: 29919 diff changeset	402 Like "stream_out," the status line added by the wire protocol is not
318a24b52eeb spelling: fixes of non-dictionary words Mads Kiilerich <madski@unity3d.com> parents: 29919 diff changeset	403 handled by this function.
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	404 """
27859 f55a5ace8e69 with: use context manager in streamclone consumev1 Bryan O'Sullivan <bryano@fb.com> parents: 27850 diff changeset	405 with repo.lock():
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	406 repo.ui.status(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	407 _(b'%d files to transfer, %s of data\n')
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	408 % (filecount, util.bytecount(bytecount))
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	409 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	410 progress = repo.ui.makeprogress(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	411 _(b'clone'), total=bytecount, unit=_(b'bytes')
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	412 )
38349 e59eaf51cc0d streamclone: use progress helper Martin von Zweigbergk <martinvonz@google.com> parents: 38164 diff changeset	413 progress.update(0)
30975 22fbca1d11ed mercurial: switch to util.timer for all interval timings Simon Farnsworth <simonfar@fb.com> parents: 30332 diff changeset	414 start = util.timer()
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	415
29919 519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29917 diff changeset	416 # TODO: get rid of (potential) inconsistency
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29917 diff changeset	417 #
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29917 diff changeset	418 # If transaction is started and any @filecache property is
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29917 diff changeset	419 # changed at this point, it causes inconsistency between
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29917 diff changeset	420 # in-memory cached property and streamclone-ed file on the
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29917 diff changeset	421 # disk. Nested transaction prevents transaction scope "clone"
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29917 diff changeset	422 # below from writing in-memory changes out at the end of it,
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29917 diff changeset	423 # even though in-memory changes are discarded at the end of it
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29917 diff changeset	424 # regardless of transaction nesting.
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29917 diff changeset	425 #
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29917 diff changeset	426 # But transaction nesting can't be simply prohibited, because
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29917 diff changeset	427 # nesting occurs also in ordinary case (e.g. enabling
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29917 diff changeset	428 # clonebundles).
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29917 diff changeset	429
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	430 with repo.transaction(b'clone'):
27897 2fdbf22a1b63 streamclone: use backgroundfilecloser (issue4889) Gregory Szorc <gregory.szorc@gmail.com> parents: 27896 diff changeset	431 with repo.svfs.backgroundclosing(repo.ui, expectedcount=filecount):
49284 d44e3c45f0e4 py3: replace `pycompat.xrange` by `range` Manuel Jacob <me@manueljacob.de> parents: 49190 diff changeset	432 for i in range(filecount):
27896 1d29893240cc streamclone: indent code Gregory Szorc <gregory.szorc@gmail.com> parents: 27882 diff changeset	433 # XXX doesn't support '\n' or '\r' in filenames
50925 d718eddf01d9 safehasattr: drop usage in favor of hasattr Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50703 diff changeset	434 if hasattr(fp, 'readline'):
50663 60f9602b413e clonebundles: add support for inline (streaming) clonebundles Mathias De Mare <mathias.de_mare@nokia.com> parents: 50637 diff changeset	435 l = fp.readline()
60f9602b413e clonebundles: add support for inline (streaming) clonebundles Mathias De Mare <mathias.de_mare@nokia.com> parents: 50637 diff changeset	436 else:
60f9602b413e clonebundles: add support for inline (streaming) clonebundles Mathias De Mare <mathias.de_mare@nokia.com> parents: 50637 diff changeset	437 # inline clonebundles use a chunkbuffer, so no readline
60f9602b413e clonebundles: add support for inline (streaming) clonebundles Mathias De Mare <mathias.de_mare@nokia.com> parents: 50637 diff changeset	438 # --> this should be small anyway, the first line
60f9602b413e clonebundles: add support for inline (streaming) clonebundles Mathias De Mare <mathias.de_mare@nokia.com> parents: 50637 diff changeset	439 # only contains the size of the bundle
60f9602b413e clonebundles: add support for inline (streaming) clonebundles Mathias De Mare <mathias.de_mare@nokia.com> parents: 50637 diff changeset	440 l_buf = []
60f9602b413e clonebundles: add support for inline (streaming) clonebundles Mathias De Mare <mathias.de_mare@nokia.com> parents: 50637 diff changeset	441 while not (l_buf and l_buf[-1] == b'\n'):
60f9602b413e clonebundles: add support for inline (streaming) clonebundles Mathias De Mare <mathias.de_mare@nokia.com> parents: 50637 diff changeset	442 l_buf.append(fp.read(1))
60f9602b413e clonebundles: add support for inline (streaming) clonebundles Mathias De Mare <mathias.de_mare@nokia.com> parents: 50637 diff changeset	443 l = b''.join(l_buf)
27896 1d29893240cc streamclone: indent code Gregory Szorc <gregory.szorc@gmail.com> parents: 27882 diff changeset	444 try:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	445 name, size = l.split(b'\0', 1)
27896 1d29893240cc streamclone: indent code Gregory Szorc <gregory.szorc@gmail.com> parents: 27882 diff changeset	446 size = int(size)
1d29893240cc streamclone: indent code Gregory Szorc <gregory.szorc@gmail.com> parents: 27882 diff changeset	447 except (ValueError, TypeError):
1d29893240cc streamclone: indent code Gregory Szorc <gregory.szorc@gmail.com> parents: 27882 diff changeset	448 raise error.ResponseError(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	449 _(b'unexpected response from remote server:'), l
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	450 )
27896 1d29893240cc streamclone: indent code Gregory Szorc <gregory.szorc@gmail.com> parents: 27882 diff changeset	451 if repo.ui.debugflag:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	452 repo.ui.debug(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	453 b'adding %s (%s)\n' % (name, util.bytecount(size))
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	454 )
27896 1d29893240cc streamclone: indent code Gregory Szorc <gregory.szorc@gmail.com> parents: 27882 diff changeset	455 # for backwards compat, name was partially encoded
27897 2fdbf22a1b63 streamclone: use backgroundfilecloser (issue4889) Gregory Szorc <gregory.szorc@gmail.com> parents: 27896 diff changeset	456 path = store.decodedir(name)
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	457 with repo.svfs(path, b'w', backgroundclose=True) as ofp:
27896 1d29893240cc streamclone: indent code Gregory Szorc <gregory.szorc@gmail.com> parents: 27882 diff changeset	458 for chunk in util.filechunkiter(fp, limit=size):
38349 e59eaf51cc0d streamclone: use progress helper Martin von Zweigbergk <martinvonz@google.com> parents: 38164 diff changeset	459 progress.increment(step=len(chunk))
27896 1d29893240cc streamclone: indent code Gregory Szorc <gregory.szorc@gmail.com> parents: 27882 diff changeset	460 ofp.write(chunk)
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	461
29919 519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29917 diff changeset	462 # force @filecache properties to be reloaded from
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29917 diff changeset	463 # streamclone-ed file at next access
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29917 diff changeset	464 repo.invalidate(clearfilecache=True)
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	465
30975 22fbca1d11ed mercurial: switch to util.timer for all interval timings Simon Farnsworth <simonfar@fb.com> parents: 30332 diff changeset	466 elapsed = util.timer() - start
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	467 if elapsed <= 0:
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	468 elapsed = 0.001
38373 ef692614e601 progress: hide update(None) in a new complete() method Martin von Zweigbergk <martinvonz@google.com> parents: 38360 diff changeset	469 progress.complete()
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	470 repo.ui.status(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	471 _(b'transferred %s in %.1f seconds (%s/sec)\n')
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	472 % (
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	473 util.bytecount(bytecount),
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	474 elapsed,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	475 util.bytecount(bytecount / elapsed),
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	476 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	477 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	478
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	479
27882 319b0bf6ecc9 streamclone: extract code for reading header fields Gregory Szorc <gregory.szorc@gmail.com> parents: 27870 diff changeset	480 def readbundle1header(fp):
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	481 compression = fp.read(2)
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	482 if compression != b'UN':
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	483 raise error.Abort(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	484 _(
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	485 b'only uncompressed stream clone bundles are '
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	486 b'supported; got %s'
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	487 )
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	488 % compression
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	489 )
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	490
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	491 filecount, bytecount = struct.unpack(b'>QQ', fp.read(16))
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	492 requireslen = struct.unpack(b'>H', fp.read(2))[0]
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	493 requires = fp.read(requireslen)
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	494
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	495 if not requires.endswith(b'\0'):
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	496 raise error.Abort(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	497 _(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	498 b'malformed stream clone bundle: '
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	499 b'requirements not properly encoded'
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	500 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	501 )
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	502
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	503 requirements = set(requires.rstrip(b'\0').split(b','))
27882 319b0bf6ecc9 streamclone: extract code for reading header fields Gregory Szorc <gregory.szorc@gmail.com> parents: 27870 diff changeset	504
319b0bf6ecc9 streamclone: extract code for reading header fields Gregory Szorc <gregory.szorc@gmail.com> parents: 27870 diff changeset	505 return filecount, bytecount, requirements
319b0bf6ecc9 streamclone: extract code for reading header fields Gregory Szorc <gregory.szorc@gmail.com> parents: 27870 diff changeset	506
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	507
27882 319b0bf6ecc9 streamclone: extract code for reading header fields Gregory Szorc <gregory.szorc@gmail.com> parents: 27870 diff changeset	508 def applybundlev1(repo, fp):
319b0bf6ecc9 streamclone: extract code for reading header fields Gregory Szorc <gregory.szorc@gmail.com> parents: 27870 diff changeset	509 """Apply the content from a stream clone bundle version 1.
319b0bf6ecc9 streamclone: extract code for reading header fields Gregory Szorc <gregory.szorc@gmail.com> parents: 27870 diff changeset	510
319b0bf6ecc9 streamclone: extract code for reading header fields Gregory Szorc <gregory.szorc@gmail.com> parents: 27870 diff changeset	511 We assume the 4 byte header has been read and validated and the file handle
319b0bf6ecc9 streamclone: extract code for reading header fields Gregory Szorc <gregory.szorc@gmail.com> parents: 27870 diff changeset	512 is at the 2 byte compression identifier.
319b0bf6ecc9 streamclone: extract code for reading header fields Gregory Szorc <gregory.szorc@gmail.com> parents: 27870 diff changeset	513 """
319b0bf6ecc9 streamclone: extract code for reading header fields Gregory Szorc <gregory.szorc@gmail.com> parents: 27870 diff changeset	514 if len(repo):
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	515 raise error.Abort(
43117 8ff1ecfadcd1 cleanup: join string literals that are already on one line Martin von Zweigbergk <martinvonz@google.com> parents: 43085 diff changeset	516 _(b'cannot apply stream clone bundle on non-empty repo')
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	517 )
27882 319b0bf6ecc9 streamclone: extract code for reading header fields Gregory Szorc <gregory.szorc@gmail.com> parents: 27870 diff changeset	518
319b0bf6ecc9 streamclone: extract code for reading header fields Gregory Szorc <gregory.szorc@gmail.com> parents: 27870 diff changeset	519 filecount, bytecount, requirements = readbundle1header(fp)
48670 6d2ddea0721a stream-clone: filter possible missing requirements using all supported one Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47878 diff changeset	520 missingreqs = requirements - repo.supported
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	521 if missingreqs:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	522 raise error.Abort(
43117 8ff1ecfadcd1 cleanup: join string literals that are already on one line Martin von Zweigbergk <martinvonz@google.com> parents: 43085 diff changeset	523 _(b'unable to apply stream clone: unsupported format: %s')
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	524 % b', '.join(sorted(missingreqs))
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	525 )
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	526
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	527 consumev1(repo, fp, filecount, bytecount)
48693 de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48681 diff changeset	528 nodemap.post_stream_cleanup(repo)
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	529
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	530
48946 642e31cb55f0 py3: use class X: instead of class X(object): Gregory Szorc <gregory.szorc@gmail.com> parents: 48875 diff changeset	531 class streamcloneapplier:
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	532 """Class to manage applying streaming clone bundles.
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	533
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	534 We need to wrap ``applybundlev1()`` in a dedicated type to enable bundle
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	535 readers to perform bundle type-specific functionality.
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	536 """
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	537
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	538 def __init__(self, fh):
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	539 self._fh = fh
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	540
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	541 def apply(self, repo):
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	542 return applybundlev1(repo, self._fh)
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	543
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	544
35765 56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	545 # type of file to stream
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	546 _fileappend = 0 # append only file
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	547 _filefull = 1 # full snapshot file
35765 56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	548
35767 5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	549 # Source of the file
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	550 _srcstore = b's' # store (svfs)
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	551 _srccache = b'c' # cache (cache)
35767 5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	552
51699 ca7bde5dbafb black: format the codebase with 23.3.0 Rapha?l Gom?s <rgomes@octobus.net> parents: 51696 diff changeset	553
35765 56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	554 # This is it's own function so extensions can override it.
56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	555 def _walkstreamfullstorefiles(repo):
56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	556 """list snapshot file from the store"""
56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	557 fnames = []
56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	558 if not repo.publishing():
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	559 fnames.append(b'phaseroots')
35765 56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	560 return fnames
56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	561
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	562
35767 5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	563 def _filterfull(entry, copy, vfsmap):
35765 56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	564 """actually copy the snapshot files"""
35767 5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	565 src, name, ftype, data = entry
35765 56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	566 if ftype != _filefull:
56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	567 return entry
35767 5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	568 return (src, name, ftype, copy(vfsmap[src].join(name)))
35765 56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	569
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	570
52012 e308439339e2 stream: rename TempCopyManager to VolatileManager Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51859 diff changeset	571 class VolatileManager:
52015 a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	572 """Manage temporary backups of volatile files during stream clone.
50627 8c7b04e69894 stream-clone: introduce a richer TempCopyManager object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50521 diff changeset	573
52015 a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	574 This class will keep open file handles for the volatile files, writing the
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	575 smaller ones on disk if the number of open file handles grow too much.
50627 8c7b04e69894 stream-clone: introduce a richer TempCopyManager object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50521 diff changeset	576
52015 a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	577 This should be used as a Python context, the file handles and copies will
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	578 be discarded when exiting the context.
49190 4ff4e23de7df clone: use better names for temp files Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 48946 diff changeset	579
52015 a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	580 The preservation can be done by calling the object on the real path
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	581 (encoded full path).
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	582
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	583 Valid filehandles for any file should be retrieved by calling `open(path)`.
50627 8c7b04e69894 stream-clone: introduce a richer TempCopyManager object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50521 diff changeset	584 """
8c7b04e69894 stream-clone: introduce a richer TempCopyManager object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50521 diff changeset	585
52015 a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	586 # arbitrarily picked as "it seemed fine" and much higher than the current
52184 e4b242f9d4d9 streamclone: disable the volatile file open handle optimization on Windows Matt Harbison <matt_harbison@yahoo.com> parents: 52015 diff changeset	587 # usage. The Windows value of 2 is actually 1 file open at a time, due to
e4b242f9d4d9 streamclone: disable the volatile file open handle optimization on Windows Matt Harbison <matt_harbison@yahoo.com> parents: 52015 diff changeset	588 # the `flush_count = self.MAX_OPEN // 2` and `self.MAX_OPEN - 1` threshold
e4b242f9d4d9 streamclone: disable the volatile file open handle optimization on Windows Matt Harbison <matt_harbison@yahoo.com> parents: 52015 diff changeset	589 # for flushing to disk in __call__().
e4b242f9d4d9 streamclone: disable the volatile file open handle optimization on Windows Matt Harbison <matt_harbison@yahoo.com> parents: 52015 diff changeset	590 MAX_OPEN = 2 if pycompat.iswindows else 100
52015 a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	591
50627 8c7b04e69894 stream-clone: introduce a richer TempCopyManager object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50521 diff changeset	592 def __init__(self):
52015 a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	593 self._counter = 0
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	594 self._volatile_fps = None
50627 8c7b04e69894 stream-clone: introduce a richer TempCopyManager object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50521 diff changeset	595 self._copies = None
8c7b04e69894 stream-clone: introduce a richer TempCopyManager object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50521 diff changeset	596 self._dst_dir = None
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	597
50627 8c7b04e69894 stream-clone: introduce a richer TempCopyManager object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50521 diff changeset	598 def __enter__(self):
52015 a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	599 if self._counter == 0:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	600 assert self._volatile_fps is None
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	601 self._volatile_fps = {}
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	602 self._counter += 1
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	603 return self
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	604
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	605 def __exit__(self, args, *kwars):
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	606 """discard all backups"""
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	607 self._counter -= 1
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	608 if self._counter == 0:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	609 for _size, fp in self._volatile_fps.values():
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	610 fp.close()
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	611 self._volatile_fps = None
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	612 if self._copies is not None:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	613 for tmp in self._copies.values():
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	614 util.tryunlink(tmp)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	615 util.tryrmdir(self._dst_dir)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	616 self._copies = None
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	617 self._dst_dir = None
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	618 assert self._volatile_fps is None
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	619 assert self._copies is None
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	620 assert self._dst_dir is None
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	621
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	622 def _init_tmp_copies(self):
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	623 """prepare a temporary directory to save volatile files
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	624
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	625 This will be used as backup if we have too many files open"""
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	626 assert 0 < self._counter
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	627 assert self._copies is None
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	628 assert self._dst_dir is None
50627 8c7b04e69894 stream-clone: introduce a richer TempCopyManager object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50521 diff changeset	629 self._copies = {}
8c7b04e69894 stream-clone: introduce a richer TempCopyManager object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50521 diff changeset	630 self._dst_dir = pycompat.mkdtemp(prefix=b'hg-clone-')
52015 a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	631
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	632 def _flush_some_on_disk(self):
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	633 """move some of the open files to tempory files on disk"""
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	634 if self._copies is None:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	635 self._init_tmp_copies()
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	636 flush_count = self.MAX_OPEN // 2
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	637 for src, (size, fp) in sorted(self._volatile_fps.items())[:flush_count]:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	638 prefix = os.path.basename(src)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	639 fd, dst = pycompat.mkstemp(prefix=prefix, dir=self._dst_dir)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	640 self._copies[src] = dst
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	641 os.close(fd)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	642 # we no longer hardlink, but on the other hand we rarely do this,
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	643 # and we do it for the smallest file only and not at all in the
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	644 # common case.
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	645 with open(dst, 'wb') as bck:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	646 fp.seek(0)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	647 bck.write(fp.read())
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	648 del self._volatile_fps[src]
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	649 fp.close()
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	650
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	651 def _keep_one(self, src):
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	652 """preserve an open file handle for a given path"""
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	653 # store the file quickly to ensure we close it if any error happens
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	654 _, fp = self._volatile_fps[src] = (None, open(src, 'rb'))
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	655 fp.seek(0, os.SEEK_END)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	656 size = fp.tell()
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	657 self._volatile_fps[src] = (size, fp)
52359 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	658 return size
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	659
50627 8c7b04e69894 stream-clone: introduce a richer TempCopyManager object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50521 diff changeset	660 def __call__(self, src):
52015 a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	661 """preserve the volatile file at src"""
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	662 assert 0 < self._counter
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	663 if len(self._volatile_fps) >= (self.MAX_OPEN - 1):
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	664 self._flush_some_on_disk()
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	665 self._keep_one(src)
50627 8c7b04e69894 stream-clone: introduce a richer TempCopyManager object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50521 diff changeset	666
52359 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	667 def try_keep(self, src) -> Optional[int]:
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	668 """record a volatile file and returns it size
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	669
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	670 return None if the file does not exists.
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	671
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	672 Used for cache file that are not lock protected.
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	673 """
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	674 assert 0 < self._counter
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	675 if len(self._volatile_fps) >= (self.MAX_OPEN - 1):
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	676 self._flush_some_on_disk()
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	677 try:
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	678 return self._keep_one(src)
52640 24ee91ba9aa8 pyupgrade: drop usage of py3 aliases for `OSError` Matt Harbison <matt_harbison@yahoo.com> parents: 52359 diff changeset	679 except OSError as err:
52359 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	680 if err.errno not in (errno.ENOENT, errno.EPERM):
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	681 raise
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	682 return None
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	683
52013 0ad269e24075 stream: open volatile file through the manager Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52012 diff changeset	684 @contextlib.contextmanager
0ad269e24075 stream: open volatile file through the manager Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52012 diff changeset	685 def open(self, src):
52015 a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	686 assert 0 < self._counter
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	687 entry = self._volatile_fps.get(src)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	688 if entry is not None:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	689 _size, fp = entry
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	690 fp.seek(0)
52013 0ad269e24075 stream: open volatile file through the manager Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52012 diff changeset	691 yield fp
52015 a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	692 else:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	693 if self._copies is None:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	694 actual_path = src
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	695 else:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	696 actual_path = self._copies.get(src, src)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	697 with open(actual_path, 'rb') as fp:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	698 yield fp
35765 56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	699
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	700
35767 5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	701 def _makemap(repo):
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	702 """make a (src -> vfs) map for the repo"""
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	703 vfsmap = {
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	704 _srcstore: repo.svfs,
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	705 _srccache: repo.cachevfs,
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	706 }
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	707 # we keep repo.vfs out of the on purpose, ther are too many danger there
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	708 # (eg: .hg/hgrc)
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	709 assert repo.vfs not in vfsmap.values()
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	710
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	711 return vfsmap
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	712
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	713
50632 3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2 Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50631 diff changeset	714 def _emit2(repo, entries):
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	715 """actually emit the stream bundle"""
35767 5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	716 vfsmap = _makemap(repo)
47502 65c519661991 stream: double check that self.vfs is not in the vfsmap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	717 # we keep repo.vfs out of the on purpose, ther are too many danger there
65c519661991 stream: double check that self.vfs is not in the vfsmap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	718 # (eg: .hg/hgrc),
65c519661991 stream: double check that self.vfs is not in the vfsmap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	719 #
65c519661991 stream: double check that self.vfs is not in the vfsmap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	720 # this assert is duplicated (from _makemap) as author might think this is
65c519661991 stream: double check that self.vfs is not in the vfsmap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	721 # fine, while this is really not fine.
65c519661991 stream: double check that self.vfs is not in the vfsmap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	722 if repo.vfs in vfsmap.values():
65c519661991 stream: double check that self.vfs is not in the vfsmap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	723 raise error.ProgrammingError(
65c519661991 stream: double check that self.vfs is not in the vfsmap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	724 b'repo.vfs must not be added to vfsmap for security reasons'
65c519661991 stream: double check that self.vfs is not in the vfsmap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	725 )
65c519661991 stream: double check that self.vfs is not in the vfsmap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	726
50632 3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2 Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50631 diff changeset	727 # translate the vfs one
3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2 Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50631 diff changeset	728 entries = [(vfs_key, vfsmap[vfs_key], e) for (vfs_key, e) in entries]
52359 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	729 _test_sync_point_walk_1_2(repo)
50632 3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2 Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50631 diff changeset	730
50637 9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50636 diff changeset	731 max_linkrev = len(repo)
50632 3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2 Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50631 diff changeset	732 file_count = totalfilesize = 0
52356 46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	733 with VolatileManager() as volatiles:
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	734 # make sure we preserve volatile files
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	735 with util.nogc():
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	736 # record the expected size of every file
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	737 for k, vfs, e in entries:
52357 a260d326458f stream: create a `e.preserve_volatiles` method directly on StoreEntry Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52356 diff changeset	738 e.preserve_volatiles(vfs, volatiles)
52356 46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	739 for f in e.files():
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	740 file_count += 1
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	741 totalfilesize += f.file_size(vfs)
50632 3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2 Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50631 diff changeset	742
52356 46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	743 progress = repo.ui.makeprogress(
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	744 _(b'bundle'), total=totalfilesize, unit=_(b'bytes')
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	745 )
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	746 progress.update(0)
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	747 with progress:
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	748 # the first yield release the lock on the repository
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	749 yield file_count, totalfilesize
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	750 totalbytecount = 0
35765 56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	751
52356 46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	752 for src, vfs, e in entries:
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	753 entry_streams = e.get_streams(
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	754 repo=repo,
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	755 vfs=vfs,
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	756 volatiles=volatiles,
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	757 max_changeset=max_linkrev,
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	758 preserve_file_count=True,
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	759 )
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	760 for name, stream, size in entry_streams:
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	761 yield src
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	762 yield util.uvarintencode(len(name))
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	763 yield util.uvarintencode(size)
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	764 yield name
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	765 bytecount = 0
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	766 for chunk in stream:
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	767 bytecount += len(chunk)
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	768 totalbytecount += len(chunk)
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	769 progress.update(totalbytecount)
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	770 yield chunk
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	771 if bytecount != size:
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	772 # Would most likely be caused by a race due to `hg
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	773 # strip` or a revlog split
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	774 msg = _(
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	775 b'clone could only read %d bytes from %s, but '
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	776 b'expected %d bytes'
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	777 )
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	778 raise error.Abort(msg % (bytecount, name, size))
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	779
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	780
50703 0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	781 def _emit3(repo, entries):
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	782 """actually emit the stream bundle (v3)"""
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	783 vfsmap = _makemap(repo)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	784 # we keep repo.vfs out of the map on purpose, ther are too many dangers
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	785 # there (eg: .hg/hgrc),
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	786 #
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	787 # this assert is duplicated (from _makemap) as authors might think this is
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	788 # fine, while this is really not fine.
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	789 if repo.vfs in vfsmap.values():
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	790 raise error.ProgrammingError(
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	791 b'repo.vfs must not be added to vfsmap for security reasons'
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	792 )
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	793
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	794 # translate the vfs once
52359 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	795 # we only turn this into a list for the `_test_sync`, this is not ideal
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	796 base_entries = list(entries)
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	797 _test_sync_point_walk_1_2(repo)
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	798 entries = []
52356 46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	799 with VolatileManager() as volatiles:
52012 e308439339e2 stream: rename TempCopyManager to VolatileManager Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51859 diff changeset	800 # make sure we preserve volatile files
52359 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	801 for vfs_key, e in base_entries:
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	802 vfs = vfsmap[vfs_key]
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	803 any_files = True
51549 49faa72b994e streamclone: stop listing files for entries that have no volatile files Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51548 diff changeset	804 if e.maybe_volatile:
52359 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	805 any_files = False
52357 a260d326458f stream: create a `e.preserve_volatiles` method directly on StoreEntry Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52356 diff changeset	806 e.preserve_volatiles(vfs, volatiles)
51549 49faa72b994e streamclone: stop listing files for entries that have no volatile files Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51548 diff changeset	807 for f in e.files():
49faa72b994e streamclone: stop listing files for entries that have no volatile files Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51548 diff changeset	808 if f.is_volatile:
49faa72b994e streamclone: stop listing files for entries that have no volatile files Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51548 diff changeset	809 # record the expected size under lock
49faa72b994e streamclone: stop listing files for entries that have no volatile files Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51548 diff changeset	810 f.file_size(vfs)
52359 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	811 any_files = True
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	812 if any_files:
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	813 entries.append((vfs_key, vfsmap[vfs_key], e))
52356 46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	814
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	815 total_entry_count = len(entries)
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	816
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	817 max_linkrev = len(repo)
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	818 progress = repo.ui.makeprogress(
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	819 _(b'bundle'),
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	820 total=total_entry_count,
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	821 unit=_(b'entry'),
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	822 )
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	823 progress.update(0)
50703 0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	824 # the first yield release the lock on the repository
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	825 yield None
52356 46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	826 with progress:
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	827 yield util.uvarintencode(total_entry_count)
50703 0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	828
52356 46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	829 for src, vfs, e in entries:
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	830 entry_streams = e.get_streams(
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	831 repo=repo,
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	832 vfs=vfs,
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	833 volatiles=volatiles,
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	834 max_changeset=max_linkrev,
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	835 )
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	836 yield util.uvarintencode(len(entry_streams))
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	837 for name, stream, size in entry_streams:
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	838 yield src
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	839 yield util.uvarintencode(len(name))
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	840 yield util.uvarintencode(size)
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	841 yield name
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	842 yield from stream
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	843 progress.increment()
50703 0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	844
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	845
52359 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	846 def _test_sync_point_walk_1_2(repo):
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	847 """a function for synchronisation during tests
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	848
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	849 Triggered after gather entry, but before starting to process/preserve them
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	850 under lock.
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	851
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	852 (on v1 is triggered before the actual walk start)
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	853 """
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	854
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	855
52358 11484a19cd77 stream: rename all test hook point one number up Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52357 diff changeset	856 def _test_sync_point_walk_3(repo):
52359 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	857 """a function for synchronisation during tests
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	858
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	859 Triggered right before releasing the lock, but after computing what need
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	860 needed to compute under lock.
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	861 """
46986 faa43f09ad98 streamclone: remove sleep based "synchronisation" in tests Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46895 diff changeset	862
faa43f09ad98 streamclone: remove sleep based "synchronisation" in tests Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46895 diff changeset	863
52358 11484a19cd77 stream: rename all test hook point one number up Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52357 diff changeset	864 def _test_sync_point_walk_4(repo):
52359 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	865 """a function for synchronisation during tests
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	866
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	867 Triggered right after releasing the lock.
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	868 """
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	869
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	870
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	871 # not really a StoreEntry, but close enough
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	872 class CacheEntry(store.SimpleStoreEntry):
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	873 """Represent an entry for Cache files
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	874
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	875 It has special logic to preserve cache file early and accept optional
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	876 presence.
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	877
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	878
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	879 (Yes... this is not really a StoreEntry, but close enough. We could have a
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	880 BaseEntry base class, bbut the store one would be identical)
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	881 """
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	882
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	883 def __init__(self, entry_path):
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	884 super().__init__(
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	885 entry_path,
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	886 # we will directly deal with that in `setup_cache_file`
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	887 is_volatile=True,
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	888 )
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	889
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	890 def preserve_volatiles(self, vfs, volatiles):
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	891 self._file_size = volatiles.try_keep(vfs.join(self._entry_path))
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	892 if self._file_size is None:
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	893 self._files = []
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	894 else:
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	895 assert self._is_volatile
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	896 self._files = [
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	897 CacheFile(
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	898 unencoded_path=self._entry_path,
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	899 file_size=self._file_size,
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	900 is_volatile=self._is_volatile,
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	901 )
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	902 ]
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	903
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	904 def files(self):
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	905 if self._files is None:
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	906 self._files = [
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	907 CacheFile(
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	908 unencoded_path=self._entry_path,
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	909 is_volatile=self._is_volatile,
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	910 )
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	911 ]
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	912 return super().files()
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	913
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	914
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	915 class CacheFile(store.StoreFile):
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	916 # inform the "copy/hardlink" version that this file might be missing
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	917 # without consequences.
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	918 optional = True
46986 faa43f09ad98 streamclone: remove sleep based "synchronisation" in tests Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46895 diff changeset	919
faa43f09ad98 streamclone: remove sleep based "synchronisation" in tests Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46895 diff changeset	920
50512 06d580b8f432 stream-clone: introduce a _entries_walk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50511 diff changeset	921 def _entries_walk(repo, includes, excludes, includeobsmarkers):
06d580b8f432 stream-clone: introduce a _entries_walk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50511 diff changeset	922 """emit a seris of files information useful to clone a repo
06d580b8f432 stream-clone: introduce a _entries_walk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50511 diff changeset	923
06d580b8f432 stream-clone: introduce a _entries_walk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50511 diff changeset	924 return (vfs-key, entry) iterator
06d580b8f432 stream-clone: introduce a _entries_walk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50511 diff changeset	925
06d580b8f432 stream-clone: introduce a _entries_walk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50511 diff changeset	926 Where `entry` is StoreEntry. (used even for cache entries)
06d580b8f432 stream-clone: introduce a _entries_walk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50511 diff changeset	927 """
06d580b8f432 stream-clone: introduce a _entries_walk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50511 diff changeset	928 assert repo._currentlock(repo._lockref) is not None
06d580b8f432 stream-clone: introduce a _entries_walk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50511 diff changeset	929
06d580b8f432 stream-clone: introduce a _entries_walk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50511 diff changeset	930 matcher = None
06d580b8f432 stream-clone: introduce a _entries_walk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50511 diff changeset	931 if includes or excludes:
06d580b8f432 stream-clone: introduce a _entries_walk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50511 diff changeset	932 matcher = narrowspec.match(repo.root, includes, excludes)
06d580b8f432 stream-clone: introduce a _entries_walk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50511 diff changeset	933
06d580b8f432 stream-clone: introduce a _entries_walk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50511 diff changeset	934 phase = not repo.publishing()
51547 463e63aa547c stream-clone: disable gc for `_entries_walk` duration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51545 diff changeset	935 # Python is getting crazy at all the small container we creates, disabling
463e63aa547c stream-clone: disable gc for `_entries_walk` duration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51545 diff changeset	936 # the gc while we do so helps performance a lot.
463e63aa547c stream-clone: disable gc for `_entries_walk` duration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51545 diff changeset	937 with util.nogc():
463e63aa547c stream-clone: disable gc for `_entries_walk` duration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51545 diff changeset	938 entries = _walkstreamfiles(
463e63aa547c stream-clone: disable gc for `_entries_walk` duration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51545 diff changeset	939 repo,
463e63aa547c stream-clone: disable gc for `_entries_walk` duration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51545 diff changeset	940 matcher,
463e63aa547c stream-clone: disable gc for `_entries_walk` duration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51545 diff changeset	941 phase=phase,
463e63aa547c stream-clone: disable gc for `_entries_walk` duration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51545 diff changeset	942 obsolescence=includeobsmarkers,
463e63aa547c stream-clone: disable gc for `_entries_walk` duration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51545 diff changeset	943 )
463e63aa547c stream-clone: disable gc for `_entries_walk` duration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51545 diff changeset	944 for entry in entries:
463e63aa547c stream-clone: disable gc for `_entries_walk` duration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51545 diff changeset	945 yield (_srcstore, entry)
50512 06d580b8f432 stream-clone: introduce a _entries_walk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50511 diff changeset	946
51547 463e63aa547c stream-clone: disable gc for `_entries_walk` duration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51545 diff changeset	947 for name in cacheutil.cachetocopy(repo):
463e63aa547c stream-clone: disable gc for `_entries_walk` duration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51545 diff changeset	948 if repo.cachevfs.exists(name):
463e63aa547c stream-clone: disable gc for `_entries_walk` duration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51545 diff changeset	949 # not really a StoreEntry, but close enough
52359 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	950 yield (_srccache, CacheEntry(entry_path=name))
50513 43ed1f12b00a stream-clone: yield cache entry in `_entries_walk` too Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50512 diff changeset	951
50512 06d580b8f432 stream-clone: introduce a _entries_walk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50511 diff changeset	952
40398 0ac794e0e285 streamclone: include obsstore file into stream bundle if client can read it Anton Shestakov <av6@dwimlabs.net> parents: 40339 diff changeset	953 def generatev2(repo, includes, excludes, includeobsmarkers):
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	954 """Emit content for version 2 of a streaming clone.
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	955
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	956 the data stream consists the following entries:
35767 5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	957 1) A char representing the file destination (eg: store or cache)
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	958 2) A varint containing the length of the filename
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	959 3) A varint containing the length of file data
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	960 4) N bytes containing the filename (the internal, store-agnostic form)
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	961 5) N bytes containing the file data
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	962
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	963 Returns a 3-tuple of (file count, file size, data iterator).
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	964 """
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	965
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	966 with repo.lock():
47444 2f4ca4807033 streamingclone: extract the scanning part from the generation part Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	967 repo.ui.debug(b'scanning\n')
40339 f0e8f27768eb streamclone: pass narrowing related info in _walkstreamfiles() Pulkit Goyal <pulkit@yandex-team.ru> parents: 40338 diff changeset	968
50632 3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2 Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50631 diff changeset	969 entries = _entries_walk(
47444 2f4ca4807033 streamingclone: extract the scanning part from the generation part Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	970 repo,
2f4ca4807033 streamingclone: extract the scanning part from the generation part Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	971 includes=includes,
2f4ca4807033 streamingclone: extract the scanning part from the generation part Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	972 excludes=excludes,
2f4ca4807033 streamingclone: extract the scanning part from the generation part Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	973 includeobsmarkers=includeobsmarkers,
2f4ca4807033 streamingclone: extract the scanning part from the generation part Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	974 )
50632 3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2 Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50631 diff changeset	975 chunks = _emit2(repo, entries)
35765 56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	976 first = next(chunks)
50632 3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2 Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50631 diff changeset	977 file_count, total_file_size = first
52358 11484a19cd77 stream: rename all test hook point one number up Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52357 diff changeset	978 _test_sync_point_walk_3(repo)
11484a19cd77 stream: rename all test hook point one number up Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52357 diff changeset	979 _test_sync_point_walk_4(repo)
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	980
50632 3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2 Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50631 diff changeset	981 return file_count, total_file_size, chunks
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	982
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	983
50521 58adcabc295f stream-clone: introduce the notion of an experimental "v3" version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50520 diff changeset	984 def generatev3(repo, includes, excludes, includeobsmarkers):
50703 0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	985 """Emit content for version 3 of a streaming clone.
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	986
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	987 the data stream consists the following:
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	988 1) A varint E containing the number of entries (can be 0), then E entries follow
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	989 2) For each entry:
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	990 2.1) The number of files in this entry (can be 0, but typically 1 or 2)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	991 2.2) For each file:
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	992 2.2.1) A char representing the file destination (eg: store or cache)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	993 2.2.2) A varint N containing the length of the filename
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	994 2.2.3) A varint M containing the length of file data
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	995 2.2.4) N bytes containing the filename (the internal, store-agnostic form)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	996 2.2.5) M bytes containing the file data
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	997
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	998 Returns the data iterator.
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	999
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1000 XXX This format is experimental and subject to change. Here is a
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1001 XXX non-exhaustive list of things this format could do or change:
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1002
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1003 - making it easier to write files in parallel
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1004 - holding the lock for a shorter time
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1005 - improving progress information
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1006 - ways to adjust the number of expected entries/files ?
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1007 """
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1008
51548 6e4c8366c5ce stream-clone: disable gc for the initial section for the v3 format Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51547 diff changeset	1009 # Python is getting crazy at all the small container we creates while
6e4c8366c5ce stream-clone: disable gc for the initial section for the v3 format Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51547 diff changeset	1010 # considering the files to preserve, disabling the gc while we do so helps
6e4c8366c5ce stream-clone: disable gc for the initial section for the v3 format Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51547 diff changeset	1011 # performance a lot.
6e4c8366c5ce stream-clone: disable gc for the initial section for the v3 format Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51547 diff changeset	1012 with repo.lock(), util.nogc():
50703 0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1013 repo.ui.debug(b'scanning\n')
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1014
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1015 entries = _entries_walk(
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1016 repo,
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1017 includes=includes,
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1018 excludes=excludes,
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1019 includeobsmarkers=includeobsmarkers,
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1020 )
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1021 chunks = _emit3(repo, list(entries))
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1022 first = next(chunks)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1023 assert first is None
52358 11484a19cd77 stream: rename all test hook point one number up Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52357 diff changeset	1024 _test_sync_point_walk_3(repo)
11484a19cd77 stream: rename all test hook point one number up Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52357 diff changeset	1025 _test_sync_point_walk_4(repo)
50703 0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1026
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1027 return chunks
50521 58adcabc295f stream-clone: introduce the notion of an experimental "v3" version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50520 diff changeset	1028
58adcabc295f stream-clone: introduce the notion of an experimental "v3" version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50520 diff changeset	1029
35767 5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	1030 @contextlib.contextmanager
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	1031 def nested(*ctxs):
39757 97f2992c26f6 streamclone: reimplement nested context manager Augie Fackler <augie@google.com> parents: 39700 diff changeset	1032 this = ctxs[0]
97f2992c26f6 streamclone: reimplement nested context manager Augie Fackler <augie@google.com> parents: 39700 diff changeset	1033 rest = ctxs[1:]
97f2992c26f6 streamclone: reimplement nested context manager Augie Fackler <augie@google.com> parents: 39700 diff changeset	1034 with this:
97f2992c26f6 streamclone: reimplement nested context manager Augie Fackler <augie@google.com> parents: 39700 diff changeset	1035 if rest:
97f2992c26f6 streamclone: reimplement nested context manager Augie Fackler <augie@google.com> parents: 39700 diff changeset	1036 with nested(*rest):
97f2992c26f6 streamclone: reimplement nested context manager Augie Fackler <augie@google.com> parents: 39700 diff changeset	1037 yield
97f2992c26f6 streamclone: reimplement nested context manager Augie Fackler <augie@google.com> parents: 39700 diff changeset	1038 else:
35767 5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	1039 yield
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	1040
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	1041
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1042 def consumev2(repo, fp, filecount, filesize):
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1043 """Apply the contents from a version 2 streaming clone.
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1044
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1045 Data is read from an object that only needs to provide a ``read(size)``
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1046 method.
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1047 """
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1048 with repo.lock():
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	1049 repo.ui.status(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	1050 _(b'%d files to transfer, %s of data\n')
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	1051 % (filecount, util.bytecount(filesize))
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	1052 )
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1053
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1054 start = util.timer()
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	1055 progress = repo.ui.makeprogress(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	1056 _(b'clone'), total=filesize, unit=_(b'bytes')
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	1057 )
38349 e59eaf51cc0d streamclone: use progress helper Martin von Zweigbergk <martinvonz@google.com> parents: 38164 diff changeset	1058 progress.update(0)
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1059
35767 5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	1060 vfsmap = _makemap(repo)
47502 65c519661991 stream: double check that self.vfs is not in the vfsmap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	1061 # we keep repo.vfs out of the on purpose, ther are too many danger
65c519661991 stream: double check that self.vfs is not in the vfsmap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	1062 # there (eg: .hg/hgrc),
65c519661991 stream: double check that self.vfs is not in the vfsmap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	1063 #
65c519661991 stream: double check that self.vfs is not in the vfsmap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	1064 # this assert is duplicated (from _makemap) as author might think this
65c519661991 stream: double check that self.vfs is not in the vfsmap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	1065 # is fine, while this is really not fine.
65c519661991 stream: double check that self.vfs is not in the vfsmap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	1066 if repo.vfs in vfsmap.values():
65c519661991 stream: double check that self.vfs is not in the vfsmap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	1067 raise error.ProgrammingError(
65c519661991 stream: double check that self.vfs is not in the vfsmap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	1068 b'repo.vfs must not be added to vfsmap for security reasons'
65c519661991 stream: double check that self.vfs is not in the vfsmap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	1069 )
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1070
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	1071 with repo.transaction(b'clone'):
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	1072 ctxs = (vfs.backgroundclosing(repo.ui) for vfs in vfsmap.values())
35767 5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	1073 with nested(*ctxs):
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1074 for i in range(filecount):
35803 3ad3aaeb1134 streamclone: use readexactly when reading stream v2 Boris Feld <boris.feld@octobus.net> parents: 35802 diff changeset	1075 src = util.readexactly(fp, 1)
35767 5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	1076 vfs = vfsmap[src]
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1077 namelen = util.uvarintdecodestream(fp)
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1078 datalen = util.uvarintdecodestream(fp)
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1079
35803 3ad3aaeb1134 streamclone: use readexactly when reading stream v2 Boris Feld <boris.feld@octobus.net> parents: 35802 diff changeset	1080 name = util.readexactly(fp, namelen)
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1081
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1082 if repo.ui.debugflag:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	1083 repo.ui.debug(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	1084 b'adding [%s] %s (%s)\n'
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	1085 % (src, name, util.bytecount(datalen))
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	1086 )
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1087
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	1088 with vfs(name, b'w') as ofp:
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1089 for chunk in util.filechunkiter(fp, limit=datalen):
38349 e59eaf51cc0d streamclone: use progress helper Martin von Zweigbergk <martinvonz@google.com> parents: 38164 diff changeset	1090 progress.increment(step=len(chunk))
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1091 ofp.write(chunk)
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1092
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1093 # force @filecache properties to be reloaded from
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1094 # streamclone-ed file at next access
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1095 repo.invalidate(clearfilecache=True)
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1096
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1097 elapsed = util.timer() - start
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1098 if elapsed <= 0:
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1099 elapsed = 0.001
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	1100 repo.ui.status(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	1101 _(b'transferred %s in %.1f seconds (%s/sec)\n')
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	1102 % (
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	1103 util.bytecount(progress.pos),
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	1104 elapsed,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	1105 util.bytecount(progress.pos / elapsed),
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	1106 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	1107 )
38373 ef692614e601 progress: hide update(None) in a new complete() method Martin von Zweigbergk <martinvonz@google.com> parents: 38360 diff changeset	1108 progress.complete()
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1109
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	1110
50703 0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1111 def consumev3(repo, fp):
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1112 """Apply the contents from a version 3 streaming clone.
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1113
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1114 Data is read from an object that only needs to provide a ``read(size)``
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1115 method.
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1116 """
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1117 with repo.lock():
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1118 start = util.timer()
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1119
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1120 entrycount = util.uvarintdecodestream(fp)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1121 repo.ui.status(_(b'%d entries to transfer\n') % (entrycount))
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1122
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1123 progress = repo.ui.makeprogress(
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1124 _(b'clone'),
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1125 total=entrycount,
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1126 unit=_(b'entries'),
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1127 )
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1128 progress.update(0)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1129 bytes_transferred = 0
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1130
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1131 vfsmap = _makemap(repo)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1132 # we keep repo.vfs out of the on purpose, there are too many dangers
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1133 # there (eg: .hg/hgrc),
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1134 #
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1135 # this assert is duplicated (from _makemap) as authors might think this
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1136 # is fine, while this is really not fine.
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1137 if repo.vfs in vfsmap.values():
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1138 raise error.ProgrammingError(
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1139 b'repo.vfs must not be added to vfsmap for security reasons'
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1140 )
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1141
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1142 with repo.transaction(b'clone'):
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1143 ctxs = (vfs.backgroundclosing(repo.ui) for vfs in vfsmap.values())
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1144 with nested(*ctxs):
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1145 for i in range(entrycount):
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1146 filecount = util.uvarintdecodestream(fp)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1147 if filecount == 0:
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1148 if repo.ui.debugflag:
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1149 repo.ui.debug(b'entry with no files [%d]\n' % (i))
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1150 for i in range(filecount):
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1151 src = util.readexactly(fp, 1)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1152 vfs = vfsmap[src]
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1153 namelen = util.uvarintdecodestream(fp)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1154 datalen = util.uvarintdecodestream(fp)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1155
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1156 name = util.readexactly(fp, namelen)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1157
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1158 if repo.ui.debugflag:
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1159 msg = b'adding [%s] %s (%s)\n'
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1160 msg %= (src, name, util.bytecount(datalen))
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1161 repo.ui.debug(msg)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1162 bytes_transferred += datalen
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1163
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1164 with vfs(name, b'w') as ofp:
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1165 for chunk in util.filechunkiter(fp, limit=datalen):
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1166 ofp.write(chunk)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1167 progress.increment(step=1)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1168
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1169 # force @filecache properties to be reloaded from
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1170 # streamclone-ed file at next access
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1171 repo.invalidate(clearfilecache=True)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1172
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1173 elapsed = util.timer() - start
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1174 if elapsed <= 0:
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1175 elapsed = 0.001
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1176 msg = _(b'transferred %s in %.1f seconds (%s/sec)\n')
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1177 byte_count = util.bytecount(bytes_transferred)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1178 bytes_sec = util.bytecount(bytes_transferred / elapsed)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1179 msg %= (byte_count, elapsed, bytes_sec)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1180 repo.ui.status(msg)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1181 progress.complete()
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1182
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1183
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1184 def applybundlev2(repo, fp, filecount, filesize, requirements):
39700 b10d145837bc localrepo: extract resolving of opener options to standalone functions Gregory Szorc <gregory.szorc@gmail.com> parents: 38783 diff changeset	1185 from . import localrepo
b10d145837bc localrepo: extract resolving of opener options to standalone functions Gregory Szorc <gregory.szorc@gmail.com> parents: 38783 diff changeset	1186
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1187 missingreqs = [r for r in requirements if r not in repo.supported]
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1188 if missingreqs:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	1189 raise error.Abort(
43117 8ff1ecfadcd1 cleanup: join string literals that are already on one line Martin von Zweigbergk <martinvonz@google.com> parents: 43085 diff changeset	1190 _(b'unable to apply stream clone: unsupported format: %s')
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	1191 % b', '.join(sorted(missingreqs))
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	1192 )
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1193
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1194 consumev2(repo, fp, filecount, filesize)
35804 2d3e486d09d0 streamclone: move requirement update into consumev2 Boris Feld <boris.feld@octobus.net> parents: 35803 diff changeset	1195
48596 739f2ca3aa3f stream-clone: factor computation of new clone requirement out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47878 diff changeset	1196 repo.requirements = new_stream_clone_requirements(
739f2ca3aa3f stream-clone: factor computation of new clone requirement out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47878 diff changeset	1197 repo.requirements,
739f2ca3aa3f stream-clone: factor computation of new clone requirement out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47878 diff changeset	1198 requirements,
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	1199 )
39700 b10d145837bc localrepo: extract resolving of opener options to standalone functions Gregory Szorc <gregory.szorc@gmail.com> parents: 38783 diff changeset	1200 repo.svfs.options = localrepo.resolvestorevfsoptions(
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	1201 repo.ui, repo.requirements, repo.features
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	1202 )
45106 a03c177a4679 scmutil: add writereporequirements() and route requires writing through it Pulkit Goyal <7895pulkit@gmail.com> parents: 43117 diff changeset	1203 scmutil.writereporequirements(repo)
48693 de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48681 diff changeset	1204 nodemap.post_stream_cleanup(repo)
47447 377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1205
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1206
50703 0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1207 def applybundlev3(repo, fp, requirements):
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1208 from . import localrepo
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1209
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1210 missingreqs = [r for r in requirements if r not in repo.supported]
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1211 if missingreqs:
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1212 msg = _(b'unable to apply stream clone: unsupported format: %s')
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1213 msg %= b', '.join(sorted(missingreqs))
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1214 raise error.Abort(msg)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1215
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1216 consumev3(repo, fp)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1217
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1218 repo.requirements = new_stream_clone_requirements(
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1219 repo.requirements,
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1220 requirements,
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1221 )
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1222 repo.svfs.options = localrepo.resolvestorevfsoptions(
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1223 repo.ui, repo.requirements, repo.features
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1224 )
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1225 scmutil.writereporequirements(repo)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1226 nodemap.post_stream_cleanup(repo)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1227
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1228
47447 377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1229 def _copy_files(src_vfs_map, dst_vfs_map, entries, progress):
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1230 hardlink = [True]
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1231
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1232 def copy_used():
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1233 hardlink[0] = False
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1234 progress.topic = _(b'copying')
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1235
52359 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	1236 for k, path, optional in entries:
47447 377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1237 src_vfs = src_vfs_map[k]
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1238 dst_vfs = dst_vfs_map[k]
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1239 src_path = src_vfs.join(path)
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1240 dst_path = dst_vfs.join(path)
47871 132525ead0db clone: properly create target directories during local clone (issue6581) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47788 diff changeset	1241 # We cannot use dirname and makedirs of dst_vfs here because the store
132525ead0db clone: properly create target directories during local clone (issue6581) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47788 diff changeset	1242 # encoding confuses them. See issue 6581 for details.
132525ead0db clone: properly create target directories during local clone (issue6581) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47788 diff changeset	1243 dirname = os.path.dirname(dst_path)
132525ead0db clone: properly create target directories during local clone (issue6581) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47788 diff changeset	1244 if not os.path.exists(dirname):
132525ead0db clone: properly create target directories during local clone (issue6581) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47788 diff changeset	1245 util.makedirs(dirname)
47447 377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1246 dst_vfs.register_file(path)
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1247 # XXX we could use the #nb_bytes argument.
52359 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	1248 try:
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	1249 util.copyfile(
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	1250 src_path,
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	1251 dst_path,
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	1252 hardlink=hardlink[0],
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	1253 no_hardlink_cb=copy_used,
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	1254 check_fs_hardlink=False,
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	1255 )
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	1256 except FileNotFoundError:
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	1257 if not optional:
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	1258 raise
47447 377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1259 progress.increment()
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1260 return hardlink[0]
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1261
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1262
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1263 def local_copy(src_repo, dest_repo):
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1264 """copy all content from one local repository to another
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1265
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1266 This is useful for local clone"""
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1267 src_store_requirements = {
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1268 r
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1269 for r in src_repo.requirements
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1270 if r not in requirementsmod.WORKING_DIR_REQUIREMENTS
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1271 }
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1272 dest_store_requirements = {
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1273 r
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1274 for r in dest_repo.requirements
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1275 if r not in requirementsmod.WORKING_DIR_REQUIREMENTS
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1276 }
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1277 assert src_store_requirements == dest_store_requirements
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1278
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1279 with dest_repo.lock():
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1280 with src_repo.lock():
47448 d370256636fe clone: also report the bookmark file as copied Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47447 diff changeset	1281 # bookmark is not integrated to the streaming as it might use the
d370256636fe clone: also report the bookmark file as copied Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47447 diff changeset	1282 # `repo.vfs` and they are too many sentitive data accessible
d370256636fe clone: also report the bookmark file as copied Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47447 diff changeset	1283 # through `repo.vfs` to expose it to streaming clone.
d370256636fe clone: also report the bookmark file as copied Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47447 diff changeset	1284 src_book_vfs = bookmarks.bookmarksvfs(src_repo)
d370256636fe clone: also report the bookmark file as copied Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47447 diff changeset	1285 srcbookmarks = src_book_vfs.join(b'bookmarks')
d370256636fe clone: also report the bookmark file as copied Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47447 diff changeset	1286 bm_count = 0
d370256636fe clone: also report the bookmark file as copied Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47447 diff changeset	1287 if os.path.exists(srcbookmarks):
d370256636fe clone: also report the bookmark file as copied Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47447 diff changeset	1288 bm_count = 1
d370256636fe clone: also report the bookmark file as copied Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47447 diff changeset	1289
50630 f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50627 diff changeset	1290 entries = _entries_walk(
47447 377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1291 src_repo,
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1292 includes=None,
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1293 excludes=None,
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1294 includeobsmarkers=True,
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1295 )
50630 f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50627 diff changeset	1296 entries = list(entries)
47447 377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1297 src_vfs_map = _makemap(src_repo)
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1298 dest_vfs_map = _makemap(dest_repo)
50630 f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50627 diff changeset	1299 total_files = sum(len(e[1].files()) for e in entries) + bm_count
47447 377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1300 progress = src_repo.ui.makeprogress(
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1301 topic=_(b'linking'),
50630 f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50627 diff changeset	1302 total=total_files,
47447 377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1303 unit=_(b'files'),
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1304 )
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1305 # copy files
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1306 #
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1307 # We could copy the full file while the source repository is locked
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1308 # and the other one without the lock. However, in the linking case,
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1309 # this would also requires checks that nobody is appending any data
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1310 # to the files while we do the clone, so this is not done yet. We
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1311 # could do this blindly when copying files.
50630 f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50627 diff changeset	1312 files = [
52359 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	1313 (vfs_key, f.unencoded_path, f.optional)
50630 f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50627 diff changeset	1314 for vfs_key, e in entries
f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50627 diff changeset	1315 for f in e.files()
f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50627 diff changeset	1316 ]
47447 377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1317 hardlink = _copy_files(src_vfs_map, dest_vfs_map, files, progress)
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1318
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1319 # copy bookmarks over
47448 d370256636fe clone: also report the bookmark file as copied Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47447 diff changeset	1320 if bm_count:
d370256636fe clone: also report the bookmark file as copied Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47447 diff changeset	1321 dst_book_vfs = bookmarks.bookmarksvfs(dest_repo)
d370256636fe clone: also report the bookmark file as copied Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47447 diff changeset	1322 dstbookmarks = dst_book_vfs.join(b'bookmarks')
47447 377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1323 util.copyfile(srcbookmarks, dstbookmarks)
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1324 progress.complete()
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1325 if hardlink:
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1326 msg = b'linked %d files\n'
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1327 else:
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1328 msg = b'copied %d files\n'
50630 f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50627 diff changeset	1329 src_repo.ui.debug(msg % total_files)
47447 377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1330
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1331 with dest_repo.transaction(b"localclone") as tr:
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1332 dest_repo.store.write(tr)
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1333
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1334 # clean up transaction file as they do not make sense
50288 d89eecf9605e undo-files: no longer pass the `repo` to `cleanup_undo_files` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50286 diff changeset	1335 transaction.cleanup_undo_files(dest_repo.ui.warn, dest_repo.vfs_map)

Mercurial > public > mercurial-scm > hg

annotate mercurial/streamclone.py @ 52644:e627cc25b6f3