mercurial-scm/hg-stable: mercurial/streamclone.py annotate

annotate mercurial/streamclone.py @ 43076:2372284d9457

formatting: blacken the codebase This is using my patch to black (https://github.com/psf/black/pull/826) so we don't un-wrap collection literals. Done with: hg files 'set:**.py - mercurial/thirdparty/** - "contrib/python-zstandard/**"' | xargs black -S # skip-blame mass-reformatting only # no-check-commit reformats foo_bar functions Differential Revision: https://phab.mercurial-scm.org/D6971

author	Augie Fackler <augie@google.com>
date	Sun, 06 Oct 2019 09:45:02 -0400
parents	268662aac075
children	687b865b95ad

rev	line source
26441 56527b886d1d streamclone: move applystreamclone() from localrepo.py Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1 # streamclone.py - producing and consuming streaming repository data
56527b886d1d streamclone: move applystreamclone() from localrepo.py Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	2 #
56527b886d1d streamclone: move applystreamclone() from localrepo.py Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	3 # Copyright 2015 Gregory Szorc <gregory.szorc@gmail.com>
56527b886d1d streamclone: move applystreamclone() from localrepo.py Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	4 #
56527b886d1d streamclone: move applystreamclone() from localrepo.py Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	5 # This software may be used and distributed according to the terms of the
56527b886d1d streamclone: move applystreamclone() from localrepo.py Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	6 # GNU General Public License version 2 or any later version.
56527b886d1d streamclone: move applystreamclone() from localrepo.py Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	7
56527b886d1d streamclone: move applystreamclone() from localrepo.py Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	8 from __future__ import absolute_import
56527b886d1d streamclone: move applystreamclone() from localrepo.py Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	9
35765 56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	10 import contextlib
56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	11 import os
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	12 import struct
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	13
26442 ef8d27f53204 streamclone: move stream_in() from localrepo Gregory Szorc <gregory.szorc@gmail.com> parents: 26441 diff changeset	14 from .i18n import _
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	15 from .interfaces import repository
26441 56527b886d1d streamclone: move applystreamclone() from localrepo.py Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	16 from . import (
35767 5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	17 cacheutil,
26442 ef8d27f53204 streamclone: move stream_in() from localrepo Gregory Szorc <gregory.szorc@gmail.com> parents: 26441 diff changeset	18 error,
40339 f0e8f27768eb streamclone: pass narrowing related info in _walkstreamfiles() Pulkit Goyal <pulkit@yandex-team.ru> parents: 40338 diff changeset	19 narrowspec,
32764 33b7283a3828 streamclone: consider secret changesets (BC) (issue5589) Gregory Szorc <gregory.szorc@gmail.com> parents: 32299 diff changeset	20 phases,
38197 aac4be30e250 py3: wrap tempfile.mkstemp() to use bytes path Yuya Nishihara <yuya@tcha.org> parents: 37638 diff changeset	21 pycompat,
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	22 store,
26442 ef8d27f53204 streamclone: move stream_in() from localrepo Gregory Szorc <gregory.szorc@gmail.com> parents: 26441 diff changeset	23 util,
26441 56527b886d1d streamclone: move applystreamclone() from localrepo.py Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	24 )
56527b886d1d streamclone: move applystreamclone() from localrepo.py Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	25
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	26
35757 bbf7abd09ff0 streamclone: rework canperformstreamclone Boris Feld <boris.feld@octobus.net> parents: 35756 diff changeset	27 def canperformstreamclone(pullop, bundle2=False):
26446 3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	28 """Whether it is possible to perform a streaming clone as part of pull.
26445 f134fb33c906 streamclone: move streaming clone logic from localrepo Gregory Szorc <gregory.szorc@gmail.com> parents: 26444 diff changeset	29
35757 bbf7abd09ff0 streamclone: rework canperformstreamclone Boris Feld <boris.feld@octobus.net> parents: 35756 diff changeset	30 ``bundle2`` will cause the function to consider stream clone through
bbf7abd09ff0 streamclone: rework canperformstreamclone Boris Feld <boris.feld@octobus.net> parents: 35756 diff changeset	31 bundle2 and only through bundle2.
26467 ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware Gregory Szorc <gregory.szorc@gmail.com> parents: 26466 diff changeset	32
26446 3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	33 Returns a tuple of (supported, requirements). ``supported`` is True if
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	34 streaming clone is supported and False otherwise. ``requirements`` is
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	35 a set of repo requirements from the remote, or ``None`` if stream clone
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	36 isn't supported.
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	37 """
26466 3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop Gregory Szorc <gregory.szorc@gmail.com> parents: 26462 diff changeset	38 repo = pullop.repo
3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop Gregory Szorc <gregory.szorc@gmail.com> parents: 26462 diff changeset	39 remote = pullop.remote
3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop Gregory Szorc <gregory.szorc@gmail.com> parents: 26462 diff changeset	40
26467 ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware Gregory Szorc <gregory.szorc@gmail.com> parents: 26466 diff changeset	41 bundle2supported = False
ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware Gregory Szorc <gregory.szorc@gmail.com> parents: 26466 diff changeset	42 if pullop.canusebundle2:
35757 bbf7abd09ff0 streamclone: rework canperformstreamclone Boris Feld <boris.feld@octobus.net> parents: 35756 diff changeset	43 if 'v2' in pullop.remotebundle2caps.get('stream', []):
26467 ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware Gregory Szorc <gregory.szorc@gmail.com> parents: 26466 diff changeset	44 bundle2supported = True
ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware Gregory Szorc <gregory.szorc@gmail.com> parents: 26466 diff changeset	45 # else
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	46 # Server doesn't support bundle2 stream clone or doesn't support
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	47 # the versions we support. Fall back and possibly allow legacy.
26467 ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware Gregory Szorc <gregory.szorc@gmail.com> parents: 26466 diff changeset	48
ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware Gregory Szorc <gregory.szorc@gmail.com> parents: 26466 diff changeset	49 # Ensures legacy code path uses available bundle2.
35757 bbf7abd09ff0 streamclone: rework canperformstreamclone Boris Feld <boris.feld@octobus.net> parents: 35756 diff changeset	50 if bundle2supported and not bundle2:
26467 ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware Gregory Szorc <gregory.szorc@gmail.com> parents: 26466 diff changeset	51 return False, None
ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware Gregory Szorc <gregory.szorc@gmail.com> parents: 26466 diff changeset	52 # Ensures bundle2 doesn't try to do a stream clone if it isn't supported.
35757 bbf7abd09ff0 streamclone: rework canperformstreamclone Boris Feld <boris.feld@octobus.net> parents: 35756 diff changeset	53 elif bundle2 and not bundle2supported:
bbf7abd09ff0 streamclone: rework canperformstreamclone Boris Feld <boris.feld@octobus.net> parents: 35756 diff changeset	54 return False, None
26467 ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware Gregory Szorc <gregory.szorc@gmail.com> parents: 26466 diff changeset	55
26447 591088f7028a streamclone: add explicit check for empty local repo Gregory Szorc <gregory.szorc@gmail.com> parents: 26446 diff changeset	56 # Streaming clone only works on empty repositories.
591088f7028a streamclone: add explicit check for empty local repo Gregory Szorc <gregory.szorc@gmail.com> parents: 26446 diff changeset	57 if len(repo):
591088f7028a streamclone: add explicit check for empty local repo Gregory Szorc <gregory.szorc@gmail.com> parents: 26446 diff changeset	58 return False, None
591088f7028a streamclone: add explicit check for empty local repo Gregory Szorc <gregory.szorc@gmail.com> parents: 26446 diff changeset	59
26446 3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	60 # Streaming clone only works if all data is being requested.
26466 3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop Gregory Szorc <gregory.szorc@gmail.com> parents: 26462 diff changeset	61 if pullop.heads:
26446 3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	62 return False, None
26445 f134fb33c906 streamclone: move streaming clone logic from localrepo Gregory Szorc <gregory.szorc@gmail.com> parents: 26444 diff changeset	63
26466 3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop Gregory Szorc <gregory.szorc@gmail.com> parents: 26462 diff changeset	64 streamrequested = pullop.streamclonerequested
3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop Gregory Szorc <gregory.szorc@gmail.com> parents: 26462 diff changeset	65
26446 3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	66 # If we don't have a preference, let the server decide for us. This
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	67 # likely only comes into play in LANs.
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	68 if streamrequested is None:
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	69 # The server can advertise whether to prefer streaming clone.
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	70 streamrequested = remote.capable('stream-preferred')
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	71
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	72 if not streamrequested:
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	73 return False, None
26445 f134fb33c906 streamclone: move streaming clone logic from localrepo Gregory Szorc <gregory.szorc@gmail.com> parents: 26444 diff changeset	74
26446 3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	75 # In order for stream clone to work, the client has to support all the
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	76 # requirements advertised by the server.
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	77 #
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	78 # The server advertises its requirements via the "stream" and "streamreqs"
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	79 # capability. "stream" (a value-less capability) is advertised if and only
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	80 # if the only requirement is "revlogv1." Else, the "streamreqs" capability
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	81 # is advertised and contains a comma-delimited list of requirements.
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	82 requirements = set()
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	83 if remote.capable('stream'):
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	84 requirements.add('revlogv1')
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	85 else:
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	86 streamreqs = remote.capable('streamreqs')
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	87 # This is weird and shouldn't happen with modern servers.
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	88 if not streamreqs:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	89 pullop.repo.ui.warn(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	90 _(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	91 'warning: stream clone requested but server has them '
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	92 'disabled\n'
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	93 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	94 )
26446 3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	95 return False, None
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	96
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	97 streamreqs = set(streamreqs.split(','))
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	98 # Server requires something we don't support. Bail.
32299 076f1ff43f0f clone: warn when streaming was requested but couldn't be performed Siddharth Agarwal <sid0@fb.com> parents: 30995 diff changeset	99 missingreqs = streamreqs - repo.supportedformats
076f1ff43f0f clone: warn when streaming was requested but couldn't be performed Siddharth Agarwal <sid0@fb.com> parents: 30995 diff changeset	100 if missingreqs:
076f1ff43f0f clone: warn when streaming was requested but couldn't be performed Siddharth Agarwal <sid0@fb.com> parents: 30995 diff changeset	101 pullop.repo.ui.warn(
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	102 _(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	103 'warning: stream clone requested but client is missing '
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	104 'requirements: %s\n'
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	105 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	106 % ', '.join(sorted(missingreqs))
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	107 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	108 pullop.repo.ui.warn(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	109 _(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	110 '(see https://www.mercurial-scm.org/wiki/MissingRequirement '
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	111 'for more information)\n'
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	112 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	113 )
26446 3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	114 return False, None
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	115 requirements = streamreqs
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	116
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	117 return True, requirements
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	118
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	119
26462 3b0ec09192ae streamclone: rename and document maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26461 diff changeset	120 def maybeperformlegacystreamclone(pullop):
3b0ec09192ae streamclone: rename and document maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26461 diff changeset	121 """Possibly perform a legacy stream clone operation.
3b0ec09192ae streamclone: rename and document maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26461 diff changeset	122
3b0ec09192ae streamclone: rename and document maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26461 diff changeset	123 Legacy stream clones are performed as part of pull but before all other
3b0ec09192ae streamclone: rename and document maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26461 diff changeset	124 operations.
3b0ec09192ae streamclone: rename and document maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26461 diff changeset	125
3b0ec09192ae streamclone: rename and document maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26461 diff changeset	126 A legacy stream clone will not be performed if a bundle2 stream clone is
3b0ec09192ae streamclone: rename and document maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26461 diff changeset	127 supported.
3b0ec09192ae streamclone: rename and document maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26461 diff changeset	128 """
39716 b10d145837bc localrepo: extract resolving of opener options to standalone functions Gregory Szorc <gregory.szorc@gmail.com> parents: 38823 diff changeset	129 from . import localrepo
b10d145837bc localrepo: extract resolving of opener options to standalone functions Gregory Szorc <gregory.szorc@gmail.com> parents: 38823 diff changeset	130
26466 3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop Gregory Szorc <gregory.szorc@gmail.com> parents: 26462 diff changeset	131 supported, requirements = canperformstreamclone(pullop)
26458 362793295640 streamclone: refactor maybeperformstreamclone to take a pullop Gregory Szorc <gregory.szorc@gmail.com> parents: 26447 diff changeset	132
26446 3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	133 if not supported:
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	134 return
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	135
26466 3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop Gregory Szorc <gregory.szorc@gmail.com> parents: 26462 diff changeset	136 repo = pullop.repo
3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop Gregory Szorc <gregory.szorc@gmail.com> parents: 26462 diff changeset	137 remote = pullop.remote
3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop Gregory Szorc <gregory.szorc@gmail.com> parents: 26462 diff changeset	138
26459 3b28ffde133a streamclone: move streamin() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26458 diff changeset	139 # Save remote branchmap. We will use it later to speed up branchcache
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26458 diff changeset	140 # creation.
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26458 diff changeset	141 rbranchmap = None
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26458 diff changeset	142 if remote.capable('branchmap'):
37638 65b86ee69383 streamclone: use command executor for wire protocol commands Gregory Szorc <gregory.szorc@gmail.com> parents: 35804 diff changeset	143 with remote.commandexecutor() as e:
65b86ee69383 streamclone: use command executor for wire protocol commands Gregory Szorc <gregory.szorc@gmail.com> parents: 35804 diff changeset	144 rbranchmap = e.callcommand('branchmap', {}).result()
26459 3b28ffde133a streamclone: move streamin() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26458 diff changeset	145
26470 4b5647d9ee13 streamclone: move "streaming all changes" message location Gregory Szorc <gregory.szorc@gmail.com> parents: 26469 diff changeset	146 repo.ui.status(_('streaming all changes\n'))
4b5647d9ee13 streamclone: move "streaming all changes" message location Gregory Szorc <gregory.szorc@gmail.com> parents: 26469 diff changeset	147
37638 65b86ee69383 streamclone: use command executor for wire protocol commands Gregory Szorc <gregory.szorc@gmail.com> parents: 35804 diff changeset	148 with remote.commandexecutor() as e:
65b86ee69383 streamclone: use command executor for wire protocol commands Gregory Szorc <gregory.szorc@gmail.com> parents: 35804 diff changeset	149 fp = e.callcommand('stream_out', {}).result()
65b86ee69383 streamclone: use command executor for wire protocol commands Gregory Szorc <gregory.szorc@gmail.com> parents: 35804 diff changeset	150
65b86ee69383 streamclone: use command executor for wire protocol commands Gregory Szorc <gregory.szorc@gmail.com> parents: 35804 diff changeset	151 # TODO strictly speaking, this code should all be inside the context
65b86ee69383 streamclone: use command executor for wire protocol commands Gregory Szorc <gregory.szorc@gmail.com> parents: 35804 diff changeset	152 # manager because the context manager is supposed to ensure all wire state
65b86ee69383 streamclone: use command executor for wire protocol commands Gregory Szorc <gregory.szorc@gmail.com> parents: 35804 diff changeset	153 # is flushed when exiting. But the legacy peers don't do this, so it
65b86ee69383 streamclone: use command executor for wire protocol commands Gregory Szorc <gregory.szorc@gmail.com> parents: 35804 diff changeset	154 # doesn't matter.
26459 3b28ffde133a streamclone: move streamin() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26458 diff changeset	155 l = fp.readline()
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26458 diff changeset	156 try:
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26458 diff changeset	157 resp = int(l)
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26458 diff changeset	158 except ValueError:
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26458 diff changeset	159 raise error.ResponseError(
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	160 _('unexpected response from remote server:'), l
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	161 )
26459 3b28ffde133a streamclone: move streamin() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26458 diff changeset	162 if resp == 1:
26587 56b2bcea2529 error: get Abort from 'error' instead of 'util' Pierre-Yves David <pierre-yves.david@fb.com> parents: 26470 diff changeset	163 raise error.Abort(_('operation forbidden by server'))
26459 3b28ffde133a streamclone: move streamin() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26458 diff changeset	164 elif resp == 2:
26587 56b2bcea2529 error: get Abort from 'error' instead of 'util' Pierre-Yves David <pierre-yves.david@fb.com> parents: 26470 diff changeset	165 raise error.Abort(_('locking the remote repository failed'))
26459 3b28ffde133a streamclone: move streamin() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26458 diff changeset	166 elif resp != 0:
26587 56b2bcea2529 error: get Abort from 'error' instead of 'util' Pierre-Yves David <pierre-yves.david@fb.com> parents: 26470 diff changeset	167 raise error.Abort(_('the server sent an unknown error code'))
26459 3b28ffde133a streamclone: move streamin() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26458 diff changeset	168
26468 19bbd53af46d streamclone: move payload header line consumption Gregory Szorc <gregory.szorc@gmail.com> parents: 26467 diff changeset	169 l = fp.readline()
19bbd53af46d streamclone: move payload header line consumption Gregory Szorc <gregory.szorc@gmail.com> parents: 26467 diff changeset	170 try:
19bbd53af46d streamclone: move payload header line consumption Gregory Szorc <gregory.szorc@gmail.com> parents: 26467 diff changeset	171 filecount, bytecount = map(int, l.split(' ', 1))
19bbd53af46d streamclone: move payload header line consumption Gregory Szorc <gregory.szorc@gmail.com> parents: 26467 diff changeset	172 except (ValueError, TypeError):
19bbd53af46d streamclone: move payload header line consumption Gregory Szorc <gregory.szorc@gmail.com> parents: 26467 diff changeset	173 raise error.ResponseError(
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	174 _('unexpected response from remote server:'), l
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	175 )
26468 19bbd53af46d streamclone: move payload header line consumption Gregory Szorc <gregory.szorc@gmail.com> parents: 26467 diff changeset	176
27850 49cfddbf54ba with: use context manager in maybeperformlegacystreamclone Bryan O'Sullivan <bryano@fb.com> parents: 27845 diff changeset	177 with repo.lock():
26468 19bbd53af46d streamclone: move payload header line consumption Gregory Szorc <gregory.szorc@gmail.com> parents: 26467 diff changeset	178 consumev1(repo, fp, filecount, bytecount)
26461 09cc3c2e9ece streamclone: move applyremotedata() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26460 diff changeset	179
09cc3c2e9ece streamclone: move applyremotedata() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26460 diff changeset	180 # new requirements = old non-format requirements +
09cc3c2e9ece streamclone: move applyremotedata() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26460 diff changeset	181 # new format-related remote requirements
09cc3c2e9ece streamclone: move applyremotedata() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26460 diff changeset	182 # requirements from the streamed-in repository
09cc3c2e9ece streamclone: move applyremotedata() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26460 diff changeset	183 repo.requirements = requirements \| (
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	184 repo.requirements - repo.supportedformats
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	185 )
39716 b10d145837bc localrepo: extract resolving of opener options to standalone functions Gregory Szorc <gregory.szorc@gmail.com> parents: 38823 diff changeset	186 repo.svfs.options = localrepo.resolvestorevfsoptions(
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	187 repo.ui, repo.requirements, repo.features
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	188 )
26461 09cc3c2e9ece streamclone: move applyremotedata() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26460 diff changeset	189 repo._writerequirements()
09cc3c2e9ece streamclone: move applyremotedata() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26460 diff changeset	190
09cc3c2e9ece streamclone: move applyremotedata() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26460 diff changeset	191 if rbranchmap:
41626 328ca3b9e545 branchmap: encapsulate cache updating in the map itself Martijn Pieters <mj@octobus.net> parents: 40544 diff changeset	192 repo._branchcaches.replace(repo, rbranchmap)
26461 09cc3c2e9ece streamclone: move applyremotedata() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26460 diff changeset	193
09cc3c2e9ece streamclone: move applyremotedata() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26460 diff changeset	194 repo.invalidate()
26445 f134fb33c906 streamclone: move streaming clone logic from localrepo Gregory Szorc <gregory.szorc@gmail.com> parents: 26444 diff changeset	195
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	196
32764 33b7283a3828 streamclone: consider secret changesets (BC) (issue5589) Gregory Szorc <gregory.szorc@gmail.com> parents: 32299 diff changeset	197 def allowservergeneration(repo):
26444 623743010133 streamclone: move _allowstream() from wireproto Gregory Szorc <gregory.szorc@gmail.com> parents: 26443 diff changeset	198 """Whether streaming clones are allowed from the server."""
40029 51f10e6d66c7 streamclone: don't support stream clone unless repo feature present Gregory Szorc <gregory.szorc@gmail.com> parents: 39855 diff changeset	199 if repository.REPO_FEATURE_STREAM_CLONE not in repo.features:
51f10e6d66c7 streamclone: don't support stream clone unless repo feature present Gregory Szorc <gregory.szorc@gmail.com> parents: 39855 diff changeset	200 return False
51f10e6d66c7 streamclone: don't support stream clone unless repo feature present Gregory Szorc <gregory.szorc@gmail.com> parents: 39855 diff changeset	201
33499 0407a51b9d8c codemod: register core configitems using a script Jun Wu <quark@fb.com> parents: 33411 diff changeset	202 if not repo.ui.configbool('server', 'uncompressed', untrusted=True):
32764 33b7283a3828 streamclone: consider secret changesets (BC) (issue5589) Gregory Szorc <gregory.szorc@gmail.com> parents: 32299 diff changeset	203 return False
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589) Gregory Szorc <gregory.szorc@gmail.com> parents: 32299 diff changeset	204
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589) Gregory Szorc <gregory.szorc@gmail.com> parents: 32299 diff changeset	205 # The way stream clone works makes it impossible to hide secret changesets.
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589) Gregory Szorc <gregory.szorc@gmail.com> parents: 32299 diff changeset	206 # So don't allow this by default.
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589) Gregory Szorc <gregory.szorc@gmail.com> parents: 32299 diff changeset	207 secret = phases.hassecret(repo)
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589) Gregory Szorc <gregory.szorc@gmail.com> parents: 32299 diff changeset	208 if secret:
33226 b045344fe35e configitems: register the 'server.uncompressedallowsecret' config Pierre-Yves David <pierre-yves.david@octobus.net> parents: 32764 diff changeset	209 return repo.ui.configbool('server', 'uncompressedallowsecret')
32764 33b7283a3828 streamclone: consider secret changesets (BC) (issue5589) Gregory Szorc <gregory.szorc@gmail.com> parents: 32299 diff changeset	210
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589) Gregory Szorc <gregory.szorc@gmail.com> parents: 32299 diff changeset	211 return True
26444 623743010133 streamclone: move _allowstream() from wireproto Gregory Szorc <gregory.szorc@gmail.com> parents: 26443 diff changeset	212
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	213
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	214 # This is it's own function so extensions can override it.
40339 f0e8f27768eb streamclone: pass narrowing related info in _walkstreamfiles() Pulkit Goyal <pulkit@yandex-team.ru> parents: 40338 diff changeset	215 def _walkstreamfiles(repo, matcher=None):
f0e8f27768eb streamclone: pass narrowing related info in _walkstreamfiles() Pulkit Goyal <pulkit@yandex-team.ru> parents: 40338 diff changeset	216 return repo.store.walk(matcher)
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	217
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	218
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	219 def generatev1(repo):
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	220 """Emit content for version 1 of a streaming clone.
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	221
26469 fb743268510e streamclone: move payload header generation into own function Gregory Szorc <gregory.szorc@gmail.com> parents: 26468 diff changeset	222 This returns a 3-tuple of (file count, byte size, data iterator).
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	223
26469 fb743268510e streamclone: move payload header generation into own function Gregory Szorc <gregory.szorc@gmail.com> parents: 26468 diff changeset	224 The data iterator consists of N entries for each file being transferred.
fb743268510e streamclone: move payload header generation into own function Gregory Szorc <gregory.szorc@gmail.com> parents: 26468 diff changeset	225 Each file entry starts as a line with the file name and integer size
fb743268510e streamclone: move payload header generation into own function Gregory Szorc <gregory.szorc@gmail.com> parents: 26468 diff changeset	226 delimited by a null byte.
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	227
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	228 The raw file data follows. Following the raw file data is the next file
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	229 entry, or EOF.
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	230
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	231 When used on the wire protocol, an additional line indicating protocol
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	232 success will be prepended to the stream. This function is not responsible
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	233 for adding it.
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	234
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	235 This function will obtain a repository lock to ensure a consistent view of
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	236 the store is captured. It therefore may raise LockError.
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	237 """
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	238 entries = []
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	239 total_bytes = 0
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	240 # Get consistent snapshot of repo, lock during scan.
27845 7417e1c10253 with: use context manager in streamclone generatev1 Bryan O'Sullivan <bryano@fb.com> parents: 27794 diff changeset	241 with repo.lock():
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	242 repo.ui.debug('scanning\n')
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	243 for name, ename, size in _walkstreamfiles(repo):
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	244 if size:
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	245 entries.append((name, size))
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	246 total_bytes += size
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	247
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	248 repo.ui.debug(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	249 '%d files, %d bytes to transfer\n' % (len(entries), total_bytes)
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	250 )
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	251
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	252 svfs = repo.svfs
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	253 debugflag = repo.ui.debugflag
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	254
26469 fb743268510e streamclone: move payload header generation into own function Gregory Szorc <gregory.szorc@gmail.com> parents: 26468 diff changeset	255 def emitrevlogdata():
33258 761ccfeff8b1 streamclone: stop using 'vfs.mustaudit = False' Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33257 diff changeset	256 for name, size in entries:
761ccfeff8b1 streamclone: stop using 'vfs.mustaudit = False' Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33257 diff changeset	257 if debugflag:
761ccfeff8b1 streamclone: stop using 'vfs.mustaudit = False' Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33257 diff changeset	258 repo.ui.debug('sending %s (%d bytes)\n' % (name, size))
761ccfeff8b1 streamclone: stop using 'vfs.mustaudit = False' Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33257 diff changeset	259 # partially encode name over the wire for backwards compat
761ccfeff8b1 streamclone: stop using 'vfs.mustaudit = False' Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33257 diff changeset	260 yield '%s\0%d\n' % (store.encodedir(name), size)
33411 50b49bb0fff3 streamclone: comment why path auditing is disabled in generatev1() Yuya Nishihara <yuya@tcha.org> parents: 33410 diff changeset	261 # auditing at this stage is both pointless (paths are already
50b49bb0fff3 streamclone: comment why path auditing is disabled in generatev1() Yuya Nishihara <yuya@tcha.org> parents: 33410 diff changeset	262 # trusted by the local repo) and expensive
33410 c784308305c6 streamclone: close large revlog files explicitly in generatev1() Yuya Nishihara <yuya@tcha.org> parents: 33258 diff changeset	263 with svfs(name, 'rb', auditpath=False) as fp:
c784308305c6 streamclone: close large revlog files explicitly in generatev1() Yuya Nishihara <yuya@tcha.org> parents: 33258 diff changeset	264 if size <= 65536:
33258 761ccfeff8b1 streamclone: stop using 'vfs.mustaudit = False' Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33257 diff changeset	265 yield fp.read(size)
33410 c784308305c6 streamclone: close large revlog files explicitly in generatev1() Yuya Nishihara <yuya@tcha.org> parents: 33258 diff changeset	266 else:
c784308305c6 streamclone: close large revlog files explicitly in generatev1() Yuya Nishihara <yuya@tcha.org> parents: 33258 diff changeset	267 for chunk in util.filechunkiter(fp, limit=size):
c784308305c6 streamclone: close large revlog files explicitly in generatev1() Yuya Nishihara <yuya@tcha.org> parents: 33258 diff changeset	268 yield chunk
26469 fb743268510e streamclone: move payload header generation into own function Gregory Szorc <gregory.szorc@gmail.com> parents: 26468 diff changeset	269
fb743268510e streamclone: move payload header generation into own function Gregory Szorc <gregory.szorc@gmail.com> parents: 26468 diff changeset	270 return len(entries), total_bytes, emitrevlogdata()
fb743268510e streamclone: move payload header generation into own function Gregory Szorc <gregory.szorc@gmail.com> parents: 26468 diff changeset	271
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	272
26469 fb743268510e streamclone: move payload header generation into own function Gregory Szorc <gregory.szorc@gmail.com> parents: 26468 diff changeset	273 def generatev1wireproto(repo):
fb743268510e streamclone: move payload header generation into own function Gregory Szorc <gregory.szorc@gmail.com> parents: 26468 diff changeset	274 """Emit content for version 1 of streaming clone suitable for the wire.
fb743268510e streamclone: move payload header generation into own function Gregory Szorc <gregory.szorc@gmail.com> parents: 26468 diff changeset	275
35495 ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	276 This is the data output from ``generatev1()`` with 2 header lines. The
ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	277 first line indicates overall success. The 2nd contains the file count and
ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	278 byte size of payload.
ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	279
ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	280 The success line contains "0" for success, "1" for stream generation not
ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	281 allowed, and "2" for error locking the repository (possibly indicating
ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	282 a permissions error for the server process).
26469 fb743268510e streamclone: move payload header generation into own function Gregory Szorc <gregory.szorc@gmail.com> parents: 26468 diff changeset	283 """
35495 ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	284 if not allowservergeneration(repo):
ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	285 yield '1\n'
ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	286 return
ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	287
ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	288 try:
ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	289 filecount, bytecount, it = generatev1(repo)
ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	290 except error.LockError:
ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	291 yield '2\n'
ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	292 return
ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	293
ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	294 # Indicates successful response.
ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	295 yield '0\n'
26469 fb743268510e streamclone: move payload header generation into own function Gregory Szorc <gregory.szorc@gmail.com> parents: 26468 diff changeset	296 yield '%d %d\n' % (filecount, bytecount)
fb743268510e streamclone: move payload header generation into own function Gregory Szorc <gregory.szorc@gmail.com> parents: 26468 diff changeset	297 for chunk in it:
fb743268510e streamclone: move payload header generation into own function Gregory Szorc <gregory.szorc@gmail.com> parents: 26468 diff changeset	298 yield chunk
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	299
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	300
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	301 def generatebundlev1(repo, compression='UN'):
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	302 """Emit content for version 1 of a stream clone bundle.
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	303
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	304 The first 4 bytes of the output ("HGS1") denote this as stream clone
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	305 bundle version 1.
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	306
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	307 The next 2 bytes indicate the compression type. Only "UN" is currently
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	308 supported.
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	309
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	310 The next 16 bytes are two 64-bit big endian unsigned integers indicating
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	311 file count and byte count, respectively.
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	312
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	313 The next 2 bytes is a 16-bit big endian unsigned short declaring the length
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	314 of the requirements string, including a trailing \0. The following N bytes
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	315 are the requirements string, which is ASCII containing a comma-delimited
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	316 list of repo requirements that are needed to support the data.
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	317
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	318 The remaining content is the output of ``generatev1()`` (which may be
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	319 compressed in the future).
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	320
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	321 Returns a tuple of (requirements, data generator).
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	322 """
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	323 if compression != 'UN':
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	324 raise ValueError('we do not support the compression argument yet')
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	325
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	326 requirements = repo.requirements & repo.supportedformats
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	327 requires = ','.join(sorted(requirements))
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	328
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	329 def gen():
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	330 yield 'HGS1'
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	331 yield compression
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	332
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	333 filecount, bytecount, it = generatev1(repo)
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	334 repo.ui.status(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	335 _('writing %d bytes for %d files\n') % (bytecount, filecount)
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	336 )
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	337
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	338 yield struct.pack('>QQ', filecount, bytecount)
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	339 yield struct.pack('>H', len(requires) + 1)
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	340 yield requires + '\0'
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	341
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	342 # This is where we'll add compression in the future.
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	343 assert compression == 'UN'
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	344
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	345 progress = repo.ui.makeprogress(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	346 _('bundle'), total=bytecount, unit=_('bytes')
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	347 )
38355 e59eaf51cc0d streamclone: use progress helper Martin von Zweigbergk <martinvonz@google.com> parents: 38197 diff changeset	348 progress.update(0)
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	349
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	350 for chunk in it:
38355 e59eaf51cc0d streamclone: use progress helper Martin von Zweigbergk <martinvonz@google.com> parents: 38197 diff changeset	351 progress.increment(step=len(chunk))
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	352 yield chunk
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	353
38379 ef692614e601 progress: hide update(None) in a new complete() method Martin von Zweigbergk <martinvonz@google.com> parents: 38366 diff changeset	354 progress.complete()
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	355
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	356 return requirements, gen()
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	357
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	358
26468 19bbd53af46d streamclone: move payload header line consumption Gregory Szorc <gregory.szorc@gmail.com> parents: 26467 diff changeset	359 def consumev1(repo, fp, filecount, bytecount):
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	360 """Apply the contents from version 1 of a streaming clone file handle.
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	361
30342 318a24b52eeb spelling: fixes of non-dictionary words Mads Kiilerich <madski@unity3d.com> parents: 29923 diff changeset	362 This takes the output from "stream_out" and applies it to the specified
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	363 repository.
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	364
30342 318a24b52eeb spelling: fixes of non-dictionary words Mads Kiilerich <madski@unity3d.com> parents: 29923 diff changeset	365 Like "stream_out," the status line added by the wire protocol is not
318a24b52eeb spelling: fixes of non-dictionary words Mads Kiilerich <madski@unity3d.com> parents: 29923 diff changeset	366 handled by this function.
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	367 """
27859 f55a5ace8e69 with: use context manager in streamclone consumev1 Bryan O'Sullivan <bryano@fb.com> parents: 27850 diff changeset	368 with repo.lock():
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	369 repo.ui.status(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	370 _('%d files to transfer, %s of data\n')
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	371 % (filecount, util.bytecount(bytecount))
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	372 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	373 progress = repo.ui.makeprogress(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	374 _('clone'), total=bytecount, unit=_('bytes')
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	375 )
38355 e59eaf51cc0d streamclone: use progress helper Martin von Zweigbergk <martinvonz@google.com> parents: 38197 diff changeset	376 progress.update(0)
30995 22fbca1d11ed mercurial: switch to util.timer for all interval timings Simon Farnsworth <simonfar@fb.com> parents: 30342 diff changeset	377 start = util.timer()
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	378
29923 519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29921 diff changeset	379 # TODO: get rid of (potential) inconsistency
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29921 diff changeset	380 #
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29921 diff changeset	381 # If transaction is started and any @filecache property is
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29921 diff changeset	382 # changed at this point, it causes inconsistency between
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29921 diff changeset	383 # in-memory cached property and streamclone-ed file on the
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29921 diff changeset	384 # disk. Nested transaction prevents transaction scope "clone"
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29921 diff changeset	385 # below from writing in-memory changes out at the end of it,
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29921 diff changeset	386 # even though in-memory changes are discarded at the end of it
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29921 diff changeset	387 # regardless of transaction nesting.
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29921 diff changeset	388 #
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29921 diff changeset	389 # But transaction nesting can't be simply prohibited, because
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29921 diff changeset	390 # nesting occurs also in ordinary case (e.g. enabling
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29921 diff changeset	391 # clonebundles).
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29921 diff changeset	392
27870 ed9950ba091e with: use context manager for transaction in consumev1 Bryan O'Sullivan <bryano@fb.com> parents: 27859 diff changeset	393 with repo.transaction('clone'):
27897 2fdbf22a1b63 streamclone: use backgroundfilecloser (issue4889) Gregory Szorc <gregory.szorc@gmail.com> parents: 27896 diff changeset	394 with repo.svfs.backgroundclosing(repo.ui, expectedcount=filecount):
38823 e7aa113b14f7 global: use pycompat.xrange() Gregory Szorc <gregory.szorc@gmail.com> parents: 38380 diff changeset	395 for i in pycompat.xrange(filecount):
27896 1d29893240cc streamclone: indent code Gregory Szorc <gregory.szorc@gmail.com> parents: 27882 diff changeset	396 # XXX doesn't support '\n' or '\r' in filenames
1d29893240cc streamclone: indent code Gregory Szorc <gregory.szorc@gmail.com> parents: 27882 diff changeset	397 l = fp.readline()
1d29893240cc streamclone: indent code Gregory Szorc <gregory.szorc@gmail.com> parents: 27882 diff changeset	398 try:
1d29893240cc streamclone: indent code Gregory Szorc <gregory.szorc@gmail.com> parents: 27882 diff changeset	399 name, size = l.split('\0', 1)
1d29893240cc streamclone: indent code Gregory Szorc <gregory.szorc@gmail.com> parents: 27882 diff changeset	400 size = int(size)
1d29893240cc streamclone: indent code Gregory Szorc <gregory.szorc@gmail.com> parents: 27882 diff changeset	401 except (ValueError, TypeError):
1d29893240cc streamclone: indent code Gregory Szorc <gregory.szorc@gmail.com> parents: 27882 diff changeset	402 raise error.ResponseError(
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	403 _('unexpected response from remote server:'), l
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	404 )
27896 1d29893240cc streamclone: indent code Gregory Szorc <gregory.szorc@gmail.com> parents: 27882 diff changeset	405 if repo.ui.debugflag:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	406 repo.ui.debug(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	407 'adding %s (%s)\n' % (name, util.bytecount(size))
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	408 )
27896 1d29893240cc streamclone: indent code Gregory Szorc <gregory.szorc@gmail.com> parents: 27882 diff changeset	409 # for backwards compat, name was partially encoded
27897 2fdbf22a1b63 streamclone: use backgroundfilecloser (issue4889) Gregory Szorc <gregory.szorc@gmail.com> parents: 27896 diff changeset	410 path = store.decodedir(name)
2fdbf22a1b63 streamclone: use backgroundfilecloser (issue4889) Gregory Szorc <gregory.szorc@gmail.com> parents: 27896 diff changeset	411 with repo.svfs(path, 'w', backgroundclose=True) as ofp:
27896 1d29893240cc streamclone: indent code Gregory Szorc <gregory.szorc@gmail.com> parents: 27882 diff changeset	412 for chunk in util.filechunkiter(fp, limit=size):
38355 e59eaf51cc0d streamclone: use progress helper Martin von Zweigbergk <martinvonz@google.com> parents: 38197 diff changeset	413 progress.increment(step=len(chunk))
27896 1d29893240cc streamclone: indent code Gregory Szorc <gregory.szorc@gmail.com> parents: 27882 diff changeset	414 ofp.write(chunk)
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	415
29923 519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29921 diff changeset	416 # force @filecache properties to be reloaded from
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29921 diff changeset	417 # streamclone-ed file at next access
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29921 diff changeset	418 repo.invalidate(clearfilecache=True)
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	419
30995 22fbca1d11ed mercurial: switch to util.timer for all interval timings Simon Farnsworth <simonfar@fb.com> parents: 30342 diff changeset	420 elapsed = util.timer() - start
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	421 if elapsed <= 0:
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	422 elapsed = 0.001
38379 ef692614e601 progress: hide update(None) in a new complete() method Martin von Zweigbergk <martinvonz@google.com> parents: 38366 diff changeset	423 progress.complete()
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	424 repo.ui.status(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	425 _('transferred %s in %.1f seconds (%s/sec)\n')
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	426 % (
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	427 util.bytecount(bytecount),
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	428 elapsed,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	429 util.bytecount(bytecount / elapsed),
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	430 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	431 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	432
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	433
27882 319b0bf6ecc9 streamclone: extract code for reading header fields Gregory Szorc <gregory.szorc@gmail.com> parents: 27870 diff changeset	434 def readbundle1header(fp):
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	435 compression = fp.read(2)
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	436 if compression != 'UN':
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	437 raise error.Abort(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	438 _('only uncompressed stream clone bundles are ' 'supported; got %s')
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	439 % compression
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	440 )
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	441
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	442 filecount, bytecount = struct.unpack('>QQ', fp.read(16))
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	443 requireslen = struct.unpack('>H', fp.read(2))[0]
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	444 requires = fp.read(requireslen)
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	445
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	446 if not requires.endswith('\0'):
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	447 raise error.Abort(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	448 _(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	449 'malformed stream clone bundle: '
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	450 'requirements not properly encoded'
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	451 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	452 )
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	453
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	454 requirements = set(requires.rstrip('\0').split(','))
27882 319b0bf6ecc9 streamclone: extract code for reading header fields Gregory Szorc <gregory.szorc@gmail.com> parents: 27870 diff changeset	455
319b0bf6ecc9 streamclone: extract code for reading header fields Gregory Szorc <gregory.szorc@gmail.com> parents: 27870 diff changeset	456 return filecount, bytecount, requirements
319b0bf6ecc9 streamclone: extract code for reading header fields Gregory Szorc <gregory.szorc@gmail.com> parents: 27870 diff changeset	457
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	458
27882 319b0bf6ecc9 streamclone: extract code for reading header fields Gregory Szorc <gregory.szorc@gmail.com> parents: 27870 diff changeset	459 def applybundlev1(repo, fp):
319b0bf6ecc9 streamclone: extract code for reading header fields Gregory Szorc <gregory.szorc@gmail.com> parents: 27870 diff changeset	460 """Apply the content from a stream clone bundle version 1.
319b0bf6ecc9 streamclone: extract code for reading header fields Gregory Szorc <gregory.szorc@gmail.com> parents: 27870 diff changeset	461
319b0bf6ecc9 streamclone: extract code for reading header fields Gregory Szorc <gregory.szorc@gmail.com> parents: 27870 diff changeset	462 We assume the 4 byte header has been read and validated and the file handle
319b0bf6ecc9 streamclone: extract code for reading header fields Gregory Szorc <gregory.szorc@gmail.com> parents: 27870 diff changeset	463 is at the 2 byte compression identifier.
319b0bf6ecc9 streamclone: extract code for reading header fields Gregory Szorc <gregory.szorc@gmail.com> parents: 27870 diff changeset	464 """
319b0bf6ecc9 streamclone: extract code for reading header fields Gregory Szorc <gregory.szorc@gmail.com> parents: 27870 diff changeset	465 if len(repo):
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	466 raise error.Abort(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	467 _('cannot apply stream clone bundle on non-empty ' 'repo')
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	468 )
27882 319b0bf6ecc9 streamclone: extract code for reading header fields Gregory Szorc <gregory.szorc@gmail.com> parents: 27870 diff changeset	469
319b0bf6ecc9 streamclone: extract code for reading header fields Gregory Szorc <gregory.szorc@gmail.com> parents: 27870 diff changeset	470 filecount, bytecount, requirements = readbundle1header(fp)
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	471 missingreqs = requirements - repo.supportedformats
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	472 if missingreqs:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	473 raise error.Abort(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	474 _('unable to apply stream clone: ' 'unsupported format: %s')
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	475 % ', '.join(sorted(missingreqs))
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	476 )
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	477
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	478 consumev1(repo, fp, filecount, bytecount)
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	479
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	480
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	481 class streamcloneapplier(object):
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	482 """Class to manage applying streaming clone bundles.
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	483
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	484 We need to wrap ``applybundlev1()`` in a dedicated type to enable bundle
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	485 readers to perform bundle type-specific functionality.
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	486 """
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	487
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	488 def __init__(self, fh):
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	489 self._fh = fh
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	490
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	491 def apply(self, repo):
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	492 return applybundlev1(repo, self._fh)
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	493
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	494
35765 56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	495 # type of file to stream
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	496 _fileappend = 0 # append only file
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	497 _filefull = 1 # full snapshot file
35765 56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	498
35767 5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	499 # Source of the file
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	500 _srcstore = 's' # store (svfs)
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	501 _srccache = 'c' # cache (cache)
35767 5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	502
35765 56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	503 # This is it's own function so extensions can override it.
56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	504 def _walkstreamfullstorefiles(repo):
56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	505 """list snapshot file from the store"""
56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	506 fnames = []
56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	507 if not repo.publishing():
56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	508 fnames.append('phaseroots')
56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	509 return fnames
56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	510
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	511
35767 5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	512 def _filterfull(entry, copy, vfsmap):
35765 56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	513 """actually copy the snapshot files"""
35767 5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	514 src, name, ftype, data = entry
35765 56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	515 if ftype != _filefull:
56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	516 return entry
35767 5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	517 return (src, name, ftype, copy(vfsmap[src].join(name)))
35765 56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	518
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	519
35765 56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	520 @contextlib.contextmanager
56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	521 def maketempcopies():
56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	522 """return a function to temporary copy file"""
56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	523 files = []
56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	524 try:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	525
35765 56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	526 def copy(src):
38197 aac4be30e250 py3: wrap tempfile.mkstemp() to use bytes path Yuya Nishihara <yuya@tcha.org> parents: 37638 diff changeset	527 fd, dst = pycompat.mkstemp()
35765 56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	528 os.close(fd)
56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	529 files.append(dst)
56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	530 util.copyfiles(src, dst, hardlink=True)
56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	531 return dst
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	532
35765 56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	533 yield copy
56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	534 finally:
56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	535 for tmp in files:
56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	536 util.tryunlink(tmp)
56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	537
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	538
35767 5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	539 def _makemap(repo):
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	540 """make a (src -> vfs) map for the repo"""
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	541 vfsmap = {
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	542 _srcstore: repo.svfs,
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	543 _srccache: repo.cachevfs,
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	544 }
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	545 # we keep repo.vfs out of the on purpose, ther are too many danger there
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	546 # (eg: .hg/hgrc)
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	547 assert repo.vfs not in vfsmap.values()
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	548
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	549 return vfsmap
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	550
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	551
35802 bbc07357b567 streamclone: rename '_emit' to '_emit2' for clarity Boris Feld <boris.feld@octobus.net> parents: 35767 diff changeset	552 def _emit2(repo, entries, totalfilesize):
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	553 """actually emit the stream bundle"""
35767 5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	554 vfsmap = _makemap(repo)
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	555 progress = repo.ui.makeprogress(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	556 _('bundle'), total=totalfilesize, unit=_('bytes')
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	557 )
38355 e59eaf51cc0d streamclone: use progress helper Martin von Zweigbergk <martinvonz@google.com> parents: 38197 diff changeset	558 progress.update(0)
38380 800f5a2c869e progress: make the progress helper a context manager Martin von Zweigbergk <martinvonz@google.com> parents: 38379 diff changeset	559 with maketempcopies() as copy, progress:
800f5a2c869e progress: make the progress helper a context manager Martin von Zweigbergk <martinvonz@google.com> parents: 38379 diff changeset	560 # copy is delayed until we are in the try
800f5a2c869e progress: make the progress helper a context manager Martin von Zweigbergk <martinvonz@google.com> parents: 38379 diff changeset	561 entries = [_filterfull(e, copy, vfsmap) for e in entries]
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	562 yield None # this release the lock on the repository
38380 800f5a2c869e progress: make the progress helper a context manager Martin von Zweigbergk <martinvonz@google.com> parents: 38379 diff changeset	563 seen = 0
35765 56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	564
38380 800f5a2c869e progress: make the progress helper a context manager Martin von Zweigbergk <martinvonz@google.com> parents: 38379 diff changeset	565 for src, name, ftype, data in entries:
800f5a2c869e progress: make the progress helper a context manager Martin von Zweigbergk <martinvonz@google.com> parents: 38379 diff changeset	566 vfs = vfsmap[src]
800f5a2c869e progress: make the progress helper a context manager Martin von Zweigbergk <martinvonz@google.com> parents: 38379 diff changeset	567 yield src
800f5a2c869e progress: make the progress helper a context manager Martin von Zweigbergk <martinvonz@google.com> parents: 38379 diff changeset	568 yield util.uvarintencode(len(name))
800f5a2c869e progress: make the progress helper a context manager Martin von Zweigbergk <martinvonz@google.com> parents: 38379 diff changeset	569 if ftype == _fileappend:
800f5a2c869e progress: make the progress helper a context manager Martin von Zweigbergk <martinvonz@google.com> parents: 38379 diff changeset	570 fp = vfs(name)
800f5a2c869e progress: make the progress helper a context manager Martin von Zweigbergk <martinvonz@google.com> parents: 38379 diff changeset	571 size = data
800f5a2c869e progress: make the progress helper a context manager Martin von Zweigbergk <martinvonz@google.com> parents: 38379 diff changeset	572 elif ftype == _filefull:
800f5a2c869e progress: make the progress helper a context manager Martin von Zweigbergk <martinvonz@google.com> parents: 38379 diff changeset	573 fp = open(data, 'rb')
800f5a2c869e progress: make the progress helper a context manager Martin von Zweigbergk <martinvonz@google.com> parents: 38379 diff changeset	574 size = util.fstat(fp).st_size
800f5a2c869e progress: make the progress helper a context manager Martin von Zweigbergk <martinvonz@google.com> parents: 38379 diff changeset	575 try:
800f5a2c869e progress: make the progress helper a context manager Martin von Zweigbergk <martinvonz@google.com> parents: 38379 diff changeset	576 yield util.uvarintencode(size)
800f5a2c869e progress: make the progress helper a context manager Martin von Zweigbergk <martinvonz@google.com> parents: 38379 diff changeset	577 yield name
800f5a2c869e progress: make the progress helper a context manager Martin von Zweigbergk <martinvonz@google.com> parents: 38379 diff changeset	578 if size <= 65536:
800f5a2c869e progress: make the progress helper a context manager Martin von Zweigbergk <martinvonz@google.com> parents: 38379 diff changeset	579 chunks = (fp.read(size),)
800f5a2c869e progress: make the progress helper a context manager Martin von Zweigbergk <martinvonz@google.com> parents: 38379 diff changeset	580 else:
800f5a2c869e progress: make the progress helper a context manager Martin von Zweigbergk <martinvonz@google.com> parents: 38379 diff changeset	581 chunks = util.filechunkiter(fp, limit=size)
800f5a2c869e progress: make the progress helper a context manager Martin von Zweigbergk <martinvonz@google.com> parents: 38379 diff changeset	582 for chunk in chunks:
800f5a2c869e progress: make the progress helper a context manager Martin von Zweigbergk <martinvonz@google.com> parents: 38379 diff changeset	583 seen += len(chunk)
800f5a2c869e progress: make the progress helper a context manager Martin von Zweigbergk <martinvonz@google.com> parents: 38379 diff changeset	584 progress.update(seen)
800f5a2c869e progress: make the progress helper a context manager Martin von Zweigbergk <martinvonz@google.com> parents: 38379 diff changeset	585 yield chunk
800f5a2c869e progress: make the progress helper a context manager Martin von Zweigbergk <martinvonz@google.com> parents: 38379 diff changeset	586 finally:
800f5a2c869e progress: make the progress helper a context manager Martin von Zweigbergk <martinvonz@google.com> parents: 38379 diff changeset	587 fp.close()
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	588
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	589
40398 0ac794e0e285 streamclone: include obsstore file into stream bundle if client can read it Anton Shestakov <av6@dwimlabs.net> parents: 40339 diff changeset	590 def generatev2(repo, includes, excludes, includeobsmarkers):
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	591 """Emit content for version 2 of a streaming clone.
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	592
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	593 the data stream consists the following entries:
35767 5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	594 1) A char representing the file destination (eg: store or cache)
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	595 2) A varint containing the length of the filename
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	596 3) A varint containing the length of file data
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	597 4) N bytes containing the filename (the internal, store-agnostic form)
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	598 5) N bytes containing the file data
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	599
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	600 Returns a 3-tuple of (file count, file size, data iterator).
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	601 """
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	602
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	603 with repo.lock():
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	604
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	605 entries = []
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	606 totalfilesize = 0
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	607
40339 f0e8f27768eb streamclone: pass narrowing related info in _walkstreamfiles() Pulkit Goyal <pulkit@yandex-team.ru> parents: 40338 diff changeset	608 matcher = None
f0e8f27768eb streamclone: pass narrowing related info in _walkstreamfiles() Pulkit Goyal <pulkit@yandex-team.ru> parents: 40338 diff changeset	609 if includes or excludes:
f0e8f27768eb streamclone: pass narrowing related info in _walkstreamfiles() Pulkit Goyal <pulkit@yandex-team.ru> parents: 40338 diff changeset	610 matcher = narrowspec.match(repo.root, includes, excludes)
f0e8f27768eb streamclone: pass narrowing related info in _walkstreamfiles() Pulkit Goyal <pulkit@yandex-team.ru> parents: 40338 diff changeset	611
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	612 repo.ui.debug('scanning\n')
40339 f0e8f27768eb streamclone: pass narrowing related info in _walkstreamfiles() Pulkit Goyal <pulkit@yandex-team.ru> parents: 40338 diff changeset	613 for name, ename, size in _walkstreamfiles(repo, matcher):
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	614 if size:
35767 5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	615 entries.append((_srcstore, name, _fileappend, size))
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	616 totalfilesize += size
35765 56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	617 for name in _walkstreamfullstorefiles(repo):
56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	618 if repo.svfs.exists(name):
56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	619 totalfilesize += repo.svfs.lstat(name).st_size
35767 5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	620 entries.append((_srcstore, name, _filefull, None))
40398 0ac794e0e285 streamclone: include obsstore file into stream bundle if client can read it Anton Shestakov <av6@dwimlabs.net> parents: 40339 diff changeset	621 if includeobsmarkers and repo.svfs.exists('obsstore'):
0ac794e0e285 streamclone: include obsstore file into stream bundle if client can read it Anton Shestakov <av6@dwimlabs.net> parents: 40339 diff changeset	622 totalfilesize += repo.svfs.lstat('obsstore').st_size
0ac794e0e285 streamclone: include obsstore file into stream bundle if client can read it Anton Shestakov <av6@dwimlabs.net> parents: 40339 diff changeset	623 entries.append((_srcstore, 'obsstore', _filefull, None))
35767 5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	624 for name in cacheutil.cachetocopy(repo):
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	625 if repo.cachevfs.exists(name):
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	626 totalfilesize += repo.cachevfs.lstat(name).st_size
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	627 entries.append((_srccache, name, _filefull, None))
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	628
35802 bbc07357b567 streamclone: rename '_emit' to '_emit2' for clarity Boris Feld <boris.feld@octobus.net> parents: 35767 diff changeset	629 chunks = _emit2(repo, entries, totalfilesize)
35765 56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	630 first = next(chunks)
56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	631 assert first is None
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	632
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	633 return len(entries), totalfilesize, chunks
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	634
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	635
35767 5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	636 @contextlib.contextmanager
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	637 def nested(*ctxs):
39773 97f2992c26f6 streamclone: reimplement nested context manager Augie Fackler <augie@google.com> parents: 39716 diff changeset	638 this = ctxs[0]
97f2992c26f6 streamclone: reimplement nested context manager Augie Fackler <augie@google.com> parents: 39716 diff changeset	639 rest = ctxs[1:]
97f2992c26f6 streamclone: reimplement nested context manager Augie Fackler <augie@google.com> parents: 39716 diff changeset	640 with this:
97f2992c26f6 streamclone: reimplement nested context manager Augie Fackler <augie@google.com> parents: 39716 diff changeset	641 if rest:
97f2992c26f6 streamclone: reimplement nested context manager Augie Fackler <augie@google.com> parents: 39716 diff changeset	642 with nested(*rest):
97f2992c26f6 streamclone: reimplement nested context manager Augie Fackler <augie@google.com> parents: 39716 diff changeset	643 yield
97f2992c26f6 streamclone: reimplement nested context manager Augie Fackler <augie@google.com> parents: 39716 diff changeset	644 else:
35767 5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	645 yield
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	646
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	647
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	648 def consumev2(repo, fp, filecount, filesize):
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	649 """Apply the contents from a version 2 streaming clone.
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	650
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	651 Data is read from an object that only needs to provide a ``read(size)``
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	652 method.
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	653 """
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	654 with repo.lock():
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	655 repo.ui.status(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	656 _('%d files to transfer, %s of data\n')
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	657 % (filecount, util.bytecount(filesize))
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	658 )
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	659
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	660 start = util.timer()
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	661 progress = repo.ui.makeprogress(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	662 _('clone'), total=filesize, unit=_('bytes')
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	663 )
38355 e59eaf51cc0d streamclone: use progress helper Martin von Zweigbergk <martinvonz@google.com> parents: 38197 diff changeset	664 progress.update(0)
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	665
35767 5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	666 vfsmap = _makemap(repo)
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	667
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	668 with repo.transaction('clone'):
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	669 ctxs = (vfs.backgroundclosing(repo.ui) for vfs in vfsmap.values())
35767 5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	670 with nested(*ctxs):
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	671 for i in range(filecount):
35803 3ad3aaeb1134 streamclone: use readexactly when reading stream v2 Boris Feld <boris.feld@octobus.net> parents: 35802 diff changeset	672 src = util.readexactly(fp, 1)
35767 5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	673 vfs = vfsmap[src]
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	674 namelen = util.uvarintdecodestream(fp)
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	675 datalen = util.uvarintdecodestream(fp)
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	676
35803 3ad3aaeb1134 streamclone: use readexactly when reading stream v2 Boris Feld <boris.feld@octobus.net> parents: 35802 diff changeset	677 name = util.readexactly(fp, namelen)
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	678
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	679 if repo.ui.debugflag:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	680 repo.ui.debug(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	681 'adding [%s] %s (%s)\n'
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	682 % (src, name, util.bytecount(datalen))
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	683 )
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	684
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	685 with vfs(name, 'w') as ofp:
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	686 for chunk in util.filechunkiter(fp, limit=datalen):
38355 e59eaf51cc0d streamclone: use progress helper Martin von Zweigbergk <martinvonz@google.com> parents: 38197 diff changeset	687 progress.increment(step=len(chunk))
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	688 ofp.write(chunk)
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	689
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	690 # force @filecache properties to be reloaded from
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	691 # streamclone-ed file at next access
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	692 repo.invalidate(clearfilecache=True)
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	693
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	694 elapsed = util.timer() - start
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	695 if elapsed <= 0:
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	696 elapsed = 0.001
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	697 repo.ui.status(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	698 _('transferred %s in %.1f seconds (%s/sec)\n')
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	699 % (
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	700 util.bytecount(progress.pos),
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	701 elapsed,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	702 util.bytecount(progress.pos / elapsed),
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	703 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	704 )
38379 ef692614e601 progress: hide update(None) in a new complete() method Martin von Zweigbergk <martinvonz@google.com> parents: 38366 diff changeset	705 progress.complete()
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	706
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	707
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	708 def applybundlev2(repo, fp, filecount, filesize, requirements):
39716 b10d145837bc localrepo: extract resolving of opener options to standalone functions Gregory Szorc <gregory.szorc@gmail.com> parents: 38823 diff changeset	709 from . import localrepo
b10d145837bc localrepo: extract resolving of opener options to standalone functions Gregory Szorc <gregory.szorc@gmail.com> parents: 38823 diff changeset	710
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	711 missingreqs = [r for r in requirements if r not in repo.supported]
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	712 if missingreqs:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	713 raise error.Abort(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	714 _('unable to apply stream clone: ' 'unsupported format: %s')
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	715 % ', '.join(sorted(missingreqs))
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	716 )
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	717
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	718 consumev2(repo, fp, filecount, filesize)
35804 2d3e486d09d0 streamclone: move requirement update into consumev2 Boris Feld <boris.feld@octobus.net> parents: 35803 diff changeset	719
2d3e486d09d0 streamclone: move requirement update into consumev2 Boris Feld <boris.feld@octobus.net> parents: 35803 diff changeset	720 # new requirements = old non-format requirements +
2d3e486d09d0 streamclone: move requirement update into consumev2 Boris Feld <boris.feld@octobus.net> parents: 35803 diff changeset	721 # new format-related remote requirements
2d3e486d09d0 streamclone: move requirement update into consumev2 Boris Feld <boris.feld@octobus.net> parents: 35803 diff changeset	722 # requirements from the streamed-in repository
2d3e486d09d0 streamclone: move requirement update into consumev2 Boris Feld <boris.feld@octobus.net> parents: 35803 diff changeset	723 repo.requirements = set(requirements) \| (
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	724 repo.requirements - repo.supportedformats
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	725 )
39716 b10d145837bc localrepo: extract resolving of opener options to standalone functions Gregory Szorc <gregory.szorc@gmail.com> parents: 38823 diff changeset	726 repo.svfs.options = localrepo.resolvestorevfsoptions(
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	727 repo.ui, repo.requirements, repo.features
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	728 )
35804 2d3e486d09d0 streamclone: move requirement update into consumev2 Boris Feld <boris.feld@octobus.net> parents: 35803 diff changeset	729 repo._writerequirements()

Mercurial > public > mercurial-scm > hg-stable

annotate mercurial/streamclone.py @ 43076:2372284d9457