Mercurial > public > mercurial-scm > evolve

annotate hgext3rd/pullbundle.py @ 4143:2ddd8785f8d7

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
pullbundle: add data about the cached bundle size and hit
author Pierre-Yves David <pierre-yves.david@octobus.net>
date Tue, 25 Sep 2018 18:41:51 +0200
parents 683ceec8d37e
children b5cd26712e4b
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
4128
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
1 # Extension to provide automatic caching of bundle server for pull
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
2 #
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
3 # Copyright 2018 Pierre-Yves David <pierre-yves.david@ens-lyon.org>
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
4 #
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
5 # This software may be used and distributed according to the terms of the
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
6 # GNU General Public License version 2 or any later version.
4132
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
7 """pullbundle: automatic server side bundle caching
4128
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
8
4132
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
9 General principle
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
10 =================
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
11
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
12 This extension provides a means for server to use pre-computed bundle for
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
13 serving arbitrary pulls. If missing, the necessary pre-computed bundle will be
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
14 generated on demand.
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
15
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
16 To maximize usage of existing cached bundle, each pull will be served through
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
17 multiple bundles. The bundle will be created using "standard range" from the
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
18 "stablerange" principle. The "stablerange" concept if already used for
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
19 obsmarkers discovery in the evolve extensions.
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
20
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
21 Using pull Bundle
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
22 =================
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
23
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
24 All configuration is only required server side.
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
25
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
26 The "stablerange" code currently still live in the evolve extensions, so for
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
27 now enabling that extensions is required:
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
28
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
29 You need at minimum the following configuration:
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
30
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
31 [extensions]
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
32 evolve=yes
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
33 pullbundle=yes
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
34 [experimental]
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
35 obshashrange.warm-cache = yes
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
36
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
37 If you do not want to use evolution server side, you should disable obsmarkers exchange:
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
38
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
39 [experimental]
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
40 evolution.exchange=no
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
41
4134
ab77f37fedf3 pullbundle: add a config option for the cache directory
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4133
diff changeset
42 Extra Configuration
ab77f37fedf3 pullbundle: add a config option for the cache directory
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4133
diff changeset
43 ===================
ab77f37fedf3 pullbundle: add a config option for the cache directory
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4133
diff changeset
44
ab77f37fedf3 pullbundle: add a config option for the cache directory
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4133
diff changeset
45 [pullbundle]
ab77f37fedf3 pullbundle: add a config option for the cache directory
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4133
diff changeset
46 # By default bundles are stored `.hg/cache/pullbundles/.
ab77f37fedf3 pullbundle: add a config option for the cache directory
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4133
diff changeset
47 # This can be changed with the following config:
ab77f37fedf3 pullbundle: add a config option for the cache directory
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4133
diff changeset
48 cache-directory=/absolute/path
ab77f37fedf3 pullbundle: add a config option for the cache directory
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4133
diff changeset
49
4132
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
50 Implementation status
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
51 =====================
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
52
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
53 Both for stablerange and pullbundle use "simple" initial implementations.
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
54 Theses implemenations focus on testing the algorithms and proving the features
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
55 works. Yet they are already useful and used in production.
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
56
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
57 Performances are expected to greatly improved in the final implementation,
4141
1421ff5c5c96 pullbundle: point out the output flooding issue from core
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4140
diff changeset
58 especially if some of it end up being compiled code.
4132
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
59
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
60 This first implementation lacks the ability to server the cached bundle from a
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
61 CDN. We'll want this limitation to be lifted quickly.
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
62
4141
1421ff5c5c96 pullbundle: point out the output flooding issue from core
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4140
diff changeset
63 The way mercurial core report progress is designed for the receival of a single
1421ff5c5c96 pullbundle: point out the output flooding issue from core
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4140
diff changeset
64 changegroup. So currently using pullbundle means flooding the user with output.
1421ff5c5c96 pullbundle: point out the output flooding issue from core
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4140
diff changeset
65 This will have to be fixed.
1421ff5c5c96 pullbundle: point out the output flooding issue from core
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4140
diff changeset
66
4132
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
67 Why is does this live in the same repository as evolve
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
68 ======================================================
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
69
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
70 There is no fundamental reasons for live in the same repository. However, the
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
71 stablerange data-structure lives in evolve, so it was simpler to put this new
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
72 extensions next to it. As soon as stable range have been upstreamed, we won't
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
73 need the dependency to the evolve extension anymore.
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
74 """
4142
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
75
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
76 import collections
4129
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
77 import errno
4142
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
78 import random
4129
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
79 import os
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
80
4128
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
81 from mercurial import (
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
82 changegroup,
4129
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
83 discovery,
4142
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
84 error,
4128
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
85 exchange,
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
86 narrowspec,
4129
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
87 node as nodemod,
4134
ab77f37fedf3 pullbundle: add a config option for the cache directory
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4133
diff changeset
88 registrar,
4142
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
89 scmutil,
4129
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
90 util,
4128
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
91 )
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
92
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
93 from mercurial.i18n import _
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
94
4135
47f1d7b4305d pullbundle: add basic extension metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4134
diff changeset
95 __version__ = '0.1.0.dev'
47f1d7b4305d pullbundle: add basic extension metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4134
diff changeset
96 testedwith = '4.7.1'
47f1d7b4305d pullbundle: add basic extension metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4134
diff changeset
97 # minimumhgversion = ''
47f1d7b4305d pullbundle: add basic extension metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4134
diff changeset
98 buglink = 'https://bz.mercurial-scm.org/'
47f1d7b4305d pullbundle: add basic extension metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4134
diff changeset
99
4142
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
100 cmdtable = {}
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
101 command = registrar.command(cmdtable)
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
102
4134
ab77f37fedf3 pullbundle: add a config option for the cache directory
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4133
diff changeset
103 configtable = {}
ab77f37fedf3 pullbundle: add a config option for the cache directory
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4133
diff changeset
104 configitem = registrar.configitem(configtable)
ab77f37fedf3 pullbundle: add a config option for the cache directory
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4133
diff changeset
105
ab77f37fedf3 pullbundle: add a config option for the cache directory
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4133
diff changeset
106 configitem('pullbundle', 'cache-directory',
ab77f37fedf3 pullbundle: add a config option for the cache directory
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4133
diff changeset
107 default=None,
ab77f37fedf3 pullbundle: add a config option for the cache directory
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4133
diff changeset
108 )
ab77f37fedf3 pullbundle: add a config option for the cache directory
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4133
diff changeset
109
4129
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
110 # generic wrapping
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
111
4128
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
112 def uisetup(ui):
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
113 exchange.getbundle2partsmapping['changegroup'] = _getbundlechangegrouppart
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
114
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
115 def _getbundlechangegrouppart(bundler, repo, source, bundlecaps=None,
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
116 b2caps=None, heads=None, common=None, **kwargs):
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
117 """add a changegroup part to the requested bundle"""
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
118 if not kwargs.get(r'cg', True):
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
119 return
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
120
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
121 version = '01'
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
122 cgversions = b2caps.get('changegroup')
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
123 if cgversions: # 3.1 and 3.2 ship with an empty value
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
124 cgversions = [v for v in cgversions
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
125 if v in changegroup.supportedoutgoingversions(repo)]
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
126 if not cgversions:
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
127 raise ValueError(_('no common changegroup version'))
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
128 version = max(cgversions)
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
129
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
130 outgoing = exchange._computeoutgoing(repo, heads, common)
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
131 if not outgoing.missing:
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
132 return
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
133
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
134 if kwargs.get(r'narrow', False):
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
135 include = sorted(filter(bool, kwargs.get(r'includepats', [])))
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
136 exclude = sorted(filter(bool, kwargs.get(r'excludepats', [])))
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
137 filematcher = narrowspec.match(repo.root, include=include,
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
138 exclude=exclude)
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
139 else:
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
140 filematcher = None
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
141
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
142 # START OF ALTERED PART
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
143 makeallcgpart(bundler.newpart, repo, outgoing, version, source, bundlecaps,
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
144 filematcher, cgversions)
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
145 # END OF ALTERED PART
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
146
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
147 if kwargs.get(r'narrow', False) and (include or exclude):
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
148 narrowspecpart = bundler.newpart('narrow:spec')
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
149 if include:
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
150 narrowspecpart.addparam(
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
151 'include', '\n'.join(include), mandatory=True)
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
152 if exclude:
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
153 narrowspecpart.addparam(
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
154 'exclude', '\n'.join(exclude), mandatory=True)
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
155
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
156 def makeallcgpart(newpart, repo, outgoing, version, source,
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
157 bundlecaps, filematcher, cgversions):
4129
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
158
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
159 pullbundle = not filematcher
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
160 if pullbundle and not util.safehasattr(repo, 'stablerange'):
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
161 repo.ui.warn('pullbundle: required extension "evolve" are missing, skipping pullbundle\n')
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
162 pullbundle = False
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
163 if filematcher:
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
164 makeonecgpart(newpart, repo, None, outgoing, version, source, bundlecaps,
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
165 filematcher, cgversions)
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
166 else:
4133
1293625d274d pullbundle: add an output about the time spent slicing the set
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4132
diff changeset
167 start = util.timer()
1293625d274d pullbundle: add an output about the time spent slicing the set
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4132
diff changeset
168 slices = sliceoutgoing(repo, outgoing)
1293625d274d pullbundle: add an output about the time spent slicing the set
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4132
diff changeset
169 end = util.timer()
1293625d274d pullbundle: add an output about the time spent slicing the set
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4132
diff changeset
170 msg = _('pullbundle-cache: "missing" set sliced into %d subranges '
1293625d274d pullbundle: add an output about the time spent slicing the set
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4132
diff changeset
171 'in %s seconds\n')
1293625d274d pullbundle: add an output about the time spent slicing the set
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4132
diff changeset
172 repo.ui.write(msg % (len(slices), end - start))
1293625d274d pullbundle: add an output about the time spent slicing the set
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4132
diff changeset
173 for sliceid, sliceout in slices:
4129
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
174 makeonecgpart(newpart, repo, sliceid, sliceout, version, source, bundlecaps,
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
175 filematcher, cgversions)
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
176
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
177 # stable range slicing
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
178
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
179 def sliceoutgoing(repo, outgoing):
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
180 cl = repo.changelog
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
181 rev = cl.nodemap.get
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
182 node = cl.node
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
183 revsort = repo.stablesort
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
184
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
185 missingrevs = set(rev(n) for n in outgoing.missing)
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
186 allslices = []
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
187 missingheads = [rev(n) for n in outgoing.missingheads]
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
188 for head in missingheads:
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
189 localslices = []
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
190 localmissing = set(repo.revs('%ld and ::%d', missingrevs, head))
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
191 while localmissing:
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
192 slicerevs = []
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
193 for r in revsort.walkfrom(repo, head):
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
194 if r not in missingrevs:
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
195 break
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
196 slicerevs.append(r)
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
197 slicenodes = [node(r) for r in slicerevs]
4140
9b71aa222f8e pullbundle: issue local branches in the right order
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4139
diff changeset
198 localslices.append(canonicalslices(repo, slicenodes))
4129
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
199 missingrevs.difference_update(slicerevs)
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
200 localmissing.difference_update(slicerevs)
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
201 if localmissing:
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
202 head = max(localmissing)
4140
9b71aa222f8e pullbundle: issue local branches in the right order
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4139
diff changeset
203 for s in reversed(localslices):
9b71aa222f8e pullbundle: issue local branches in the right order
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4139
diff changeset
204 allslices.extend(s)
4136
be3a94d3105f pullbundle: attempt to save stablerange cache after each computation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4135
diff changeset
205 # unknown subrange might had to be computed
be3a94d3105f pullbundle: attempt to save stablerange cache after each computation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4135
diff changeset
206 repo.stablerange.save(repo)
4129
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
207 return [(rangeid, outgoingfromnodes(repo, nodes))
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
208 for rangeid, nodes in allslices]
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
209
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
210 def canonicalslices(repo, nodes):
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
211 depth = repo.depthcache.get
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
212 stablerange = repo.stablerange
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
213 rangelength = lambda x: stablerange.rangelength(repo, x)
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
214 headrev = repo.changelog.rev(nodes[0])
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
215 nbrevs = len(nodes)
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
216 headdepth = depth(headrev)
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
217 skipped = headdepth - nbrevs
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
218 rangeid = (headrev, skipped)
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
219
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
220 subranges = canonicalsubranges(repo, stablerange, rangeid)
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
221 idx = 0
4130
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
222 slices = []
4129
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
223 nodes.reverse()
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
224 for rangeid in subranges:
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
225 size = rangelength(rangeid)
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
226 slices.append((rangeid, nodes[idx:idx + size]))
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
227 idx += size
4138
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
228 ### slow code block to validate ranges content
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
229 # rev = repo.changelog.nodemap.get
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
230 # for ri, ns in slices:
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
231 # a = set(rev(n) for n in ns)
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
232 # b = set(repo.stablerange.revsfromrange(repo, ri))
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
233 # l = repo.stablerange.rangelength(repo, ri)
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
234 # repo.ui.write('range-length: %d-%d %s %s\n' % (ri[0], ri[1], l, len(a)))
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
235 # if a != b:
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
236 # d = (ri[0], ri[1], b - a, a - b)
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
237 # repo.ui.write("mismatching content: %d-%d -%s +%s\n" % d)
4129
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
238 return slices
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
239
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
240 def canonicalsubranges(repo, stablerange, rangeid):
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
241 """slice a size of nodes into most reusable subranges
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
242
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
243 We try to slice a range into a set of "largest" and "canonical" stable
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
244 range.
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
245
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
246 It might make sense to move this function as a 'stablerange' method.
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
247 """
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
248 headrev, skip = rangeid
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
249 rangedepth = stablerange.depthrev(repo, rangeid[0])
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
250 canonicals = []
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
251
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
252 # 0. find the first power of 2 higher than this range depth
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
253 cursor = 1
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
254 while cursor <= rangedepth:
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
255 cursor *= 2
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
256
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
257 # 1. find first cupt
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
258 precut = cut = 0
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
259 while True:
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
260 if skip <= cut:
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
261 break
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
262 if cut + cursor < rangedepth:
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
263 precut = cut
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
264 cut += cursor
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
265 if cursor == 1:
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
266 break
4130
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
267 cursor //= 2
4129
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
268
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
269 # 2. optimise, bottom part
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
270 if skip != cut:
4138
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
271 currentsize = tailsize = cut - skip
4129
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
272 assert 0 < tailsize, tailsize
4138
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
273
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
274 # we need to take several "standard cut" in the bottom part
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
275 #
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
276 # This is similar to what we will do for the top part, we reusing the
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
277 # existing structure is a bit more complex.
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
278 allcuts = list(reversed(standardcut(tailsize)))
4129
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
279 prerange = (headrev, precut)
4138
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
280 ### slow code block to check we operate on the right data
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
281 # rev = repo.changelog.nodemap.get
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
282 # allrevs = [rev(n) for n in nodes]
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
283 # allrevs.reverse()
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
284 # prerevs = repo.stablerange.revsfromrange(repo, prerange)
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
285 # assert allrevs == prerevs[(len(prerevs) - len(allrevs)):]
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
286 # end of check
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
287 sub = list(stablerange.subranges(repo, prerange)[:-1])
4129
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
288
4138
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
289 bottomranges = []
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
290 # XXX we might be able to reuse core stable-range logic instead of
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
291 # redoing this manually
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
292 currentrange = sub.pop()
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
293 currentsize = stablerange.rangelength(repo, currentrange)
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
294 currentcut = None
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
295 while allcuts or currentcut is not None:
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
296 # get the next cut if needed
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
297 if currentcut is None:
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
298 currentcut = allcuts.pop()
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
299 # deal attemp a cut
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
300 if currentsize == currentcut:
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
301 bottomranges.append(currentrange)
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
302 currentcut = None
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
303 elif currentsize < currentcut:
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
304 bottomranges.append(currentrange)
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
305 currentcut -= currentsize
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
306 else: # currentsize > currentcut
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
307 newskip = currentrange[1] + (currentsize - currentcut)
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
308 currentsub = stablerange._slicesrangeat(repo, currentrange, newskip)
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
309 bottomranges.append(currentsub.pop())
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
310 sub.extend(currentsub)
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
311 currentcut = None
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
312 currentrange = sub.pop()
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
313 currentsize = stablerange.rangelength(repo, currentrange)
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
314 bottomranges.reverse()
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
315 canonicals.extend(bottomranges)
4129
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
316
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
317 # 3. take recursive subrange until we get to a power of two size?
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
318 current = (headrev, cut)
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
319 while not poweroftwo(stablerange.rangelength(repo, current)):
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
320 sub = stablerange.subranges(repo, current)
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
321 canonicals.extend(sub[:-1])
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
322 current = sub[-1]
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
323 canonicals.append(current)
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
324
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
325 return canonicals
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
326
4138
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
327 def standardcut(size):
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
328 assert 0 < size
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
329 # 0. find the first power of 2 higher than this range depth
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
330 cut = 1
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
331 while cut <= size:
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
332 cut *= 2
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
333
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
334 allcuts = []
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
335 # 1. find all standard expected cut
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
336 while 1 < cut and size:
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
337 cut //= 2
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
338 if cut <= size:
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
339 allcuts.append(cut)
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
340 size -= cut
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
341 return allcuts
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
342
4129
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
343 def poweroftwo(num):
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
344 return num and not num & (num - 1)
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
345
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
346 def outgoingfromnodes(repo, nodes):
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
347 return discovery.outgoing(repo,
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
348 missingroots=nodes,
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
349 missingheads=nodes)
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
350
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
351 # changegroup part construction
4128
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
352
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
353 def _changegroupinfo(repo, nodes, source):
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
354 if repo.ui.verbose or source == 'bundle':
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
355 repo.ui.status(_("%d changesets found\n") % len(nodes))
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
356
4130
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
357 def _makenewstream(newpart, repo, outgoing, version, source,
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
358 bundlecaps, filematcher, cgversions):
4128
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
359 old = changegroup._changegroupinfo
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
360 try:
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
361 changegroup._changegroupinfo = _changegroupinfo
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
362 cgstream = changegroup.makestream(repo, outgoing, version, source,
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
363 bundlecaps=bundlecaps,
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
364 filematcher=filematcher)
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
365 finally:
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
366 changegroup._changegroupinfo = old
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
367
4130
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
368 nbchanges = len(outgoing.missing)
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
369 pversion = None
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
370 if cgversions:
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
371 pversion = version
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
372 return (cgstream, nbchanges, pversion)
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
373
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
374 def _makepartfromstream(newpart, repo, cgstream, nbchanges, version):
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
375 # same as upstream code
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
376
4128
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
377 part = newpart('changegroup', data=cgstream)
4130
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
378 if version:
4128
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
379 part.addparam('version', version)
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
380
4130
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
381 part.addparam('nbchanges', '%d' % nbchanges,
4128
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
382 mandatory=False)
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
383
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
384 if 'treemanifest' in repo.requirements:
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
385 part.addparam('treemanifest', '1')
4130
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
386
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
387 # cache management
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
388
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
389 def cachedir(repo):
4134
ab77f37fedf3 pullbundle: add a config option for the cache directory
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4133
diff changeset
390 cachedir = repo.ui.config('pullbundle', 'cache-directory')
ab77f37fedf3 pullbundle: add a config option for the cache directory
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4133
diff changeset
391 if cachedir is not None:
ab77f37fedf3 pullbundle: add a config option for the cache directory
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4133
diff changeset
392 return cachedir
4130
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
393 return repo.cachevfs.join('pullbundles')
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
394
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
395 def getcache(repo, bundlename):
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
396 cdir = cachedir(repo)
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
397 bundlepath = os.path.join(cdir, bundlename)
4139
2bd652bece97 pullbundle: delay cache file opening
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4138
diff changeset
398 if not os.path.exists(bundlepath):
4130
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
399 return None
4139
2bd652bece97 pullbundle: delay cache file opening
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4138
diff changeset
400 # delay file opening as much as possible this introduce a small race
2bd652bece97 pullbundle: delay cache file opening
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4138
diff changeset
401 # condition if someone remove the file before we actually use it. However
2bd652bece97 pullbundle: delay cache file opening
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4138
diff changeset
402 # opening too many file will not work.
2bd652bece97 pullbundle: delay cache file opening
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4138
diff changeset
403
2bd652bece97 pullbundle: delay cache file opening
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4138
diff changeset
404 def data():
2bd652bece97 pullbundle: delay cache file opening
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4138
diff changeset
405 with open(bundlepath, 'rb') as fd:
2bd652bece97 pullbundle: delay cache file opening
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4138
diff changeset
406 for chunk in util.filechunkiter(fd):
2bd652bece97 pullbundle: delay cache file opening
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4138
diff changeset
407 yield chunk
2bd652bece97 pullbundle: delay cache file opening
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4138
diff changeset
408 return data()
4130
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
409
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
410 def cachewriter(repo, bundlename, stream):
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
411 cdir = cachedir(repo)
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
412 bundlepath = os.path.join(cdir, bundlename)
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
413 try:
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
414 os.makedirs(cdir)
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
415 except OSError as exc:
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
416 if exc.errno == errno.EEXIST:
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
417 pass
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
418 with util.atomictempfile(bundlepath) as cachefile:
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
419 for chunk in stream:
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
420 cachefile.write(chunk)
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
421 yield chunk
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
422
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
423 BUNDLEMASK = "%s-%s-%010iskip-%010isize.hg"
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
424
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
425 def makeonecgpart(newpart, repo, rangeid, outgoing, version, source,
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
426 bundlecaps, filematcher, cgversions):
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
427 bundlename = cachedata = None
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
428 if rangeid is not None:
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
429 nbchanges = repo.stablerange.rangelength(repo, rangeid)
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
430 headnode = nodemod.hex(repo.changelog.node(rangeid[0]))
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
431 # XXX do we need to use cgversion in there?
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
432 bundlename = BUNDLEMASK % (version, headnode, rangeid[1], nbchanges)
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
433 cachedata = getcache(repo, bundlename)
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
434 if cachedata is None:
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
435 partdata = _makenewstream(newpart, repo, outgoing, version, source,
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
436 bundlecaps, filematcher, cgversions)
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
437 if bundlename is not None:
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
438 cgstream = cachewriter(repo, bundlename, partdata[0])
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
439 partdata = (cgstream,) + partdata[1:]
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
440 else:
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
441 if repo.ui.verbose or source == 'bundle':
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
442 repo.ui.status(_("%d changesets found in caches\n") % nbchanges)
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
443 pversion = None
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
444 if cgversions:
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
445 pversion = version
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
446 partdata = (cachedata, nbchanges, pversion)
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
447 return _makepartfromstream(newpart, repo, *partdata)
4142
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
448
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
449 @command('^debugpullbundlecacheoverlap',
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
450 [('', 'count', 100, _('of "client" pulling')),
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
451 ],
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
452 _('hg debugpullbundlecacheoverlap [--client 100] REVSET'))
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
453 def debugpullbundlecacheoverlap(ui, repo, *revs, **opts):
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
454 '''Display statistic on bundle cache hit
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
455
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
456 This command "simulate pulls from multiple clients. Each using a random
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
457 subset of revisions defined by REVSET. And display statistic about the
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
458 overlap in bundle necessary to serve them.
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
459 '''
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
460 actionrevs = scmutil.revrange(repo, revs)
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
461 if not revs:
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
462 raise error.Abort('No revision selected')
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
463 count = opts['count']
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
464
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
465 bundlehits = collections.defaultdict(lambda: 0)
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
466 pullstats = []
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
467
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
468 rlen = lambda rangeid: repo.stablerange.rangelength(repo, rangeid)
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
469
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
470 repo.ui.write("gathering %d sample pulls within %d revisions\n"
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
471 % (count, len(actionrevs)))
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
472 for i in xrange(count):
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
473 repo.ui.progress('gathering data', i, total=count)
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
474 outgoing = takeonesample(repo, actionrevs)
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
475 ranges = sliceoutgoing(repo, outgoing)
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
476 hitranges = 0
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
477 hitchanges = 0
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
478 totalchanges = 0
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
479 for rangeid, __ in ranges:
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
480 length = rlen(rangeid)
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
481 totalchanges += length
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
482 if bundlehits[rangeid]:
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
483 hitranges += 1
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
484 hitchanges += rlen(rangeid)
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
485 bundlehits[rangeid] += 1
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
486 stats = (len(outgoing.missing),
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
487 totalchanges,
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
488 hitchanges,
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
489 len(ranges),
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
490 hitranges,
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
491 )
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
492 pullstats.append(stats)
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
493 repo.ui.progress('gathering data', None)
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
494
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
495 sizes = []
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
496 changesmissing = []
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
497 totalchanges = 0
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
498 totalcached = 0
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
499 changesratio = []
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
500 rangesratio = []
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
501 bundlecount = []
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
502 for entry in pullstats:
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
503 sizes.append(entry[0])
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
504 changesmissing.append(entry[1] - entry[2])
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
505 changesratio.append(entry[2] / float(entry[1]))
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
506 rangesratio.append(entry[4] / float(entry[3]))
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
507 bundlecount.append(entry[3])
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
508 totalchanges += entry[1]
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
509 totalcached += entry[2]
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
510
4143
2ddd8785f8d7 pullbundle: add data about the cached bundle size and hit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4142
diff changeset
511 cachedsizes = []
2ddd8785f8d7 pullbundle: add data about the cached bundle size and hit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4142
diff changeset
512 cachedhits = []
2ddd8785f8d7 pullbundle: add data about the cached bundle size and hit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4142
diff changeset
513 for rangeid, hits in bundlehits.items():
2ddd8785f8d7 pullbundle: add data about the cached bundle size and hit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4142
diff changeset
514 length = rlen(rangeid)
2ddd8785f8d7 pullbundle: add data about the cached bundle size and hit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4142
diff changeset
515 cachedsizes.append(length)
2ddd8785f8d7 pullbundle: add data about the cached bundle size and hit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4142
diff changeset
516 cachedhits.append(hits)
2ddd8785f8d7 pullbundle: add data about the cached bundle size and hit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4142
diff changeset
517
4142
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
518 sizesdist = distribution(sizes)
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
519 repo.ui.write(fmtdist('pull size', sizesdist))
4143
2ddd8785f8d7 pullbundle: add data about the cached bundle size and hit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4142
diff changeset
520
4142
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
521 changesmissingdist = distribution(changesmissing)
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
522 repo.ui.write(fmtdist('non-cached changesets', changesmissingdist))
4143
2ddd8785f8d7 pullbundle: add data about the cached bundle size and hit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4142
diff changeset
523
4142
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
524 changesratiodist = distribution(changesratio)
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
525 repo.ui.write(fmtdist('ratio of cached changesets', changesratiodist))
4143
2ddd8785f8d7 pullbundle: add data about the cached bundle size and hit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4142
diff changeset
526
4142
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
527 bundlecountdist = distribution(bundlecount)
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
528 repo.ui.write(fmtdist('bundle count', bundlecountdist))
4143
2ddd8785f8d7 pullbundle: add data about the cached bundle size and hit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4142
diff changeset
529
4142
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
530 rangesratiodist = distribution(rangesratio)
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
531 repo.ui.write(fmtdist('ratio of cached bundles', rangesratiodist))
4143
2ddd8785f8d7 pullbundle: add data about the cached bundle size and hit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4142
diff changeset
532
4142
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
533 repo.ui.write('changesets served:\n')
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
534 repo.ui.write(' total: %7d\n' % totalchanges)
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
535 repo.ui.write(' from cache: %7d (%2d%%)\n'
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
536 % (totalcached, (totalcached * 100 // totalchanges)))
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
537 repo.ui.write(' bundle: %7d\n' % sum(bundlecount))
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
538
4143
2ddd8785f8d7 pullbundle: add data about the cached bundle size and hit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4142
diff changeset
539 cachedsizesdist = distribution(cachedsizes)
2ddd8785f8d7 pullbundle: add data about the cached bundle size and hit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4142
diff changeset
540 repo.ui.write(fmtdist('size of cached bundles', cachedsizesdist))
2ddd8785f8d7 pullbundle: add data about the cached bundle size and hit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4142
diff changeset
541
2ddd8785f8d7 pullbundle: add data about the cached bundle size and hit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4142
diff changeset
542 cachedhitsdist = distribution(cachedhits)
2ddd8785f8d7 pullbundle: add data about the cached bundle size and hit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4142
diff changeset
543 repo.ui.write(fmtdist('hit on cached bundles', cachedhitsdist))
2ddd8785f8d7 pullbundle: add data about the cached bundle size and hit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4142
diff changeset
544
4142
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
545 def takeonesample(repo, revs):
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
546 node = repo.changelog.node
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
547 pulled = random.sample(revs, max(4, len(revs) // 1000))
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
548 pulled = repo.revs('%ld::%ld', pulled, pulled)
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
549 nodes = [node(r) for r in pulled]
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
550 return outgoingfromnodes(repo, nodes)
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
551
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
552 def distribution(data):
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
553 data.sort()
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
554 length = len(data)
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
555 return {
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
556 'min': data[0],
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
557 '10%': data[length // 10],
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
558 '25%': data[length // 4],
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
559 '50%': data[length // 2],
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
560 '75%': data[(length // 4) * 3],
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
561 '90%': data[(length // 10) * 9],
4143
2ddd8785f8d7 pullbundle: add data about the cached bundle size and hit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4142
diff changeset
562 '95%': data[(length // 20) * 19],
4142
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
563 'max': data[-1],
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
564 }
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
565
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
566 STATSFORMAT = """{name}:
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
567 min: {min}
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
568 10%: {10%}
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
569 25%: {25%}
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
570 50%: {50%}
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
571 75%: {75%}
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
572 90%: {90%}
4143
2ddd8785f8d7 pullbundle: add data about the cached bundle size and hit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4142
diff changeset
573 95%: {95%}
4142
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
574 max: {max}
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
575 """
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
576
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
577 def fmtdist(name, data):
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
578 return STATSFORMAT.format(name=name, **data)