Mercurial > public > mercurial-scm > hg

annotate mercurial/streamclone.py @ 5909:f45f7390c1c5

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
strip: calculate list of extra nodes to save and pass it to changegroupsubset When we remove revision N from the repository, all revisions >= N are affected: either it's a descendant from N and will also be removed, or it's not a descendant of N and will be renumbered. As a consequence, we have to (at least temporarily) remove all filelog and manifest revisions that have a linkrev >= N, readding some of them later. Unfortunately, it's possible to have a revlog with two revisions r1 and r2 such that r1 < r2, but linkrev(r1) > linkrev(r2). If we try to strip revision linkrev(r1) from the repository, we'll also lose revision r2 when we truncate this revlog. We already use changegroupsubset to create a temporary changegroup containing the revisions that have to be restored, but that function is unable to detect that we also wanted to save the r2 in the case above. So we manually calculate these extra nodes and pass it to changegroupsubset. This should fix issue764.
author Alexis S. L. Carvalho <alexis@cecm.usp.br>
date Sat, 19 Jan 2008 18:01:16 -0200
parents a58d415b272e
children e75aab656f46
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
2612
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
1 # streamclone.py - streaming clone server support for mercurial
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
2 #
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
3 # Copyright 2006 Vadim Gelfer <vadim.gelfer@gmail.com>
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
4 #
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
5 # This software may be used and distributed according to the terms
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
6 # of the GNU General Public License, incorporated herein by reference.
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
7
3891
6b4127c7d52a Simplify i18n imports
Matt Mackall <mpm@selenic.com>
parents: 3877
diff changeset
8 from i18n import _
5396
5105b119edd2 Add osutil module, containing a listdir function.
Bryan O'Sullivan <bos@serpentine.com>
parents: 4915
diff changeset
9 import os, osutil, stat, util, lock
2612
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
10
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
11 # if server supports streaming clone, it advertises "stream"
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
12 # capability with value that is version+flags of repo it is serving.
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
13 # client only streams if it can read that repo format.
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
14
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
15 def walkrepo(root):
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
16 '''iterate over metadata files in repository.
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
17 walk in natural (sorted) order.
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
18 yields 2-tuples: name of .d or .i file, size of file.'''
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
19
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
20 strip_count = len(root) + len(os.sep)
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
21 def walk(path, recurse):
5396
5105b119edd2 Add osutil module, containing a listdir function.
Bryan O'Sullivan <bos@serpentine.com>
parents: 4915
diff changeset
22 for e, kind, st in osutil.listdir(path, stat=True):
2612
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
23 pe = os.path.join(path, e)
5396
5105b119edd2 Add osutil module, containing a listdir function.
Bryan O'Sullivan <bos@serpentine.com>
parents: 4915
diff changeset
24 if kind == stat.S_IFDIR:
2612
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
25 if recurse:
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
26 for x in walk(pe, True):
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
27 yield x
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
28 else:
5396
5105b119edd2 Add osutil module, containing a listdir function.
Bryan O'Sullivan <bos@serpentine.com>
parents: 4915
diff changeset
29 if kind != stat.S_IFREG or len(e) < 2:
2612
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
30 continue
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
31 sfx = e[-2:]
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
32 if sfx in ('.d', '.i'):
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
33 yield pe[strip_count:], st.st_size
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
34 # write file data first
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
35 for x in walk(os.path.join(root, 'data'), True):
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
36 yield x
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
37 # write manifest before changelog
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
38 meta = list(walk(root, False))
2623
d1cbfe9e13cd fix problem with uncompressed clone and python 2.3.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2622
diff changeset
39 meta.sort()
d1cbfe9e13cd fix problem with uncompressed clone and python 2.3.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2622
diff changeset
40 meta.reverse()
2612
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
41 for x in meta:
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
42 yield x
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
43
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
44 # stream file format is simple.
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
45 #
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
46 # server writes out line that says how many files, how many total
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
47 # bytes. separator is ascii space, byte counts are strings.
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
48 #
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
49 # then for each file:
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
50 #
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
51 # server writes out line that says file name, how many bytes in
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
52 # file. separator is ascii nul, byte count is string.
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
53 #
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
54 # server writes out raw file data.
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
55
4834
439e2f2fde42 Fix inconsistency for the stream_out capability in hgweb
Edouard Gomez <ed.gomez@free.fr>
parents: 4134
diff changeset
56 def stream_out(repo, fileobj, untrusted=False):
2612
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
57 '''stream out all metadata files in repository.
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
58 writes to file-like object, must support write() and optional flush().'''
2621
5a5852a417b1 clone: disable stream support on server side by default.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2612
diff changeset
59
4834
439e2f2fde42 Fix inconsistency for the stream_out capability in hgweb
Edouard Gomez <ed.gomez@free.fr>
parents: 4134
diff changeset
60 if not repo.ui.configbool('server', 'uncompressed', untrusted=untrusted):
2621
5a5852a417b1 clone: disable stream support on server side by default.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2612
diff changeset
61 fileobj.write('1\n')
5a5852a417b1 clone: disable stream support on server side by default.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2612
diff changeset
62 return
5a5852a417b1 clone: disable stream support on server side by default.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2612
diff changeset
63
2612
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
64 # get consistent snapshot of repo. lock during scan so lock not
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
65 # needed while we stream, and commits can happen.
5456
a58d415b272e fix typo
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 5396
diff changeset
66 repolock = None
3687
d5dd0a2a44bc Handle locking exceptions if streaming clone can't lock the repo. (Issue324)
Thomas Arendsen Hein <thomas@intevation.de>
parents: 2623
diff changeset
67 try:
4915
97b734fb9c6f Use try/finally pattern to cleanup locks and transactions
Matt Mackall <mpm@selenic.com>
parents: 4834
diff changeset
68 try:
97b734fb9c6f Use try/finally pattern to cleanup locks and transactions
Matt Mackall <mpm@selenic.com>
parents: 4834
diff changeset
69 repolock = repo.lock()
97b734fb9c6f Use try/finally pattern to cleanup locks and transactions
Matt Mackall <mpm@selenic.com>
parents: 4834
diff changeset
70 except (lock.LockHeld, lock.LockUnavailable), inst:
97b734fb9c6f Use try/finally pattern to cleanup locks and transactions
Matt Mackall <mpm@selenic.com>
parents: 4834
diff changeset
71 repo.ui.warn('locking the repository failed: %s\n' % (inst,))
97b734fb9c6f Use try/finally pattern to cleanup locks and transactions
Matt Mackall <mpm@selenic.com>
parents: 4834
diff changeset
72 fileobj.write('2\n')
97b734fb9c6f Use try/finally pattern to cleanup locks and transactions
Matt Mackall <mpm@selenic.com>
parents: 4834
diff changeset
73 return
3687
d5dd0a2a44bc Handle locking exceptions if streaming clone can't lock the repo. (Issue324)
Thomas Arendsen Hein <thomas@intevation.de>
parents: 2623
diff changeset
74
4915
97b734fb9c6f Use try/finally pattern to cleanup locks and transactions
Matt Mackall <mpm@selenic.com>
parents: 4834
diff changeset
75 fileobj.write('0\n')
97b734fb9c6f Use try/finally pattern to cleanup locks and transactions
Matt Mackall <mpm@selenic.com>
parents: 4834
diff changeset
76 repo.ui.debug('scanning\n')
97b734fb9c6f Use try/finally pattern to cleanup locks and transactions
Matt Mackall <mpm@selenic.com>
parents: 4834
diff changeset
77 entries = []
97b734fb9c6f Use try/finally pattern to cleanup locks and transactions
Matt Mackall <mpm@selenic.com>
parents: 4834
diff changeset
78 total_bytes = 0
97b734fb9c6f Use try/finally pattern to cleanup locks and transactions
Matt Mackall <mpm@selenic.com>
parents: 4834
diff changeset
79 for name, size in walkrepo(repo.spath):
97b734fb9c6f Use try/finally pattern to cleanup locks and transactions
Matt Mackall <mpm@selenic.com>
parents: 4834
diff changeset
80 name = repo.decodefn(util.pconvert(name))
97b734fb9c6f Use try/finally pattern to cleanup locks and transactions
Matt Mackall <mpm@selenic.com>
parents: 4834
diff changeset
81 entries.append((name, size))
97b734fb9c6f Use try/finally pattern to cleanup locks and transactions
Matt Mackall <mpm@selenic.com>
parents: 4834
diff changeset
82 total_bytes += size
97b734fb9c6f Use try/finally pattern to cleanup locks and transactions
Matt Mackall <mpm@selenic.com>
parents: 4834
diff changeset
83 finally:
97b734fb9c6f Use try/finally pattern to cleanup locks and transactions
Matt Mackall <mpm@selenic.com>
parents: 4834
diff changeset
84 del repolock
2612
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
85
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
86 repo.ui.debug('%d files, %d bytes to transfer\n' %
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
87 (len(entries), total_bytes))
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
88 fileobj.write('%d %d\n' % (len(entries), total_bytes))
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
89 for name, size in entries:
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
90 repo.ui.debug('sending %s (%d bytes)\n' % (name, size))
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
91 fileobj.write('%s\0%d\n' % (name, size))
3791
8643b9f90b51 introduce localrepo.spath for the store path, sopener fixes
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 3721
diff changeset
92 for chunk in util.filechunkiter(repo.sopener(name), limit=size):
2612
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
93 fileobj.write(chunk)
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
94 flush = getattr(fileobj, 'flush', None)
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
95 if flush: flush()