Mercurial > public > mercurial-scm > hg

annotate mercurial/metadata.py @ 44941:edd08aa193fb

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
files: extract code for extra filtering of the `removed` entry into copies We want to reduce the set of `removed` files that to the set of files actually removed. That `removed` set is used as of the changeset centric algorithm, having smaller sets means less processing and faster computation. In this changeset we extract the code to be a function of it own. We will make use of it in the next changesets. Differential Revision: https://phab.mercurial-scm.org/D8588
author Pierre-Yves David <pierre-yves.david@octobus.net>
date Wed, 27 May 2020 12:45:39 +0200
parents 4c1d39215034
children 25512a65cefd
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
44940
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
1 # metadata.py -- code related to various metadata computation and access.
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
2 #
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
3 # Copyright 2019 Google, Inc <martinvonz@google.com>
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
4 # Copyright 2020 Pierre-Yves David <pierre-yves.david@octobus.net>
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
5 #
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
6 # This software may be used and distributed according to the terms of the
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
7 # GNU General Public License version 2 or any later version.
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
8 from __future__ import absolute_import, print_function
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
9
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
10 import multiprocessing
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
11
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
12 from . import (
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
13 error,
44941
edd08aa193fb files: extract code for extra filtering of the `removed` entry into copies
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44940
diff changeset
14 node,
44940
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
15 pycompat,
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
16 util,
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
17 )
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
18
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
19 from .revlogutils import (
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
20 flagutil as sidedataflag,
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
21 sidedata as sidedatamod,
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
22 )
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
23
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
24
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
25 def computechangesetfilesadded(ctx):
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
26 """return the list of files added in a changeset
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
27 """
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
28 added = []
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
29 for f in ctx.files():
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
30 if not any(f in p for p in ctx.parents()):
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
31 added.append(f)
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
32 return added
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
33
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
34
44941
edd08aa193fb files: extract code for extra filtering of the `removed` entry into copies
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44940
diff changeset
35 def get_removal_filter(ctx, x=None):
edd08aa193fb files: extract code for extra filtering of the `removed` entry into copies
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44940
diff changeset
36 """return a function to detect files "wrongly" detected as `removed`
edd08aa193fb files: extract code for extra filtering of the `removed` entry into copies
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44940
diff changeset
37
edd08aa193fb files: extract code for extra filtering of the `removed` entry into copies
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44940
diff changeset
38 When a file is removed relative to p1 in a merge, this
edd08aa193fb files: extract code for extra filtering of the `removed` entry into copies
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44940
diff changeset
39 function determines whether the absence is due to a
edd08aa193fb files: extract code for extra filtering of the `removed` entry into copies
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44940
diff changeset
40 deletion from a parent, or whether the merge commit
edd08aa193fb files: extract code for extra filtering of the `removed` entry into copies
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44940
diff changeset
41 itself deletes the file. We decide this by doing a
edd08aa193fb files: extract code for extra filtering of the `removed` entry into copies
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44940
diff changeset
42 simplified three way merge of the manifest entry for
edd08aa193fb files: extract code for extra filtering of the `removed` entry into copies
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44940
diff changeset
43 the file. There are two ways we decide the merge
edd08aa193fb files: extract code for extra filtering of the `removed` entry into copies
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44940
diff changeset
44 itself didn't delete a file:
edd08aa193fb files: extract code for extra filtering of the `removed` entry into copies
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44940
diff changeset
45 - neither parent (nor the merge) contain the file
edd08aa193fb files: extract code for extra filtering of the `removed` entry into copies
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44940
diff changeset
46 - exactly one parent contains the file, and that
edd08aa193fb files: extract code for extra filtering of the `removed` entry into copies
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44940
diff changeset
47 parent has the same filelog entry as the merge
edd08aa193fb files: extract code for extra filtering of the `removed` entry into copies
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44940
diff changeset
48 ancestor (or all of them if there two). In other
edd08aa193fb files: extract code for extra filtering of the `removed` entry into copies
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44940
diff changeset
49 words, that parent left the file unchanged while the
edd08aa193fb files: extract code for extra filtering of the `removed` entry into copies
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44940
diff changeset
50 other one deleted it.
edd08aa193fb files: extract code for extra filtering of the `removed` entry into copies
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44940
diff changeset
51 One way to think about this is that deleting a file is
edd08aa193fb files: extract code for extra filtering of the `removed` entry into copies
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44940
diff changeset
52 similar to emptying it, so the list of changed files
edd08aa193fb files: extract code for extra filtering of the `removed` entry into copies
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44940
diff changeset
53 should be similar either way. The computation
edd08aa193fb files: extract code for extra filtering of the `removed` entry into copies
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44940
diff changeset
54 described above is not done directly in _filecommit
edd08aa193fb files: extract code for extra filtering of the `removed` entry into copies
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44940
diff changeset
55 when creating the list of changed files, however
edd08aa193fb files: extract code for extra filtering of the `removed` entry into copies
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44940
diff changeset
56 it does something very similar by comparing filelog
edd08aa193fb files: extract code for extra filtering of the `removed` entry into copies
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44940
diff changeset
57 nodes.
edd08aa193fb files: extract code for extra filtering of the `removed` entry into copies
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44940
diff changeset
58 """
edd08aa193fb files: extract code for extra filtering of the `removed` entry into copies
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44940
diff changeset
59
edd08aa193fb files: extract code for extra filtering of the `removed` entry into copies
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44940
diff changeset
60 if x is not None:
edd08aa193fb files: extract code for extra filtering of the `removed` entry into copies
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44940
diff changeset
61 p1, p2, m1, m2 = x
edd08aa193fb files: extract code for extra filtering of the `removed` entry into copies
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44940
diff changeset
62 else:
edd08aa193fb files: extract code for extra filtering of the `removed` entry into copies
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44940
diff changeset
63 p1 = ctx.p1()
edd08aa193fb files: extract code for extra filtering of the `removed` entry into copies
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44940
diff changeset
64 p2 = ctx.p2()
edd08aa193fb files: extract code for extra filtering of the `removed` entry into copies
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44940
diff changeset
65 m1 = p1.manifest()
edd08aa193fb files: extract code for extra filtering of the `removed` entry into copies
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44940
diff changeset
66 m2 = p2.manifest()
edd08aa193fb files: extract code for extra filtering of the `removed` entry into copies
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44940
diff changeset
67
edd08aa193fb files: extract code for extra filtering of the `removed` entry into copies
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44940
diff changeset
68 @util.cachefunc
edd08aa193fb files: extract code for extra filtering of the `removed` entry into copies
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44940
diff changeset
69 def mas():
edd08aa193fb files: extract code for extra filtering of the `removed` entry into copies
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44940
diff changeset
70 p1n = p1.node()
edd08aa193fb files: extract code for extra filtering of the `removed` entry into copies
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44940
diff changeset
71 p2n = p2.node()
edd08aa193fb files: extract code for extra filtering of the `removed` entry into copies
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44940
diff changeset
72 cahs = ctx.repo().changelog.commonancestorsheads(p1n, p2n)
edd08aa193fb files: extract code for extra filtering of the `removed` entry into copies
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44940
diff changeset
73 if not cahs:
edd08aa193fb files: extract code for extra filtering of the `removed` entry into copies
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44940
diff changeset
74 cahs = [node.nullrev]
edd08aa193fb files: extract code for extra filtering of the `removed` entry into copies
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44940
diff changeset
75 return [ctx.repo()[r].manifest() for r in cahs]
edd08aa193fb files: extract code for extra filtering of the `removed` entry into copies
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44940
diff changeset
76
edd08aa193fb files: extract code for extra filtering of the `removed` entry into copies
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44940
diff changeset
77 def deletionfromparent(f):
edd08aa193fb files: extract code for extra filtering of the `removed` entry into copies
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44940
diff changeset
78 if f in m1:
edd08aa193fb files: extract code for extra filtering of the `removed` entry into copies
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44940
diff changeset
79 return f not in m2 and all(
edd08aa193fb files: extract code for extra filtering of the `removed` entry into copies
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44940
diff changeset
80 f in ma and ma.find(f) == m1.find(f) for ma in mas()
edd08aa193fb files: extract code for extra filtering of the `removed` entry into copies
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44940
diff changeset
81 )
edd08aa193fb files: extract code for extra filtering of the `removed` entry into copies
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44940
diff changeset
82 elif f in m2:
edd08aa193fb files: extract code for extra filtering of the `removed` entry into copies
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44940
diff changeset
83 return all(f in ma and ma.find(f) == m2.find(f) for ma in mas())
edd08aa193fb files: extract code for extra filtering of the `removed` entry into copies
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44940
diff changeset
84 else:
edd08aa193fb files: extract code for extra filtering of the `removed` entry into copies
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44940
diff changeset
85 return True
edd08aa193fb files: extract code for extra filtering of the `removed` entry into copies
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44940
diff changeset
86
edd08aa193fb files: extract code for extra filtering of the `removed` entry into copies
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44940
diff changeset
87 return deletionfromparent
edd08aa193fb files: extract code for extra filtering of the `removed` entry into copies
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44940
diff changeset
88
edd08aa193fb files: extract code for extra filtering of the `removed` entry into copies
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44940
diff changeset
89
44940
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
90 def computechangesetfilesremoved(ctx):
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
91 """return the list of files removed in a changeset
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
92 """
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
93 removed = []
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
94 for f in ctx.files():
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
95 if f not in ctx:
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
96 removed.append(f)
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
97 return removed
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
98
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
99
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
100 def computechangesetcopies(ctx):
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
101 """return the copies data for a changeset
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
102
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
103 The copies data are returned as a pair of dictionnary (p1copies, p2copies).
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
104
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
105 Each dictionnary are in the form: `{newname: oldname}`
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
106 """
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
107 p1copies = {}
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
108 p2copies = {}
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
109 p1 = ctx.p1()
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
110 p2 = ctx.p2()
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
111 narrowmatch = ctx._repo.narrowmatch()
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
112 for dst in ctx.files():
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
113 if not narrowmatch(dst) or dst not in ctx:
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
114 continue
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
115 copied = ctx[dst].renamed()
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
116 if not copied:
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
117 continue
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
118 src, srcnode = copied
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
119 if src in p1 and p1[src].filenode() == srcnode:
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
120 p1copies[dst] = src
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
121 elif src in p2 and p2[src].filenode() == srcnode:
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
122 p2copies[dst] = src
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
123 return p1copies, p2copies
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
124
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
125
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
126 def encodecopies(files, copies):
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
127 items = []
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
128 for i, dst in enumerate(files):
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
129 if dst in copies:
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
130 items.append(b'%d\0%s' % (i, copies[dst]))
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
131 if len(items) != len(copies):
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
132 raise error.ProgrammingError(
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
133 b'some copy targets missing from file list'
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
134 )
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
135 return b"\n".join(items)
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
136
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
137
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
138 def decodecopies(files, data):
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
139 try:
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
140 copies = {}
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
141 if not data:
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
142 return copies
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
143 for l in data.split(b'\n'):
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
144 strindex, src = l.split(b'\0')
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
145 i = int(strindex)
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
146 dst = files[i]
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
147 copies[dst] = src
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
148 return copies
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
149 except (ValueError, IndexError):
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
150 # Perhaps someone had chosen the same key name (e.g. "p1copies") and
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
151 # used different syntax for the value.
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
152 return None
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
153
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
154
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
155 def encodefileindices(files, subset):
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
156 subset = set(subset)
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
157 indices = []
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
158 for i, f in enumerate(files):
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
159 if f in subset:
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
160 indices.append(b'%d' % i)
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
161 return b'\n'.join(indices)
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
162
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
163
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
164 def decodefileindices(files, data):
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
165 try:
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
166 subset = []
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
167 if not data:
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
168 return subset
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
169 for strindex in data.split(b'\n'):
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
170 i = int(strindex)
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
171 if i < 0 or i >= len(files):
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
172 return None
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
173 subset.append(files[i])
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
174 return subset
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
175 except (ValueError, IndexError):
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
176 # Perhaps someone had chosen the same key name (e.g. "added") and
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
177 # used different syntax for the value.
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
178 return None
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
179
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
180
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
181 def _getsidedata(srcrepo, rev):
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
182 ctx = srcrepo[rev]
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
183 filescopies = computechangesetcopies(ctx)
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
184 filesadded = computechangesetfilesadded(ctx)
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
185 filesremoved = computechangesetfilesremoved(ctx)
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
186 sidedata = {}
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
187 if any([filescopies, filesadded, filesremoved]):
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
188 sortedfiles = sorted(ctx.files())
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
189 p1copies, p2copies = filescopies
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
190 p1copies = encodecopies(sortedfiles, p1copies)
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
191 p2copies = encodecopies(sortedfiles, p2copies)
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
192 filesadded = encodefileindices(sortedfiles, filesadded)
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
193 filesremoved = encodefileindices(sortedfiles, filesremoved)
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
194 if p1copies:
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
195 sidedata[sidedatamod.SD_P1COPIES] = p1copies
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
196 if p2copies:
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
197 sidedata[sidedatamod.SD_P2COPIES] = p2copies
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
198 if filesadded:
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
199 sidedata[sidedatamod.SD_FILESADDED] = filesadded
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
200 if filesremoved:
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
201 sidedata[sidedatamod.SD_FILESREMOVED] = filesremoved
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
202 return sidedata
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
203
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
204
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
205 def getsidedataadder(srcrepo, destrepo):
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
206 use_w = srcrepo.ui.configbool(b'experimental', b'worker.repository-upgrade')
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
207 if pycompat.iswindows or not use_w:
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
208 return _get_simple_sidedata_adder(srcrepo, destrepo)
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
209 else:
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
210 return _get_worker_sidedata_adder(srcrepo, destrepo)
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
211
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
212
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
213 def _sidedata_worker(srcrepo, revs_queue, sidedata_queue, tokens):
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
214 """The function used by worker precomputing sidedata
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
215
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
216 It read an input queue containing revision numbers
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
217 It write in an output queue containing (rev, <sidedata-map>)
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
218
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
219 The `None` input value is used as a stop signal.
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
220
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
221 The `tokens` semaphore is user to avoid having too many unprocessed
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
222 entries. The workers needs to acquire one token before fetching a task.
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
223 They will be released by the consumer of the produced data.
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
224 """
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
225 tokens.acquire()
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
226 rev = revs_queue.get()
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
227 while rev is not None:
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
228 data = _getsidedata(srcrepo, rev)
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
229 sidedata_queue.put((rev, data))
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
230 tokens.acquire()
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
231 rev = revs_queue.get()
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
232 # processing of `None` is completed, release the token.
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
233 tokens.release()
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
234
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
235
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
236 BUFF_PER_WORKER = 50
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
237
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
238
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
239 def _get_worker_sidedata_adder(srcrepo, destrepo):
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
240 """The parallel version of the sidedata computation
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
241
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
242 This code spawn a pool of worker that precompute a buffer of sidedata
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
243 before we actually need them"""
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
244 # avoid circular import copies -> scmutil -> worker -> copies
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
245 from . import worker
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
246
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
247 nbworkers = worker._numworkers(srcrepo.ui)
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
248
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
249 tokens = multiprocessing.BoundedSemaphore(nbworkers * BUFF_PER_WORKER)
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
250 revsq = multiprocessing.Queue()
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
251 sidedataq = multiprocessing.Queue()
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
252
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
253 assert srcrepo.filtername is None
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
254 # queue all tasks beforehand, revision numbers are small and it make
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
255 # synchronisation simpler
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
256 #
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
257 # Since the computation for each node can be quite expensive, the overhead
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
258 # of using a single queue is not revelant. In practice, most computation
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
259 # are fast but some are very expensive and dominate all the other smaller
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
260 # cost.
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
261 for r in srcrepo.changelog.revs():
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
262 revsq.put(r)
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
263 # queue the "no more tasks" markers
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
264 for i in range(nbworkers):
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
265 revsq.put(None)
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
266
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
267 allworkers = []
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
268 for i in range(nbworkers):
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
269 args = (srcrepo, revsq, sidedataq, tokens)
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
270 w = multiprocessing.Process(target=_sidedata_worker, args=args)
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
271 allworkers.append(w)
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
272 w.start()
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
273
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
274 # dictionnary to store results for revision higher than we one we are
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
275 # looking for. For example, if we need the sidedatamap for 42, and 43 is
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
276 # received, when shelve 43 for later use.
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
277 staging = {}
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
278
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
279 def sidedata_companion(revlog, rev):
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
280 sidedata = {}
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
281 if util.safehasattr(revlog, b'filteredrevs'): # this is a changelog
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
282 # Is the data previously shelved ?
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
283 sidedata = staging.pop(rev, None)
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
284 if sidedata is None:
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
285 # look at the queued result until we find the one we are lookig
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
286 # for (shelve the other ones)
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
287 r, sidedata = sidedataq.get()
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
288 while r != rev:
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
289 staging[r] = sidedata
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
290 r, sidedata = sidedataq.get()
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
291 tokens.release()
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
292 return False, (), sidedata
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
293
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
294 return sidedata_companion
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
295
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
296
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
297 def _get_simple_sidedata_adder(srcrepo, destrepo):
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
298 """The simple version of the sidedata computation
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
299
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
300 It just compute it in the same thread on request"""
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
301
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
302 def sidedatacompanion(revlog, rev):
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
303 sidedata = {}
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
304 if util.safehasattr(revlog, 'filteredrevs'): # this is a changelog
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
305 sidedata = _getsidedata(srcrepo, rev)
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
306 return False, (), sidedata
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
307
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
308 return sidedatacompanion
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
309
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
310
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
311 def getsidedataremover(srcrepo, destrepo):
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
312 def sidedatacompanion(revlog, rev):
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
313 f = ()
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
314 if util.safehasattr(revlog, 'filteredrevs'): # this is a changelog
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
315 if revlog.flags(rev) & sidedataflag.REVIDX_SIDEDATA:
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
316 f = (
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
317 sidedatamod.SD_P1COPIES,
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
318 sidedatamod.SD_P2COPIES,
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
319 sidedatamod.SD_FILESADDED,
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
320 sidedatamod.SD_FILESREMOVED,
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
321 )
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
322 return False, f, {}
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
323
4c1d39215034 metadata: move computation related to files touched in a dedicated module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
324 return sidedatacompanion