Mercurial > public > mercurial-scm > hg

annotate rust/hg-cpython/src/copy_tracing.rs @ 46589:620c88fb42a2

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
copies-rust: introduce PyBytesWithData to reduce GIL requirement See explanations in new doc-comments. Differential Revision: https://phab.mercurial-scm.org/D9685
author Simon Sapin <simon-commits@exyr.org>
date Thu, 26 Nov 2020 18:23:51 +0100
parents 47557ea79fc7
children 8d20abed6a1e
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
45945
50c5ee3bdf9a copies: introduce the hg-cpython wrapper for `combine_changeset_copies`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
1 use cpython::ObjectProtocol;
50c5ee3bdf9a copies: introduce the hg-cpython wrapper for `combine_changeset_copies`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
2 use cpython::PyBytes;
50c5ee3bdf9a copies: introduce the hg-cpython wrapper for `combine_changeset_copies`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
3 use cpython::PyDict;
50c5ee3bdf9a copies: introduce the hg-cpython wrapper for `combine_changeset_copies`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
4 use cpython::PyList;
50c5ee3bdf9a copies: introduce the hg-cpython wrapper for `combine_changeset_copies`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
5 use cpython::PyModule;
50c5ee3bdf9a copies: introduce the hg-cpython wrapper for `combine_changeset_copies`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
6 use cpython::PyObject;
50c5ee3bdf9a copies: introduce the hg-cpython wrapper for `combine_changeset_copies`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
7 use cpython::PyResult;
50c5ee3bdf9a copies: introduce the hg-cpython wrapper for `combine_changeset_copies`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
8 use cpython::PyTuple;
50c5ee3bdf9a copies: introduce the hg-cpython wrapper for `combine_changeset_copies`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
9 use cpython::Python;
50c5ee3bdf9a copies: introduce the hg-cpython wrapper for `combine_changeset_copies`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
10
50c5ee3bdf9a copies: introduce the hg-cpython wrapper for `combine_changeset_copies`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
11 use hg::copy_tracing::ChangedFiles;
46587
cb4b0b0c6de4 copies-rust: split up combine_changeset_copies function into a struct
Simon Sapin <simon.sapin@octobus.net>
parents: 46569
diff changeset
12 use hg::copy_tracing::CombineChangesetCopies;
45945
50c5ee3bdf9a copies: introduce the hg-cpython wrapper for `combine_changeset_copies`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
13 use hg::Revision;
50c5ee3bdf9a copies: introduce the hg-cpython wrapper for `combine_changeset_copies`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
14
46589
620c88fb42a2 copies-rust: introduce PyBytesWithData to reduce GIL requirement
Simon Sapin <simon-commits@exyr.org>
parents: 46588
diff changeset
15 use self::pybytes_with_data::PyBytesWithData;
620c88fb42a2 copies-rust: introduce PyBytesWithData to reduce GIL requirement
Simon Sapin <simon-commits@exyr.org>
parents: 46588
diff changeset
16
620c88fb42a2 copies-rust: introduce PyBytesWithData to reduce GIL requirement
Simon Sapin <simon-commits@exyr.org>
parents: 46588
diff changeset
17 // Module to encapsulate private fields
620c88fb42a2 copies-rust: introduce PyBytesWithData to reduce GIL requirement
Simon Sapin <simon-commits@exyr.org>
parents: 46588
diff changeset
18 mod pybytes_with_data {
620c88fb42a2 copies-rust: introduce PyBytesWithData to reduce GIL requirement
Simon Sapin <simon-commits@exyr.org>
parents: 46588
diff changeset
19 use cpython::{PyBytes, Python};
620c88fb42a2 copies-rust: introduce PyBytesWithData to reduce GIL requirement
Simon Sapin <simon-commits@exyr.org>
parents: 46588
diff changeset
20
620c88fb42a2 copies-rust: introduce PyBytesWithData to reduce GIL requirement
Simon Sapin <simon-commits@exyr.org>
parents: 46588
diff changeset
21 /// Safe abstraction over a `PyBytes` together with the `&[u8]` slice
620c88fb42a2 copies-rust: introduce PyBytesWithData to reduce GIL requirement
Simon Sapin <simon-commits@exyr.org>
parents: 46588
diff changeset
22 /// that borrows it.
620c88fb42a2 copies-rust: introduce PyBytesWithData to reduce GIL requirement
Simon Sapin <simon-commits@exyr.org>
parents: 46588
diff changeset
23 ///
620c88fb42a2 copies-rust: introduce PyBytesWithData to reduce GIL requirement
Simon Sapin <simon-commits@exyr.org>
parents: 46588
diff changeset
24 /// Calling `PyBytes::data` requires a GIL marker but we want to access the
620c88fb42a2 copies-rust: introduce PyBytesWithData to reduce GIL requirement
Simon Sapin <simon-commits@exyr.org>
parents: 46588
diff changeset
25 /// data in a thread that (ideally) does not need to acquire the GIL.
620c88fb42a2 copies-rust: introduce PyBytesWithData to reduce GIL requirement
Simon Sapin <simon-commits@exyr.org>
parents: 46588
diff changeset
26 /// This type allows separating the call an the use.
620c88fb42a2 copies-rust: introduce PyBytesWithData to reduce GIL requirement
Simon Sapin <simon-commits@exyr.org>
parents: 46588
diff changeset
27 pub(super) struct PyBytesWithData {
620c88fb42a2 copies-rust: introduce PyBytesWithData to reduce GIL requirement
Simon Sapin <simon-commits@exyr.org>
parents: 46588
diff changeset
28 #[allow(unused)]
620c88fb42a2 copies-rust: introduce PyBytesWithData to reduce GIL requirement
Simon Sapin <simon-commits@exyr.org>
parents: 46588
diff changeset
29 keep_alive: PyBytes,
620c88fb42a2 copies-rust: introduce PyBytesWithData to reduce GIL requirement
Simon Sapin <simon-commits@exyr.org>
parents: 46588
diff changeset
30
620c88fb42a2 copies-rust: introduce PyBytesWithData to reduce GIL requirement
Simon Sapin <simon-commits@exyr.org>
parents: 46588
diff changeset
31 /// Borrows the buffer inside `self.keep_alive`,
620c88fb42a2 copies-rust: introduce PyBytesWithData to reduce GIL requirement
Simon Sapin <simon-commits@exyr.org>
parents: 46588
diff changeset
32 /// but the borrow-checker cannot express self-referential structs.
620c88fb42a2 copies-rust: introduce PyBytesWithData to reduce GIL requirement
Simon Sapin <simon-commits@exyr.org>
parents: 46588
diff changeset
33 data: *const [u8],
620c88fb42a2 copies-rust: introduce PyBytesWithData to reduce GIL requirement
Simon Sapin <simon-commits@exyr.org>
parents: 46588
diff changeset
34 }
620c88fb42a2 copies-rust: introduce PyBytesWithData to reduce GIL requirement
Simon Sapin <simon-commits@exyr.org>
parents: 46588
diff changeset
35
620c88fb42a2 copies-rust: introduce PyBytesWithData to reduce GIL requirement
Simon Sapin <simon-commits@exyr.org>
parents: 46588
diff changeset
36 fn require_send<T: Send>() {}
620c88fb42a2 copies-rust: introduce PyBytesWithData to reduce GIL requirement
Simon Sapin <simon-commits@exyr.org>
parents: 46588
diff changeset
37
620c88fb42a2 copies-rust: introduce PyBytesWithData to reduce GIL requirement
Simon Sapin <simon-commits@exyr.org>
parents: 46588
diff changeset
38 #[allow(unused)]
620c88fb42a2 copies-rust: introduce PyBytesWithData to reduce GIL requirement
Simon Sapin <simon-commits@exyr.org>
parents: 46588
diff changeset
39 fn static_assert_pybytes_is_send() {
620c88fb42a2 copies-rust: introduce PyBytesWithData to reduce GIL requirement
Simon Sapin <simon-commits@exyr.org>
parents: 46588
diff changeset
40 require_send::<PyBytes>;
620c88fb42a2 copies-rust: introduce PyBytesWithData to reduce GIL requirement
Simon Sapin <simon-commits@exyr.org>
parents: 46588
diff changeset
41 }
620c88fb42a2 copies-rust: introduce PyBytesWithData to reduce GIL requirement
Simon Sapin <simon-commits@exyr.org>
parents: 46588
diff changeset
42
620c88fb42a2 copies-rust: introduce PyBytesWithData to reduce GIL requirement
Simon Sapin <simon-commits@exyr.org>
parents: 46588
diff changeset
43 // Safety: PyBytes is Send. Raw pointers are not by default,
620c88fb42a2 copies-rust: introduce PyBytesWithData to reduce GIL requirement
Simon Sapin <simon-commits@exyr.org>
parents: 46588
diff changeset
44 // but here sending one to another thread is fine since we ensure it stays
620c88fb42a2 copies-rust: introduce PyBytesWithData to reduce GIL requirement
Simon Sapin <simon-commits@exyr.org>
parents: 46588
diff changeset
45 // valid.
620c88fb42a2 copies-rust: introduce PyBytesWithData to reduce GIL requirement
Simon Sapin <simon-commits@exyr.org>
parents: 46588
diff changeset
46 unsafe impl Send for PyBytesWithData {}
620c88fb42a2 copies-rust: introduce PyBytesWithData to reduce GIL requirement
Simon Sapin <simon-commits@exyr.org>
parents: 46588
diff changeset
47
620c88fb42a2 copies-rust: introduce PyBytesWithData to reduce GIL requirement
Simon Sapin <simon-commits@exyr.org>
parents: 46588
diff changeset
48 impl PyBytesWithData {
620c88fb42a2 copies-rust: introduce PyBytesWithData to reduce GIL requirement
Simon Sapin <simon-commits@exyr.org>
parents: 46588
diff changeset
49 pub fn new(py: Python, bytes: PyBytes) -> Self {
620c88fb42a2 copies-rust: introduce PyBytesWithData to reduce GIL requirement
Simon Sapin <simon-commits@exyr.org>
parents: 46588
diff changeset
50 Self {
620c88fb42a2 copies-rust: introduce PyBytesWithData to reduce GIL requirement
Simon Sapin <simon-commits@exyr.org>
parents: 46588
diff changeset
51 data: bytes.data(py),
620c88fb42a2 copies-rust: introduce PyBytesWithData to reduce GIL requirement
Simon Sapin <simon-commits@exyr.org>
parents: 46588
diff changeset
52 keep_alive: bytes,
620c88fb42a2 copies-rust: introduce PyBytesWithData to reduce GIL requirement
Simon Sapin <simon-commits@exyr.org>
parents: 46588
diff changeset
53 }
620c88fb42a2 copies-rust: introduce PyBytesWithData to reduce GIL requirement
Simon Sapin <simon-commits@exyr.org>
parents: 46588
diff changeset
54 }
620c88fb42a2 copies-rust: introduce PyBytesWithData to reduce GIL requirement
Simon Sapin <simon-commits@exyr.org>
parents: 46588
diff changeset
55
620c88fb42a2 copies-rust: introduce PyBytesWithData to reduce GIL requirement
Simon Sapin <simon-commits@exyr.org>
parents: 46588
diff changeset
56 pub fn data(&self) -> &[u8] {
620c88fb42a2 copies-rust: introduce PyBytesWithData to reduce GIL requirement
Simon Sapin <simon-commits@exyr.org>
parents: 46588
diff changeset
57 // Safety: the raw pointer is valid as long as the PyBytes is still
620c88fb42a2 copies-rust: introduce PyBytesWithData to reduce GIL requirement
Simon Sapin <simon-commits@exyr.org>
parents: 46588
diff changeset
58 // alive, and the returned slice borrows `self`.
620c88fb42a2 copies-rust: introduce PyBytesWithData to reduce GIL requirement
Simon Sapin <simon-commits@exyr.org>
parents: 46588
diff changeset
59 unsafe { &*self.data }
620c88fb42a2 copies-rust: introduce PyBytesWithData to reduce GIL requirement
Simon Sapin <simon-commits@exyr.org>
parents: 46588
diff changeset
60 }
620c88fb42a2 copies-rust: introduce PyBytesWithData to reduce GIL requirement
Simon Sapin <simon-commits@exyr.org>
parents: 46588
diff changeset
61 }
620c88fb42a2 copies-rust: introduce PyBytesWithData to reduce GIL requirement
Simon Sapin <simon-commits@exyr.org>
parents: 46588
diff changeset
62 }
620c88fb42a2 copies-rust: introduce PyBytesWithData to reduce GIL requirement
Simon Sapin <simon-commits@exyr.org>
parents: 46588
diff changeset
63
45945
50c5ee3bdf9a copies: introduce the hg-cpython wrapper for `combine_changeset_copies`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
64 /// Combines copies information contained into revision `revs` to build a copy
50c5ee3bdf9a copies: introduce the hg-cpython wrapper for `combine_changeset_copies`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
65 /// map.
50c5ee3bdf9a copies: introduce the hg-cpython wrapper for `combine_changeset_copies`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
66 ///
50c5ee3bdf9a copies: introduce the hg-cpython wrapper for `combine_changeset_copies`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
67 /// See mercurial/copies.py for details
50c5ee3bdf9a copies: introduce the hg-cpython wrapper for `combine_changeset_copies`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
68 pub fn combine_changeset_copies_wrapper(
50c5ee3bdf9a copies: introduce the hg-cpython wrapper for `combine_changeset_copies`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
69 py: Python,
50c5ee3bdf9a copies: introduce the hg-cpython wrapper for `combine_changeset_copies`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
70 revs: PyList,
46149
294d5aca4ff5 copies: iterate over children directly (instead of parents)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46057
diff changeset
71 children_count: PyDict,
45945
50c5ee3bdf9a copies: introduce the hg-cpython wrapper for `combine_changeset_copies`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
72 target_rev: Revision,
50c5ee3bdf9a copies: introduce the hg-cpython wrapper for `combine_changeset_copies`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
73 rev_info: PyObject,
46588
47557ea79fc7 copies-rust: move CPU-heavy Rust processing into a child thread
Simon Sapin <simon.sapin@octobus.net>
parents: 46587
diff changeset
74 multi_thread: bool,
45945
50c5ee3bdf9a copies: introduce the hg-cpython wrapper for `combine_changeset_copies`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
75 ) -> PyResult<PyDict> {
46587
cb4b0b0c6de4 copies-rust: split up combine_changeset_copies function into a struct
Simon Sapin <simon.sapin@octobus.net>
parents: 46569
diff changeset
76 let children_count = children_count
45945
50c5ee3bdf9a copies: introduce the hg-cpython wrapper for `combine_changeset_copies`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
77 .items(py)
50c5ee3bdf9a copies: introduce the hg-cpython wrapper for `combine_changeset_copies`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
78 .iter()
46149
294d5aca4ff5 copies: iterate over children directly (instead of parents)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46057
diff changeset
79 .map(|(k, v)| Ok((k.extract(py)?, v.extract(py)?)))
46587
cb4b0b0c6de4 copies-rust: split up combine_changeset_copies function into a struct
Simon Sapin <simon.sapin@octobus.net>
parents: 46569
diff changeset
80 .collect::<PyResult<_>>()?;
cb4b0b0c6de4 copies-rust: split up combine_changeset_copies function into a struct
Simon Sapin <simon.sapin@octobus.net>
parents: 46569
diff changeset
81
cb4b0b0c6de4 copies-rust: split up combine_changeset_copies function into a struct
Simon Sapin <simon.sapin@octobus.net>
parents: 46569
diff changeset
82 /// (Revision number, parent 1, parent 2, copy data for this revision)
46589
620c88fb42a2 copies-rust: introduce PyBytesWithData to reduce GIL requirement
Simon Sapin <simon-commits@exyr.org>
parents: 46588
diff changeset
83 type RevInfo<Bytes> = (Revision, Revision, Revision, Option<Bytes>);
46587
cb4b0b0c6de4 copies-rust: split up combine_changeset_copies function into a struct
Simon Sapin <simon.sapin@octobus.net>
parents: 46569
diff changeset
84
46589
620c88fb42a2 copies-rust: introduce PyBytesWithData to reduce GIL requirement
Simon Sapin <simon-commits@exyr.org>
parents: 46588
diff changeset
85 let revs_info =
620c88fb42a2 copies-rust: introduce PyBytesWithData to reduce GIL requirement
Simon Sapin <simon-commits@exyr.org>
parents: 46588
diff changeset
86 revs.iter(py).map(|rev_py| -> PyResult<RevInfo<PyBytes>> {
620c88fb42a2 copies-rust: introduce PyBytesWithData to reduce GIL requirement
Simon Sapin <simon-commits@exyr.org>
parents: 46588
diff changeset
87 let rev = rev_py.extract(py)?;
620c88fb42a2 copies-rust: introduce PyBytesWithData to reduce GIL requirement
Simon Sapin <simon-commits@exyr.org>
parents: 46588
diff changeset
88 let tuple: PyTuple =
620c88fb42a2 copies-rust: introduce PyBytesWithData to reduce GIL requirement
Simon Sapin <simon-commits@exyr.org>
parents: 46588
diff changeset
89 rev_info.call(py, (rev_py,), None)?.cast_into(py)?;
620c88fb42a2 copies-rust: introduce PyBytesWithData to reduce GIL requirement
Simon Sapin <simon-commits@exyr.org>
parents: 46588
diff changeset
90 let p1 = tuple.get_item(py, 0).extract(py)?;
620c88fb42a2 copies-rust: introduce PyBytesWithData to reduce GIL requirement
Simon Sapin <simon-commits@exyr.org>
parents: 46588
diff changeset
91 let p2 = tuple.get_item(py, 1).extract(py)?;
620c88fb42a2 copies-rust: introduce PyBytesWithData to reduce GIL requirement
Simon Sapin <simon-commits@exyr.org>
parents: 46588
diff changeset
92 let opt_bytes = tuple.get_item(py, 2).extract(py)?;
620c88fb42a2 copies-rust: introduce PyBytesWithData to reduce GIL requirement
Simon Sapin <simon-commits@exyr.org>
parents: 46588
diff changeset
93 Ok((rev, p1, p2, opt_bytes))
620c88fb42a2 copies-rust: introduce PyBytesWithData to reduce GIL requirement
Simon Sapin <simon-commits@exyr.org>
parents: 46588
diff changeset
94 });
45945
50c5ee3bdf9a copies: introduce the hg-cpython wrapper for `combine_changeset_copies`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
95
46588
47557ea79fc7 copies-rust: move CPU-heavy Rust processing into a child thread
Simon Sapin <simon.sapin@octobus.net>
parents: 46587
diff changeset
96 let path_copies = if !multi_thread {
47557ea79fc7 copies-rust: move CPU-heavy Rust processing into a child thread
Simon Sapin <simon.sapin@octobus.net>
parents: 46587
diff changeset
97 let mut combine_changeset_copies =
47557ea79fc7 copies-rust: move CPU-heavy Rust processing into a child thread
Simon Sapin <simon.sapin@octobus.net>
parents: 46587
diff changeset
98 CombineChangesetCopies::new(children_count);
47557ea79fc7 copies-rust: move CPU-heavy Rust processing into a child thread
Simon Sapin <simon.sapin@octobus.net>
parents: 46587
diff changeset
99
47557ea79fc7 copies-rust: move CPU-heavy Rust processing into a child thread
Simon Sapin <simon.sapin@octobus.net>
parents: 46587
diff changeset
100 for rev_info in revs_info {
47557ea79fc7 copies-rust: move CPU-heavy Rust processing into a child thread
Simon Sapin <simon.sapin@octobus.net>
parents: 46587
diff changeset
101 let (rev, p1, p2, opt_bytes) = rev_info?;
47557ea79fc7 copies-rust: move CPU-heavy Rust processing into a child thread
Simon Sapin <simon.sapin@octobus.net>
parents: 46587
diff changeset
102 let files = match &opt_bytes {
47557ea79fc7 copies-rust: move CPU-heavy Rust processing into a child thread
Simon Sapin <simon.sapin@octobus.net>
parents: 46587
diff changeset
103 Some(bytes) => ChangedFiles::new(bytes.data(py)),
47557ea79fc7 copies-rust: move CPU-heavy Rust processing into a child thread
Simon Sapin <simon.sapin@octobus.net>
parents: 46587
diff changeset
104 // Python None was extracted to Option::None,
47557ea79fc7 copies-rust: move CPU-heavy Rust processing into a child thread
Simon Sapin <simon.sapin@octobus.net>
parents: 46587
diff changeset
105 // meaning there was no copy data.
47557ea79fc7 copies-rust: move CPU-heavy Rust processing into a child thread
Simon Sapin <simon.sapin@octobus.net>
parents: 46587
diff changeset
106 None => ChangedFiles::new_empty(),
47557ea79fc7 copies-rust: move CPU-heavy Rust processing into a child thread
Simon Sapin <simon.sapin@octobus.net>
parents: 46587
diff changeset
107 };
47557ea79fc7 copies-rust: move CPU-heavy Rust processing into a child thread
Simon Sapin <simon.sapin@octobus.net>
parents: 46587
diff changeset
108
47557ea79fc7 copies-rust: move CPU-heavy Rust processing into a child thread
Simon Sapin <simon.sapin@octobus.net>
parents: 46587
diff changeset
109 combine_changeset_copies.add_revision(rev, p1, p2, files)
47557ea79fc7 copies-rust: move CPU-heavy Rust processing into a child thread
Simon Sapin <simon.sapin@octobus.net>
parents: 46587
diff changeset
110 }
47557ea79fc7 copies-rust: move CPU-heavy Rust processing into a child thread
Simon Sapin <simon.sapin@octobus.net>
parents: 46587
diff changeset
111 combine_changeset_copies.finish(target_rev)
47557ea79fc7 copies-rust: move CPU-heavy Rust processing into a child thread
Simon Sapin <simon.sapin@octobus.net>
parents: 46587
diff changeset
112 } else {
47557ea79fc7 copies-rust: move CPU-heavy Rust processing into a child thread
Simon Sapin <simon.sapin@octobus.net>
parents: 46587
diff changeset
113 // Use a bounded channel to provide back-pressure:
47557ea79fc7 copies-rust: move CPU-heavy Rust processing into a child thread
Simon Sapin <simon.sapin@octobus.net>
parents: 46587
diff changeset
114 // if the child thread is slower to process revisions than this thread
47557ea79fc7 copies-rust: move CPU-heavy Rust processing into a child thread
Simon Sapin <simon.sapin@octobus.net>
parents: 46587
diff changeset
115 // is to gather data for them, an unbounded channel would keep
47557ea79fc7 copies-rust: move CPU-heavy Rust processing into a child thread
Simon Sapin <simon.sapin@octobus.net>
parents: 46587
diff changeset
116 // growing and eat memory.
47557ea79fc7 copies-rust: move CPU-heavy Rust processing into a child thread
Simon Sapin <simon.sapin@octobus.net>
parents: 46587
diff changeset
117 //
47557ea79fc7 copies-rust: move CPU-heavy Rust processing into a child thread
Simon Sapin <simon.sapin@octobus.net>
parents: 46587
diff changeset
118 // TODO: tweak the bound?
47557ea79fc7 copies-rust: move CPU-heavy Rust processing into a child thread
Simon Sapin <simon.sapin@octobus.net>
parents: 46587
diff changeset
119 let (rev_info_sender, rev_info_receiver) =
46589
620c88fb42a2 copies-rust: introduce PyBytesWithData to reduce GIL requirement
Simon Sapin <simon-commits@exyr.org>
parents: 46588
diff changeset
120 crossbeam_channel::bounded::<RevInfo<PyBytesWithData>>(1000);
46587
cb4b0b0c6de4 copies-rust: split up combine_changeset_copies function into a struct
Simon Sapin <simon.sapin@octobus.net>
parents: 46569
diff changeset
121
46588
47557ea79fc7 copies-rust: move CPU-heavy Rust processing into a child thread
Simon Sapin <simon.sapin@octobus.net>
parents: 46587
diff changeset
122 // Start a thread that does CPU-heavy processing in parallel with the
47557ea79fc7 copies-rust: move CPU-heavy Rust processing into a child thread
Simon Sapin <simon.sapin@octobus.net>
parents: 46587
diff changeset
123 // loop below.
47557ea79fc7 copies-rust: move CPU-heavy Rust processing into a child thread
Simon Sapin <simon.sapin@octobus.net>
parents: 46587
diff changeset
124 //
47557ea79fc7 copies-rust: move CPU-heavy Rust processing into a child thread
Simon Sapin <simon.sapin@octobus.net>
parents: 46587
diff changeset
125 // If the parent thread panics, `rev_info_sender` will be dropped and
47557ea79fc7 copies-rust: move CPU-heavy Rust processing into a child thread
Simon Sapin <simon.sapin@octobus.net>
parents: 46587
diff changeset
126 // “disconnected”. `rev_info_receiver` will be notified of this and
47557ea79fc7 copies-rust: move CPU-heavy Rust processing into a child thread
Simon Sapin <simon.sapin@octobus.net>
parents: 46587
diff changeset
127 // exit its own loop.
47557ea79fc7 copies-rust: move CPU-heavy Rust processing into a child thread
Simon Sapin <simon.sapin@octobus.net>
parents: 46587
diff changeset
128 let thread = std::thread::spawn(move || {
47557ea79fc7 copies-rust: move CPU-heavy Rust processing into a child thread
Simon Sapin <simon.sapin@octobus.net>
parents: 46587
diff changeset
129 let mut combine_changeset_copies =
47557ea79fc7 copies-rust: move CPU-heavy Rust processing into a child thread
Simon Sapin <simon.sapin@octobus.net>
parents: 46587
diff changeset
130 CombineChangesetCopies::new(children_count);
47557ea79fc7 copies-rust: move CPU-heavy Rust processing into a child thread
Simon Sapin <simon.sapin@octobus.net>
parents: 46587
diff changeset
131 for (rev, p1, p2, opt_bytes) in rev_info_receiver {
47557ea79fc7 copies-rust: move CPU-heavy Rust processing into a child thread
Simon Sapin <simon.sapin@octobus.net>
parents: 46587
diff changeset
132 let files = match &opt_bytes {
46589
620c88fb42a2 copies-rust: introduce PyBytesWithData to reduce GIL requirement
Simon Sapin <simon-commits@exyr.org>
parents: 46588
diff changeset
133 Some(raw) => ChangedFiles::new(raw.data()),
46588
47557ea79fc7 copies-rust: move CPU-heavy Rust processing into a child thread
Simon Sapin <simon.sapin@octobus.net>
parents: 46587
diff changeset
134 // Python None was extracted to Option::None,
47557ea79fc7 copies-rust: move CPU-heavy Rust processing into a child thread
Simon Sapin <simon.sapin@octobus.net>
parents: 46587
diff changeset
135 // meaning there was no copy data.
47557ea79fc7 copies-rust: move CPU-heavy Rust processing into a child thread
Simon Sapin <simon.sapin@octobus.net>
parents: 46587
diff changeset
136 None => ChangedFiles::new_empty(),
47557ea79fc7 copies-rust: move CPU-heavy Rust processing into a child thread
Simon Sapin <simon.sapin@octobus.net>
parents: 46587
diff changeset
137 };
47557ea79fc7 copies-rust: move CPU-heavy Rust processing into a child thread
Simon Sapin <simon.sapin@octobus.net>
parents: 46587
diff changeset
138 combine_changeset_copies.add_revision(rev, p1, p2, files)
46589
620c88fb42a2 copies-rust: introduce PyBytesWithData to reduce GIL requirement
Simon Sapin <simon-commits@exyr.org>
parents: 46588
diff changeset
139
620c88fb42a2 copies-rust: introduce PyBytesWithData to reduce GIL requirement
Simon Sapin <simon-commits@exyr.org>
parents: 46588
diff changeset
140 // The GIL is (still) implicitly acquired here through
620c88fb42a2 copies-rust: introduce PyBytesWithData to reduce GIL requirement
Simon Sapin <simon-commits@exyr.org>
parents: 46588
diff changeset
141 // `impl Drop for PyBytes`.
46588
47557ea79fc7 copies-rust: move CPU-heavy Rust processing into a child thread
Simon Sapin <simon.sapin@octobus.net>
parents: 46587
diff changeset
142 }
47557ea79fc7 copies-rust: move CPU-heavy Rust processing into a child thread
Simon Sapin <simon.sapin@octobus.net>
parents: 46587
diff changeset
143
47557ea79fc7 copies-rust: move CPU-heavy Rust processing into a child thread
Simon Sapin <simon.sapin@octobus.net>
parents: 46587
diff changeset
144 combine_changeset_copies.finish(target_rev)
47557ea79fc7 copies-rust: move CPU-heavy Rust processing into a child thread
Simon Sapin <simon.sapin@octobus.net>
parents: 46587
diff changeset
145 });
46587
cb4b0b0c6de4 copies-rust: split up combine_changeset_copies function into a struct
Simon Sapin <simon.sapin@octobus.net>
parents: 46569
diff changeset
146
46588
47557ea79fc7 copies-rust: move CPU-heavy Rust processing into a child thread
Simon Sapin <simon.sapin@octobus.net>
parents: 46587
diff changeset
147 for rev_info in revs_info {
47557ea79fc7 copies-rust: move CPU-heavy Rust processing into a child thread
Simon Sapin <simon.sapin@octobus.net>
parents: 46587
diff changeset
148 let (rev, p1, p2, opt_bytes) = rev_info?;
46589
620c88fb42a2 copies-rust: introduce PyBytesWithData to reduce GIL requirement
Simon Sapin <simon-commits@exyr.org>
parents: 46588
diff changeset
149 let opt_bytes = opt_bytes.map(|b| PyBytesWithData::new(py, b));
46588
47557ea79fc7 copies-rust: move CPU-heavy Rust processing into a child thread
Simon Sapin <simon.sapin@octobus.net>
parents: 46587
diff changeset
150
47557ea79fc7 copies-rust: move CPU-heavy Rust processing into a child thread
Simon Sapin <simon.sapin@octobus.net>
parents: 46587
diff changeset
151 // We’d prefer to avoid the child thread calling into Python code,
47557ea79fc7 copies-rust: move CPU-heavy Rust processing into a child thread
Simon Sapin <simon.sapin@octobus.net>
parents: 46587
diff changeset
152 // but this avoids a potential deadlock on the GIL if it does:
47557ea79fc7 copies-rust: move CPU-heavy Rust processing into a child thread
Simon Sapin <simon.sapin@octobus.net>
parents: 46587
diff changeset
153 py.allow_threads(|| {
47557ea79fc7 copies-rust: move CPU-heavy Rust processing into a child thread
Simon Sapin <simon.sapin@octobus.net>
parents: 46587
diff changeset
154 rev_info_sender.send((rev, p1, p2, opt_bytes)).expect(
47557ea79fc7 copies-rust: move CPU-heavy Rust processing into a child thread
Simon Sapin <simon.sapin@octobus.net>
parents: 46587
diff changeset
155 "combine_changeset_copies: channel is disconnected",
47557ea79fc7 copies-rust: move CPU-heavy Rust processing into a child thread
Simon Sapin <simon.sapin@octobus.net>
parents: 46587
diff changeset
156 );
47557ea79fc7 copies-rust: move CPU-heavy Rust processing into a child thread
Simon Sapin <simon.sapin@octobus.net>
parents: 46587
diff changeset
157 });
47557ea79fc7 copies-rust: move CPU-heavy Rust processing into a child thread
Simon Sapin <simon.sapin@octobus.net>
parents: 46587
diff changeset
158 }
47557ea79fc7 copies-rust: move CPU-heavy Rust processing into a child thread
Simon Sapin <simon.sapin@octobus.net>
parents: 46587
diff changeset
159 // We’d prefer to avoid the child thread calling into Python code,
47557ea79fc7 copies-rust: move CPU-heavy Rust processing into a child thread
Simon Sapin <simon.sapin@octobus.net>
parents: 46587
diff changeset
160 // but this avoids a potential deadlock on the GIL if it does:
47557ea79fc7 copies-rust: move CPU-heavy Rust processing into a child thread
Simon Sapin <simon.sapin@octobus.net>
parents: 46587
diff changeset
161 py.allow_threads(|| {
47557ea79fc7 copies-rust: move CPU-heavy Rust processing into a child thread
Simon Sapin <simon.sapin@octobus.net>
parents: 46587
diff changeset
162 // Disconnect the channel to signal the child thread to stop:
47557ea79fc7 copies-rust: move CPU-heavy Rust processing into a child thread
Simon Sapin <simon.sapin@octobus.net>
parents: 46587
diff changeset
163 // the `for … in rev_info_receiver` loop will end.
47557ea79fc7 copies-rust: move CPU-heavy Rust processing into a child thread
Simon Sapin <simon.sapin@octobus.net>
parents: 46587
diff changeset
164 drop(rev_info_sender);
47557ea79fc7 copies-rust: move CPU-heavy Rust processing into a child thread
Simon Sapin <simon.sapin@octobus.net>
parents: 46587
diff changeset
165
47557ea79fc7 copies-rust: move CPU-heavy Rust processing into a child thread
Simon Sapin <simon.sapin@octobus.net>
parents: 46587
diff changeset
166 // Wait for the child thread to stop, and propagate any panic.
47557ea79fc7 copies-rust: move CPU-heavy Rust processing into a child thread
Simon Sapin <simon.sapin@octobus.net>
parents: 46587
diff changeset
167 thread.join().unwrap_or_else(|panic_payload| {
47557ea79fc7 copies-rust: move CPU-heavy Rust processing into a child thread
Simon Sapin <simon.sapin@octobus.net>
parents: 46587
diff changeset
168 std::panic::resume_unwind(panic_payload)
47557ea79fc7 copies-rust: move CPU-heavy Rust processing into a child thread
Simon Sapin <simon.sapin@octobus.net>
parents: 46587
diff changeset
169 })
47557ea79fc7 copies-rust: move CPU-heavy Rust processing into a child thread
Simon Sapin <simon.sapin@octobus.net>
parents: 46587
diff changeset
170 })
47557ea79fc7 copies-rust: move CPU-heavy Rust processing into a child thread
Simon Sapin <simon.sapin@octobus.net>
parents: 46587
diff changeset
171 };
47557ea79fc7 copies-rust: move CPU-heavy Rust processing into a child thread
Simon Sapin <simon.sapin@octobus.net>
parents: 46587
diff changeset
172
45945
50c5ee3bdf9a copies: introduce the hg-cpython wrapper for `combine_changeset_copies`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
173 let out = PyDict::new(py);
46587
cb4b0b0c6de4 copies-rust: split up combine_changeset_copies function into a struct
Simon Sapin <simon.sapin@octobus.net>
parents: 46569
diff changeset
174 for (dest, source) in path_copies.into_iter() {
45945
50c5ee3bdf9a copies: introduce the hg-cpython wrapper for `combine_changeset_copies`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
175 out.set_item(
50c5ee3bdf9a copies: introduce the hg-cpython wrapper for `combine_changeset_copies`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
176 py,
50c5ee3bdf9a copies: introduce the hg-cpython wrapper for `combine_changeset_copies`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
177 PyBytes::new(py, &dest.into_vec()),
50c5ee3bdf9a copies: introduce the hg-cpython wrapper for `combine_changeset_copies`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
178 PyBytes::new(py, &source.into_vec()),
50c5ee3bdf9a copies: introduce the hg-cpython wrapper for `combine_changeset_copies`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
179 )?;
50c5ee3bdf9a copies: introduce the hg-cpython wrapper for `combine_changeset_copies`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
180 }
50c5ee3bdf9a copies: introduce the hg-cpython wrapper for `combine_changeset_copies`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
181 Ok(out)
50c5ee3bdf9a copies: introduce the hg-cpython wrapper for `combine_changeset_copies`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
182 }
50c5ee3bdf9a copies: introduce the hg-cpython wrapper for `combine_changeset_copies`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
183
50c5ee3bdf9a copies: introduce the hg-cpython wrapper for `combine_changeset_copies`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
184 /// Create the module, with `__package__` given from parent
50c5ee3bdf9a copies: introduce the hg-cpython wrapper for `combine_changeset_copies`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
185 pub fn init_module(py: Python, package: &str) -> PyResult<PyModule> {
50c5ee3bdf9a copies: introduce the hg-cpython wrapper for `combine_changeset_copies`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
186 let dotted_name = &format!("{}.copy_tracing", package);
50c5ee3bdf9a copies: introduce the hg-cpython wrapper for `combine_changeset_copies`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
187 let m = PyModule::new(py, dotted_name)?;
50c5ee3bdf9a copies: introduce the hg-cpython wrapper for `combine_changeset_copies`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
188
50c5ee3bdf9a copies: introduce the hg-cpython wrapper for `combine_changeset_copies`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
189 m.add(py, "__package__", package)?;
50c5ee3bdf9a copies: introduce the hg-cpython wrapper for `combine_changeset_copies`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
190 m.add(py, "__doc__", "Copy tracing - Rust implementation")?;
50c5ee3bdf9a copies: introduce the hg-cpython wrapper for `combine_changeset_copies`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
191
50c5ee3bdf9a copies: introduce the hg-cpython wrapper for `combine_changeset_copies`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
192 m.add(
50c5ee3bdf9a copies: introduce the hg-cpython wrapper for `combine_changeset_copies`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
193 py,
50c5ee3bdf9a copies: introduce the hg-cpython wrapper for `combine_changeset_copies`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
194 "combine_changeset_copies",
50c5ee3bdf9a copies: introduce the hg-cpython wrapper for `combine_changeset_copies`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
195 py_fn!(
50c5ee3bdf9a copies: introduce the hg-cpython wrapper for `combine_changeset_copies`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
196 py,
50c5ee3bdf9a copies: introduce the hg-cpython wrapper for `combine_changeset_copies`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
197 combine_changeset_copies_wrapper(
50c5ee3bdf9a copies: introduce the hg-cpython wrapper for `combine_changeset_copies`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
198 revs: PyList,
50c5ee3bdf9a copies: introduce the hg-cpython wrapper for `combine_changeset_copies`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
199 children: PyDict,
50c5ee3bdf9a copies: introduce the hg-cpython wrapper for `combine_changeset_copies`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
200 target_rev: Revision,
46588
47557ea79fc7 copies-rust: move CPU-heavy Rust processing into a child thread
Simon Sapin <simon.sapin@octobus.net>
parents: 46587
diff changeset
201 rev_info: PyObject,
47557ea79fc7 copies-rust: move CPU-heavy Rust processing into a child thread
Simon Sapin <simon.sapin@octobus.net>
parents: 46587
diff changeset
202 multi_thread: bool
45945
50c5ee3bdf9a copies: introduce the hg-cpython wrapper for `combine_changeset_copies`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
203 )
50c5ee3bdf9a copies: introduce the hg-cpython wrapper for `combine_changeset_copies`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
204 ),
50c5ee3bdf9a copies: introduce the hg-cpython wrapper for `combine_changeset_copies`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
205 )?;
50c5ee3bdf9a copies: introduce the hg-cpython wrapper for `combine_changeset_copies`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
206
50c5ee3bdf9a copies: introduce the hg-cpython wrapper for `combine_changeset_copies`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
207 let sys = PyModule::import(py, "sys")?;
50c5ee3bdf9a copies: introduce the hg-cpython wrapper for `combine_changeset_copies`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
208 let sys_modules: PyDict = sys.get(py, "modules")?.extract(py)?;
50c5ee3bdf9a copies: introduce the hg-cpython wrapper for `combine_changeset_copies`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
209 sys_modules.set_item(py, dotted_name, &m)?;
50c5ee3bdf9a copies: introduce the hg-cpython wrapper for `combine_changeset_copies`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
210
50c5ee3bdf9a copies: introduce the hg-cpython wrapper for `combine_changeset_copies`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
211 Ok(m)
50c5ee3bdf9a copies: introduce the hg-cpython wrapper for `combine_changeset_copies`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
212 }