Mercurial > public > mercurial-scm > hg

annotate rust/hg-core/src/revlog/nodemap_docket.rs @ 48178:f12a19d03d2c

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
fix: reduce number of tool executions By grouping together (path, ctx) pairs according to the inputs they would provide to fixer tools, we can deduplicate executions of fixer tools to significantly reduce the amount of time spent running slow tools. This change does not handle clean files in the working copy, which could still be deduplicated against the files in the checked out commit. It's a little harder to do that because the filerev is not available in the workingfilectx (and it doesn't exist for added files). Anecdotally, this change makes some real uses cases at Google 10x faster. I think we were originally hesitant to do this because the benefits weren't obvious, and implementing it efficiently is kind of tricky. If we simply memoized the formatter execution function, we would be keeping tons of file content in memory. Also included is a regression test for a corner case that I broke with my first attempt at optimizing this code. Differential Revision: https://phab.mercurial-scm.org/D11280
author Danny Hooper <hooper@google.com>
date Thu, 02 Sep 2021 14:08:45 -0700
parents 001d747c2baf
children 07d8d144c222
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
46443
43d63979a75e rust: use HgError in RevlogError and Vfs
Simon Sapin <simon.sapin@octobus.net>
parents: 46390
diff changeset
1 use crate::errors::{HgError, HgResultExt};
46630
842f2372ced6 rhg: Don?t attempt to read persistent nodemap without .hg/requires opt-in
Simon Sapin <simon.sapin@octobus.net>
parents: 46443
diff changeset
2 use crate::requirements;
46390
0800aa42bb4c rust: use the bytes-cast crate to parse persistent nodemaps
Simon Sapin <simon.sapin@octobus.net>
parents: 46167
diff changeset
3 use bytes_cast::{unaligned, BytesCast};
47955
e834b79def74 rust: Switch to the memmap2-rs crate
Simon Sapin <simon.sapin@octobus.net>
parents: 46630
diff changeset
4 use memmap2::Mmap;
46090
9eb07ab3f2d4 rhg: use persistent nodemap when available
Simon Sapin <simon-commits@exyr.org>
parents:
diff changeset
5 use std::path::{Path, PathBuf};
9eb07ab3f2d4 rhg: use persistent nodemap when available
Simon Sapin <simon-commits@exyr.org>
parents:
diff changeset
6
46167
8a4914397d02 rust: introduce Repo and Vfs types for filesystem abstraction
Simon Sapin <simon.sapin@octobus.net>
parents: 46090
diff changeset
7 use crate::repo::Repo;
46090
9eb07ab3f2d4 rhg: use persistent nodemap when available
Simon Sapin <simon-commits@exyr.org>
parents:
diff changeset
8 use crate::utils::strip_suffix;
9eb07ab3f2d4 rhg: use persistent nodemap when available
Simon Sapin <simon-commits@exyr.org>
parents:
diff changeset
9
9eb07ab3f2d4 rhg: use persistent nodemap when available
Simon Sapin <simon-commits@exyr.org>
parents:
diff changeset
10 const ONDISK_VERSION: u8 = 1;
9eb07ab3f2d4 rhg: use persistent nodemap when available
Simon Sapin <simon-commits@exyr.org>
parents:
diff changeset
11
9eb07ab3f2d4 rhg: use persistent nodemap when available
Simon Sapin <simon-commits@exyr.org>
parents:
diff changeset
12 pub(super) struct NodeMapDocket {
9eb07ab3f2d4 rhg: use persistent nodemap when available
Simon Sapin <simon-commits@exyr.org>
parents:
diff changeset
13 pub data_length: usize,
9eb07ab3f2d4 rhg: use persistent nodemap when available
Simon Sapin <simon-commits@exyr.org>
parents:
diff changeset
14 // TODO: keep here more of the data from `parse()` when we need it
9eb07ab3f2d4 rhg: use persistent nodemap when available
Simon Sapin <simon-commits@exyr.org>
parents:
diff changeset
15 }
9eb07ab3f2d4 rhg: use persistent nodemap when available
Simon Sapin <simon-commits@exyr.org>
parents:
diff changeset
16
46390
0800aa42bb4c rust: use the bytes-cast crate to parse persistent nodemaps
Simon Sapin <simon.sapin@octobus.net>
parents: 46167
diff changeset
17 #[derive(BytesCast)]
0800aa42bb4c rust: use the bytes-cast crate to parse persistent nodemaps
Simon Sapin <simon.sapin@octobus.net>
parents: 46167
diff changeset
18 #[repr(C)]
0800aa42bb4c rust: use the bytes-cast crate to parse persistent nodemaps
Simon Sapin <simon.sapin@octobus.net>
parents: 46167
diff changeset
19 struct DocketHeader {
0800aa42bb4c rust: use the bytes-cast crate to parse persistent nodemaps
Simon Sapin <simon.sapin@octobus.net>
parents: 46167
diff changeset
20 uid_size: u8,
0800aa42bb4c rust: use the bytes-cast crate to parse persistent nodemaps
Simon Sapin <simon.sapin@octobus.net>
parents: 46167
diff changeset
21 _tip_rev: unaligned::U64Be,
0800aa42bb4c rust: use the bytes-cast crate to parse persistent nodemaps
Simon Sapin <simon.sapin@octobus.net>
parents: 46167
diff changeset
22 data_length: unaligned::U64Be,
0800aa42bb4c rust: use the bytes-cast crate to parse persistent nodemaps
Simon Sapin <simon.sapin@octobus.net>
parents: 46167
diff changeset
23 _data_unused: unaligned::U64Be,
0800aa42bb4c rust: use the bytes-cast crate to parse persistent nodemaps
Simon Sapin <simon.sapin@octobus.net>
parents: 46167
diff changeset
24 tip_node_size: unaligned::U64Be,
0800aa42bb4c rust: use the bytes-cast crate to parse persistent nodemaps
Simon Sapin <simon.sapin@octobus.net>
parents: 46167
diff changeset
25 }
0800aa42bb4c rust: use the bytes-cast crate to parse persistent nodemaps
Simon Sapin <simon.sapin@octobus.net>
parents: 46167
diff changeset
26
46090
9eb07ab3f2d4 rhg: use persistent nodemap when available
Simon Sapin <simon-commits@exyr.org>
parents:
diff changeset
27 impl NodeMapDocket {
9eb07ab3f2d4 rhg: use persistent nodemap when available
Simon Sapin <simon-commits@exyr.org>
parents:
diff changeset
28 /// Return `Ok(None)` when the caller should proceed without a persistent
9eb07ab3f2d4 rhg: use persistent nodemap when available
Simon Sapin <simon-commits@exyr.org>
parents:
diff changeset
29 /// nodemap:
9eb07ab3f2d4 rhg: use persistent nodemap when available
Simon Sapin <simon-commits@exyr.org>
parents:
diff changeset
30 ///
9eb07ab3f2d4 rhg: use persistent nodemap when available
Simon Sapin <simon-commits@exyr.org>
parents:
diff changeset
31 /// * This revlog does not have a `.n` docket file (it is not generated for
9eb07ab3f2d4 rhg: use persistent nodemap when available
Simon Sapin <simon-commits@exyr.org>
parents:
diff changeset
32 /// small revlogs), or
9eb07ab3f2d4 rhg: use persistent nodemap when available
Simon Sapin <simon-commits@exyr.org>
parents:
diff changeset
33 /// * The docket has an unsupported version number (repositories created by
9eb07ab3f2d4 rhg: use persistent nodemap when available
Simon Sapin <simon-commits@exyr.org>
parents:
diff changeset
34 /// later hg, maybe that should be a requirement instead?), or
9eb07ab3f2d4 rhg: use persistent nodemap when available
Simon Sapin <simon-commits@exyr.org>
parents:
diff changeset
35 /// * The docket file points to a missing (likely deleted) data file (this
9eb07ab3f2d4 rhg: use persistent nodemap when available
Simon Sapin <simon-commits@exyr.org>
parents:
diff changeset
36 /// can happen in a rare race condition).
9eb07ab3f2d4 rhg: use persistent nodemap when available
Simon Sapin <simon-commits@exyr.org>
parents:
diff changeset
37 pub fn read_from_file(
46167
8a4914397d02 rust: introduce Repo and Vfs types for filesystem abstraction
Simon Sapin <simon.sapin@octobus.net>
parents: 46090
diff changeset
38 repo: &Repo,
46090
9eb07ab3f2d4 rhg: use persistent nodemap when available
Simon Sapin <simon-commits@exyr.org>
parents:
diff changeset
39 index_path: &Path,
47963
001d747c2baf rust: Return HgError instead of RevlogError in revlog constructors
Simon Sapin <simon.sapin@octobus.net>
parents: 47955
diff changeset
40 ) -> Result<Option<(Self, Mmap)>, HgError> {
46630
842f2372ced6 rhg: Don?t attempt to read persistent nodemap without .hg/requires opt-in
Simon Sapin <simon.sapin@octobus.net>
parents: 46443
diff changeset
41 if !repo
842f2372ced6 rhg: Don?t attempt to read persistent nodemap without .hg/requires opt-in
Simon Sapin <simon.sapin@octobus.net>
parents: 46443
diff changeset
42 .requirements()
842f2372ced6 rhg: Don?t attempt to read persistent nodemap without .hg/requires opt-in
Simon Sapin <simon.sapin@octobus.net>
parents: 46443
diff changeset
43 .contains(requirements::NODEMAP_REQUIREMENT)
842f2372ced6 rhg: Don?t attempt to read persistent nodemap without .hg/requires opt-in
Simon Sapin <simon.sapin@octobus.net>
parents: 46443
diff changeset
44 {
842f2372ced6 rhg: Don?t attempt to read persistent nodemap without .hg/requires opt-in
Simon Sapin <simon.sapin@octobus.net>
parents: 46443
diff changeset
45 // If .hg/requires does not opt it, don’t try to open a nodemap
842f2372ced6 rhg: Don?t attempt to read persistent nodemap without .hg/requires opt-in
Simon Sapin <simon.sapin@octobus.net>
parents: 46443
diff changeset
46 return Ok(None);
842f2372ced6 rhg: Don?t attempt to read persistent nodemap without .hg/requires opt-in
Simon Sapin <simon.sapin@octobus.net>
parents: 46443
diff changeset
47 }
842f2372ced6 rhg: Don?t attempt to read persistent nodemap without .hg/requires opt-in
Simon Sapin <simon.sapin@octobus.net>
parents: 46443
diff changeset
48
46090
9eb07ab3f2d4 rhg: use persistent nodemap when available
Simon Sapin <simon-commits@exyr.org>
parents:
diff changeset
49 let docket_path = index_path.with_extension("n");
46443
43d63979a75e rust: use HgError in RevlogError and Vfs
Simon Sapin <simon.sapin@octobus.net>
parents: 46390
diff changeset
50 let docket_bytes = if let Some(bytes) =
43d63979a75e rust: use HgError in RevlogError and Vfs
Simon Sapin <simon.sapin@octobus.net>
parents: 46390
diff changeset
51 repo.store_vfs().read(&docket_path).io_not_found_as_none()?
43d63979a75e rust: use HgError in RevlogError and Vfs
Simon Sapin <simon.sapin@octobus.net>
parents: 46390
diff changeset
52 {
43d63979a75e rust: use HgError in RevlogError and Vfs
Simon Sapin <simon.sapin@octobus.net>
parents: 46390
diff changeset
53 bytes
43d63979a75e rust: use HgError in RevlogError and Vfs
Simon Sapin <simon.sapin@octobus.net>
parents: 46390
diff changeset
54 } else {
43d63979a75e rust: use HgError in RevlogError and Vfs
Simon Sapin <simon.sapin@octobus.net>
parents: 46390
diff changeset
55 return Ok(None);
46090
9eb07ab3f2d4 rhg: use persistent nodemap when available
Simon Sapin <simon-commits@exyr.org>
parents:
diff changeset
56 };
9eb07ab3f2d4 rhg: use persistent nodemap when available
Simon Sapin <simon-commits@exyr.org>
parents:
diff changeset
57
46390
0800aa42bb4c rust: use the bytes-cast crate to parse persistent nodemaps
Simon Sapin <simon.sapin@octobus.net>
parents: 46167
diff changeset
58 let input = if let Some((&ONDISK_VERSION, rest)) =
46090
9eb07ab3f2d4 rhg: use persistent nodemap when available
Simon Sapin <simon-commits@exyr.org>
parents:
diff changeset
59 docket_bytes.split_first()
9eb07ab3f2d4 rhg: use persistent nodemap when available
Simon Sapin <simon-commits@exyr.org>
parents:
diff changeset
60 {
9eb07ab3f2d4 rhg: use persistent nodemap when available
Simon Sapin <simon-commits@exyr.org>
parents:
diff changeset
61 rest
9eb07ab3f2d4 rhg: use persistent nodemap when available
Simon Sapin <simon-commits@exyr.org>
parents:
diff changeset
62 } else {
9eb07ab3f2d4 rhg: use persistent nodemap when available
Simon Sapin <simon-commits@exyr.org>
parents:
diff changeset
63 return Ok(None);
9eb07ab3f2d4 rhg: use persistent nodemap when available
Simon Sapin <simon-commits@exyr.org>
parents:
diff changeset
64 };
9eb07ab3f2d4 rhg: use persistent nodemap when available
Simon Sapin <simon-commits@exyr.org>
parents:
diff changeset
65
46443
43d63979a75e rust: use HgError in RevlogError and Vfs
Simon Sapin <simon.sapin@octobus.net>
parents: 46390
diff changeset
66 /// Treat any error as a parse error
47963
001d747c2baf rust: Return HgError instead of RevlogError in revlog constructors
Simon Sapin <simon.sapin@octobus.net>
parents: 47955
diff changeset
67 fn parse<T, E>(result: Result<T, E>) -> Result<T, HgError> {
001d747c2baf rust: Return HgError instead of RevlogError in revlog constructors
Simon Sapin <simon.sapin@octobus.net>
parents: 47955
diff changeset
68 result
001d747c2baf rust: Return HgError instead of RevlogError in revlog constructors
Simon Sapin <simon.sapin@octobus.net>
parents: 47955
diff changeset
69 .map_err(|_| HgError::corrupted("nodemap docket parse error"))
46443
43d63979a75e rust: use HgError in RevlogError and Vfs
Simon Sapin <simon.sapin@octobus.net>
parents: 46390
diff changeset
70 }
43d63979a75e rust: use HgError in RevlogError and Vfs
Simon Sapin <simon.sapin@octobus.net>
parents: 46390
diff changeset
71
43d63979a75e rust: use HgError in RevlogError and Vfs
Simon Sapin <simon.sapin@octobus.net>
parents: 46390
diff changeset
72 let (header, rest) = parse(DocketHeader::from_bytes(input))?;
46390
0800aa42bb4c rust: use the bytes-cast crate to parse persistent nodemaps
Simon Sapin <simon.sapin@octobus.net>
parents: 46167
diff changeset
73 let uid_size = header.uid_size as usize;
46090
9eb07ab3f2d4 rhg: use persistent nodemap when available
Simon Sapin <simon-commits@exyr.org>
parents:
diff changeset
74 // TODO: do we care about overflow for 4 GB+ nodemap files on 32-bit
9eb07ab3f2d4 rhg: use persistent nodemap when available
Simon Sapin <simon-commits@exyr.org>
parents:
diff changeset
75 // systems?
46390
0800aa42bb4c rust: use the bytes-cast crate to parse persistent nodemaps
Simon Sapin <simon.sapin@octobus.net>
parents: 46167
diff changeset
76 let tip_node_size = header.tip_node_size.get() as usize;
0800aa42bb4c rust: use the bytes-cast crate to parse persistent nodemaps
Simon Sapin <simon.sapin@octobus.net>
parents: 46167
diff changeset
77 let data_length = header.data_length.get() as usize;
46443
43d63979a75e rust: use HgError in RevlogError and Vfs
Simon Sapin <simon.sapin@octobus.net>
parents: 46390
diff changeset
78 let (uid, rest) = parse(u8::slice_from_bytes(rest, uid_size))?;
43d63979a75e rust: use HgError in RevlogError and Vfs
Simon Sapin <simon.sapin@octobus.net>
parents: 46390
diff changeset
79 let (_tip_node, _rest) =
43d63979a75e rust: use HgError in RevlogError and Vfs
Simon Sapin <simon.sapin@octobus.net>
parents: 46390
diff changeset
80 parse(u8::slice_from_bytes(rest, tip_node_size))?;
43d63979a75e rust: use HgError in RevlogError and Vfs
Simon Sapin <simon.sapin@octobus.net>
parents: 46390
diff changeset
81 let uid = parse(std::str::from_utf8(uid))?;
46090
9eb07ab3f2d4 rhg: use persistent nodemap when available
Simon Sapin <simon-commits@exyr.org>
parents:
diff changeset
82 let docket = NodeMapDocket { data_length };
9eb07ab3f2d4 rhg: use persistent nodemap when available
Simon Sapin <simon-commits@exyr.org>
parents:
diff changeset
83
9eb07ab3f2d4 rhg: use persistent nodemap when available
Simon Sapin <simon-commits@exyr.org>
parents:
diff changeset
84 let data_path = rawdata_path(&docket_path, uid);
46443
43d63979a75e rust: use HgError in RevlogError and Vfs
Simon Sapin <simon.sapin@octobus.net>
parents: 46390
diff changeset
85 // TODO: use `vfs.read()` here when the `persistent-nodemap.mmap`
46090
9eb07ab3f2d4 rhg: use persistent nodemap when available
Simon Sapin <simon-commits@exyr.org>
parents:
diff changeset
86 // config is false?
46443
43d63979a75e rust: use HgError in RevlogError and Vfs
Simon Sapin <simon.sapin@octobus.net>
parents: 46390
diff changeset
87 if let Some(mmap) = repo
43d63979a75e rust: use HgError in RevlogError and Vfs
Simon Sapin <simon.sapin@octobus.net>
parents: 46390
diff changeset
88 .store_vfs()
43d63979a75e rust: use HgError in RevlogError and Vfs
Simon Sapin <simon.sapin@octobus.net>
parents: 46390
diff changeset
89 .mmap_open(&data_path)
43d63979a75e rust: use HgError in RevlogError and Vfs
Simon Sapin <simon.sapin@octobus.net>
parents: 46390
diff changeset
90 .io_not_found_as_none()?
43d63979a75e rust: use HgError in RevlogError and Vfs
Simon Sapin <simon.sapin@octobus.net>
parents: 46390
diff changeset
91 {
43d63979a75e rust: use HgError in RevlogError and Vfs
Simon Sapin <simon.sapin@octobus.net>
parents: 46390
diff changeset
92 if mmap.len() >= data_length {
43d63979a75e rust: use HgError in RevlogError and Vfs
Simon Sapin <simon.sapin@octobus.net>
parents: 46390
diff changeset
93 Ok(Some((docket, mmap)))
43d63979a75e rust: use HgError in RevlogError and Vfs
Simon Sapin <simon.sapin@octobus.net>
parents: 46390
diff changeset
94 } else {
47963
001d747c2baf rust: Return HgError instead of RevlogError in revlog constructors
Simon Sapin <simon.sapin@octobus.net>
parents: 47955
diff changeset
95 Err(HgError::corrupted("persistent nodemap too short"))
46090
9eb07ab3f2d4 rhg: use persistent nodemap when available
Simon Sapin <simon-commits@exyr.org>
parents:
diff changeset
96 }
46443
43d63979a75e rust: use HgError in RevlogError and Vfs
Simon Sapin <simon.sapin@octobus.net>
parents: 46390
diff changeset
97 } else {
46630
842f2372ced6 rhg: Don?t attempt to read persistent nodemap without .hg/requires opt-in
Simon Sapin <simon.sapin@octobus.net>
parents: 46443
diff changeset
98 // Even if .hg/requires opted in, some revlogs are deemed small
842f2372ced6 rhg: Don?t attempt to read persistent nodemap without .hg/requires opt-in
Simon Sapin <simon.sapin@octobus.net>
parents: 46443
diff changeset
99 // enough to not need a persistent nodemap.
46443
43d63979a75e rust: use HgError in RevlogError and Vfs
Simon Sapin <simon.sapin@octobus.net>
parents: 46390
diff changeset
100 Ok(None)
46090
9eb07ab3f2d4 rhg: use persistent nodemap when available
Simon Sapin <simon-commits@exyr.org>
parents:
diff changeset
101 }
9eb07ab3f2d4 rhg: use persistent nodemap when available
Simon Sapin <simon-commits@exyr.org>
parents:
diff changeset
102 }
9eb07ab3f2d4 rhg: use persistent nodemap when available
Simon Sapin <simon-commits@exyr.org>
parents:
diff changeset
103 }
9eb07ab3f2d4 rhg: use persistent nodemap when available
Simon Sapin <simon-commits@exyr.org>
parents:
diff changeset
104
9eb07ab3f2d4 rhg: use persistent nodemap when available
Simon Sapin <simon-commits@exyr.org>
parents:
diff changeset
105 fn rawdata_path(docket_path: &Path, uid: &str) -> PathBuf {
9eb07ab3f2d4 rhg: use persistent nodemap when available
Simon Sapin <simon-commits@exyr.org>
parents:
diff changeset
106 let docket_name = docket_path
9eb07ab3f2d4 rhg: use persistent nodemap when available
Simon Sapin <simon-commits@exyr.org>
parents:
diff changeset
107 .file_name()
9eb07ab3f2d4 rhg: use persistent nodemap when available
Simon Sapin <simon-commits@exyr.org>
parents:
diff changeset
108 .expect("expected a base name")
9eb07ab3f2d4 rhg: use persistent nodemap when available
Simon Sapin <simon-commits@exyr.org>
parents:
diff changeset
109 .to_str()
9eb07ab3f2d4 rhg: use persistent nodemap when available
Simon Sapin <simon-commits@exyr.org>
parents:
diff changeset
110 .expect("expected an ASCII file name in the store");
9eb07ab3f2d4 rhg: use persistent nodemap when available
Simon Sapin <simon-commits@exyr.org>
parents:
diff changeset
111 let prefix = strip_suffix(docket_name, ".n.a")
9eb07ab3f2d4 rhg: use persistent nodemap when available
Simon Sapin <simon-commits@exyr.org>
parents:
diff changeset
112 .or_else(|| strip_suffix(docket_name, ".n"))
9eb07ab3f2d4 rhg: use persistent nodemap when available
Simon Sapin <simon-commits@exyr.org>
parents:
diff changeset
113 .expect("expected docket path in .n or .n.a");
9eb07ab3f2d4 rhg: use persistent nodemap when available
Simon Sapin <simon-commits@exyr.org>
parents:
diff changeset
114 let name = format!("{}-{}.nd", prefix, uid);
9eb07ab3f2d4 rhg: use persistent nodemap when available
Simon Sapin <simon-commits@exyr.org>
parents:
diff changeset
115 docket_path
9eb07ab3f2d4 rhg: use persistent nodemap when available
Simon Sapin <simon-commits@exyr.org>
parents:
diff changeset
116 .parent()
9eb07ab3f2d4 rhg: use persistent nodemap when available
Simon Sapin <simon-commits@exyr.org>
parents:
diff changeset
117 .expect("expected a non-root path")
9eb07ab3f2d4 rhg: use persistent nodemap when available
Simon Sapin <simon-commits@exyr.org>
parents:
diff changeset
118 .join(name)
9eb07ab3f2d4 rhg: use persistent nodemap when available
Simon Sapin <simon-commits@exyr.org>
parents:
diff changeset
119 }