mercurial-scm/hg-stable: rust/hg-core/src/revlog/compression.rs annotate

annotate rust/hg-core/src/revlog/compression.rs @ 52297:7be39c5110c9

hg-core: add a complete VFS This will be used from Python in a later change. More changes are needed in hg-core and rhg to properly clean up the APIs of the old VFS implementation but it can be done when the dust settles and we start adding more functionality to the pure Rust VFS.

author	Rapha?l Gom?s <rgomes@octobus.net>
date	Mon, 29 Jul 2024 20:47:43 +0200
parents	0744248cc541
children	f69a3f55fa9b

rev	line source
52284 0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	1 //! Helpers around revlog compression
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	2
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	3 use std::cell::RefCell;
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	4 use std::collections::HashSet;
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	5 use std::io::Read;
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	6
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	7 use flate2::bufread::ZlibEncoder;
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	8 use flate2::read::ZlibDecoder;
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	9
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	10 use crate::config::Config;
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	11 use crate::errors::HgError;
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	12 use crate::exit_codes;
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	13
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	14 use super::corrupted;
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	15 use super::RevlogError;
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	16
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	17 /// Header byte used to identify ZSTD-compressed data
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	18 pub const ZSTD_BYTE: u8 = b'\x28';
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	19 /// Header byte used to identify Zlib-compressed data
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	20 pub const ZLIB_BYTE: u8 = b'x';
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	21
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	22 const ZSTD_DEFAULT_LEVEL: u8 = 3;
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	23 const ZLIB_DEFAULT_LEVEL: u8 = 6;
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	24 /// The length of data below which we don't even try to compress it when using
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	25 /// Zstandard.
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	26 const MINIMUM_LENGTH_ZSTD: usize = 50;
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	27 /// The length of data below which we don't even try to compress it when using
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	28 /// Zlib.
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	29 const MINIMUM_LENGTH_ZLIB: usize = 44;
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	30
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	31 /// Defines the available compression engines and their options.
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	32 #[derive(Debug, Copy, Clone, PartialEq, Eq)]
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	33 pub enum CompressionConfig {
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	34 Zlib {
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	35 /// Between 0 and 9 included
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	36 level: u8,
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	37 },
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	38 Zstd {
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	39 /// Between 0 and 22 included
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	40 level: u8,
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	41 /// Never used in practice for now
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	42 threads: u8,
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	43 },
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	44 /// No compression is performed
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	45 None,
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	46 }
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	47
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	48 impl CompressionConfig {
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	49 pub fn new(
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	50 config: &Config,
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	51 requirements: &HashSet<String>,
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	52 ) -> Result<Self, HgError> {
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	53 let mut new = Self::default();
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	54
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	55 let zlib_level = config.get_u32(b"storage", b"revlog.zlib.level")?;
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	56 let zstd_level = config.get_u32(b"storage", b"revlog.zstd.level")?;
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	57
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	58 for requirement in requirements {
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	59 if requirement.starts_with("revlog-compression-")
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	60 \|\| requirement.starts_with("exp-compression-")
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	61 {
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	62 let split = &mut requirement.splitn(3, '-');
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	63 split.next();
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	64 split.next();
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	65 new = match split.next().unwrap() {
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	66 "zstd" => CompressionConfig::zstd(zstd_level)?,
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	67 e => {
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	68 return Err(HgError::UnsupportedFeature(format!(
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	69 "Unsupported compression engine '{e}'"
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	70 )))
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	71 }
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	72 };
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	73 }
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	74 }
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	75 if let Some(level) = zlib_level {
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	76 if matches!(new, CompressionConfig::Zlib { .. }) {
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	77 new.set_level(level as usize)?;
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	78 }
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	79 }
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	80 Ok(new)
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	81 }
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	82
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	83 /// Sets the level of the current compression engine
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	84 pub fn set_level(&mut self, new_level: usize) -> Result<(), HgError> {
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	85 match self {
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	86 CompressionConfig::Zlib { level } => {
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	87 if new_level > 9 {
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	88 return Err(HgError::abort(
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	89 format!(
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	90 "invalid compression zlib compression level {}, \
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	91 expected between 0 and 9 included",
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	92 new_level
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	93 ),
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	94 exit_codes::ABORT,
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	95 None,
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	96 ));
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	97 }
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	98 *level = new_level as u8;
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	99 }
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	100 CompressionConfig::Zstd { level, .. } => {
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	101 if new_level > 22 {
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	102 return Err(HgError::abort(
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	103 format!(
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	104 "invalid compression zstd compression level {}, \
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	105 expected between 0 and 22 included",
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	106 new_level
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	107 ),
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	108 exit_codes::ABORT,
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	109 None,
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	110 ));
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	111 }
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	112 *level = new_level as u8;
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	113 }
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	114 CompressionConfig::None => {}
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	115 }
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	116 Ok(())
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	117 }
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	118
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	119 /// Return a ZSTD compression config
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	120 pub fn zstd(
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	121 zstd_level: Option<u32>,
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	122 ) -> Result<CompressionConfig, HgError> {
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	123 let mut engine = CompressionConfig::Zstd {
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	124 level: ZSTD_DEFAULT_LEVEL,
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	125 threads: 0,
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	126 };
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	127 if let Some(level) = zstd_level {
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	128 engine.set_level(level as usize)?;
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	129 }
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	130 Ok(engine)
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	131 }
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	132 }
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	133
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	134 impl Default for CompressionConfig {
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	135 fn default() -> Self {
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	136 Self::Zlib {
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	137 level: ZLIB_DEFAULT_LEVEL,
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	138 }
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	139 }
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	140 }
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	141
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	142 /// A high-level trait to define compressors that should be able to compress
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	143 /// and decompress arbitrary bytes.
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	144 pub trait Compressor {
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	145 /// Returns a new [`Vec`] with the compressed data.
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	146 /// Should return `Ok(None)` if compression does not apply (e.g. too small)
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	147 fn compress(
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	148 &mut self,
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	149 data: &[u8],
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	150 ) -> Result<Option<Vec<u8>>, RevlogError>;
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	151 /// Returns a new [`Vec`] with the decompressed data.
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	152 fn decompress(&self, data: &[u8]) -> Result<Vec<u8>, RevlogError>;
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	153 }
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	154
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	155 /// A compressor that does nothing (useful in tests)
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	156 pub struct NoneCompressor;
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	157
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	158 impl Compressor for NoneCompressor {
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	159 fn compress(
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	160 &mut self,
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	161 _data: &[u8],
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	162 ) -> Result<Option<Vec<u8>>, RevlogError> {
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	163 Ok(None)
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	164 }
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	165
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	166 fn decompress(&self, data: &[u8]) -> Result<Vec<u8>, RevlogError> {
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	167 Ok(data.to_owned())
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	168 }
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	169 }
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	170
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	171 /// A compressor for Zstandard
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	172 pub struct ZstdCompressor {
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	173 /// Level of compression to use
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	174 level: u8,
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	175 /// How many threads are used (not implemented yet)
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	176 threads: u8,
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	177 /// The underlying zstd compressor
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	178 compressor: zstd::bulk::Compressor<'static>,
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	179 }
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	180
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	181 impl ZstdCompressor {
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	182 pub fn new(level: u8, threads: u8) -> Self {
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	183 Self {
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	184 level,
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	185 threads,
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	186 compressor: zstd::bulk::Compressor::new(level.into())
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	187 .expect("invalid zstd arguments"),
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	188 }
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	189 }
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	190 }
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	191
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	192 impl Compressor for ZstdCompressor {
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	193 fn compress(
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	194 &mut self,
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	195 data: &[u8],
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	196 ) -> Result<Option<Vec<u8>>, RevlogError> {
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	197 if self.threads != 0 {
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	198 // TODO use a zstd builder + zstd cargo feature to support this
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	199 unimplemented!("zstd parallel compression is not implemented");
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	200 }
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	201 if data.len() < MINIMUM_LENGTH_ZSTD {
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	202 return Ok(None);
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	203 }
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	204 let level = self.level as i32;
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	205 if data.len() <= 1000000 {
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	206 let compressed = self.compressor.compress(data).map_err(\|e\| {
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	207 corrupted(format!("revlog compress error: {}", e))
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	208 })?;
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	209 Ok(if compressed.len() < data.len() {
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	210 Some(compressed)
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	211 } else {
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	212 None
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	213 })
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	214 } else {
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	215 Ok(Some(zstd::stream::encode_all(data, level).map_err(
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	216 \|e\| corrupted(format!("revlog compress error: {}", e)),
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	217 )?))
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	218 }
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	219 }
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	220
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	221 fn decompress(&self, data: &[u8]) -> Result<Vec<u8>, RevlogError> {
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	222 zstd::stream::decode_all(data).map_err(\|e\| {
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	223 corrupted(format!("revlog decompress error: {}", e)).into()
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	224 })
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	225 }
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	226 }
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	227
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	228 /// A compressor for Zlib
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	229 pub struct ZlibCompressor {
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	230 /// Level of compression to use
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	231 level: flate2::Compression,
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	232 }
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	233
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	234 impl ZlibCompressor {
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	235 pub fn new(level: u8) -> Self {
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	236 Self {
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	237 level: flate2::Compression::new(level.into()),
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	238 }
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	239 }
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	240 }
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	241
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	242 impl Compressor for ZlibCompressor {
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	243 fn compress(
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	244 &mut self,
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	245 data: &[u8],
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	246 ) -> Result<Option<Vec<u8>>, RevlogError> {
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	247 assert!(!data.is_empty());
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	248 if data.len() < MINIMUM_LENGTH_ZLIB {
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	249 return Ok(None);
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	250 }
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	251 let mut buf = Vec::with_capacity(data.len());
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	252 ZlibEncoder::new(data, self.level)
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	253 .read_to_end(&mut buf)
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	254 .map_err(\|e\| corrupted(format!("revlog compress error: {}", e)))?;
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	255
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	256 Ok(if buf.len() < data.len() {
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	257 buf.shrink_to_fit();
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	258 Some(buf)
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	259 } else {
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	260 None
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	261 })
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	262 }
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	263
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	264 fn decompress(&self, data: &[u8]) -> Result<Vec<u8>, RevlogError> {
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	265 let mut decoder = ZlibDecoder::new(data);
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	266 // TODO reuse the allocation somehow?
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	267 let mut buf = vec![];
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	268 decoder.read_to_end(&mut buf).map_err(\|e\| {
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	269 corrupted(format!("revlog decompress error: {}", e))
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	270 })?;
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	271 Ok(buf)
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	272 }
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	273 }
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	274
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	275 thread_local! {
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	276 // seems fine to [unwrap] here: this can only fail due to memory allocation
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	277 // failing, and it's normal for that to cause panic.
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	278 static ZSTD_DECODER : RefCell<zstd::bulk::Decompressor<'static>> =
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	279 RefCell::new(zstd::bulk::Decompressor::new().ok().unwrap());
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	280 }
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	281
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	282 /// Util to wrap the reuse of a zstd decoder while controlling its buffer size.
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	283 fn zstd_decompress_to_buffer(
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	284 bytes: &[u8],
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	285 buf: &mut Vec<u8>,
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	286 ) -> Result<usize, std::io::Error> {
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	287 ZSTD_DECODER
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	288 .with(\|decoder\| decoder.borrow_mut().decompress_to_buffer(bytes, buf))
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	289 }
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	290
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	291 /// Specialized revlog decompression to use less memory for deltas while
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	292 /// keeping performance acceptable.
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	293 pub(super) fn uncompressed_zstd_data(
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	294 bytes: &[u8],
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	295 is_delta: bool,
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	296 uncompressed_len: i32,
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	297 ) -> Result<Vec<u8>, HgError> {
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	298 let cap = uncompressed_len.max(0) as usize;
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	299 if is_delta {
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	300 // [cap] is usually an over-estimate of the space needed because
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	301 // it's the length of delta-decoded data, but we're interested
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	302 // in the size of the delta.
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	303 // This means we have to [shrink_to_fit] to avoid holding on
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	304 // to a large chunk of memory, but it also means we must have a
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	305 // fallback branch, for the case when the delta is longer than
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	306 // the original data (surprisingly, this does happen in practice)
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	307 let mut buf = Vec::with_capacity(cap);
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	308 match zstd_decompress_to_buffer(bytes, &mut buf) {
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	309 Ok(_) => buf.shrink_to_fit(),
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	310 Err(_) => {
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	311 buf.clear();
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	312 zstd::stream::copy_decode(bytes, &mut buf)
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	313 .map_err(\|e\| corrupted(e.to_string()))?;
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	314 }
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	315 };
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	316 Ok(buf)
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	317 } else {
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	318 let mut buf = Vec::with_capacity(cap);
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	319 let len = zstd_decompress_to_buffer(bytes, &mut buf)
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	320 .map_err(\|e\| corrupted(e.to_string()))?;
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	321 if len != uncompressed_len as usize {
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	322 Err(corrupted("uncompressed length does not match"))
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	323 } else {
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	324 Ok(buf)
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	325 }
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	326 }
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	327 }
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	328
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	329 #[cfg(test)]
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	330 mod tests {
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	331 use super::*;
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	332
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	333 const LARGE_TEXT: &[u8] = b"
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	334 Patali Dirapata, Cromda Cromda Ripalo, Pata Pata, Ko Ko Ko
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	335 Bokoro Dipoulito, Rondi Rondi Pepino, Pata Pata, Ko Ko Ko
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	336 Emana Karassoli, Loucra Loucra Nonponto, Pata Pata, Ko Ko Ko.";
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	337
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	338 #[test]
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	339 fn test_zlib_compressor() {
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	340 // Can return `Ok(None)`
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	341 let mut compressor = ZlibCompressor::new(1);
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	342 assert_eq!(compressor.compress(b"too small").unwrap(), None);
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	343
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	344 // Compression returns bytes
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	345 let compressed_with_1 =
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	346 compressor.compress(LARGE_TEXT).unwrap().unwrap();
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	347 assert!(compressed_with_1.len() < LARGE_TEXT.len());
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	348 // Round trip works
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	349 assert_eq!(
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	350 compressor.decompress(&compressed_with_1).unwrap(),
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	351 LARGE_TEXT
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	352 );
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	353
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	354 // Compression levels mean something
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	355 let mut compressor = ZlibCompressor::new(9);
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	356 // Compression returns bytes
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	357 let compressed = compressor.compress(LARGE_TEXT).unwrap().unwrap();
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	358 assert!(compressed.len() < compressed_with_1.len());
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	359 }
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	360
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	361 #[test]
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	362 fn test_zstd_compressor() {
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	363 // Can return `Ok(None)`
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	364 let mut compressor = ZstdCompressor::new(1, 0);
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	365 assert_eq!(compressor.compress(b"too small").unwrap(), None);
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	366
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	367 // Compression returns bytes
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	368 let compressed_with_1 =
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	369 compressor.compress(LARGE_TEXT).unwrap().unwrap();
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	370 assert!(compressed_with_1.len() < LARGE_TEXT.len());
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	371 // Round trip works
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	372 assert_eq!(
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	373 compressor.decompress(&compressed_with_1).unwrap(),
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	374 LARGE_TEXT
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	375 );
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	376
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	377 // Compression levels mean something
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	378 let mut compressor = ZstdCompressor::new(22, 0);
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	379 // Compression returns bytes
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	380 let compressed = compressor.compress(LARGE_TEXT).unwrap().unwrap();
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	381 assert!(compressed.len() < compressed_with_1.len());
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	382 }
0744248cc541 rust-revlog: add compression helpers Rapha?l Gom?s <rgomes@octobus.net> parents: diff changeset	383 }

Mercurial > public > mercurial-scm > hg-stable

annotate rust/hg-core/src/revlog/compression.rs @ 52297:7be39c5110c9