mercurial-scm/hg-stable: contrib/python-zstandard/zstandard/cffi.py annotate

annotate contrib/python-zstandard/zstandard/cffi.py @ 53040:cdd7bf612c7b stable tip

bundle-spec: properly format boolean parameter (issue6960) This was breaking automatic clone bundle generation. This changeset fixes it and add a test to catch it in the future.

author	Pierre-Yves David <pierre-yves.david@octobus.net>
date	Tue, 11 Mar 2025 02:29:42 +0100
parents	9bd6854aab86
children

rev	line source
30444 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1 # Copyright (c) 2016-present, Gregory Szorc
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	2 # All rights reserved.
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	3 #
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	4 # This software may be modified and distributed under the terms
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	5 # of the BSD license. See the LICENSE file for details.
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	6
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	7 """Python interface to the Zstandard (zstd) compression library."""
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	8
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	9
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	10 # This should match what the C extension exports.
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	11 __all__ = [
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	12 #'BufferSegment',
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	13 #'BufferSegments',
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	14 #'BufferWithSegments',
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	15 #'BufferWithSegmentsCollection',
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	16 "CompressionParameters",
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	17 "ZstdCompressionDict",
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	18 "ZstdCompressionParameters",
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	19 "ZstdCompressor",
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	20 "ZstdError",
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	21 "ZstdDecompressor",
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	22 "FrameParameters",
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	23 "estimate_decompression_context_size",
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	24 "frame_content_size",
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	25 "frame_header_size",
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	26 "get_frame_parameters",
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	27 "train_dictionary",
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	28 # Constants.
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	29 "FLUSH_BLOCK",
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	30 "FLUSH_FRAME",
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	31 "COMPRESSOBJ_FLUSH_FINISH",
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	32 "COMPRESSOBJ_FLUSH_BLOCK",
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	33 "ZSTD_VERSION",
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	34 "FRAME_HEADER",
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	35 "CONTENTSIZE_UNKNOWN",
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	36 "CONTENTSIZE_ERROR",
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	37 "MAX_COMPRESSION_LEVEL",
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	38 "COMPRESSION_RECOMMENDED_INPUT_SIZE",
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	39 "COMPRESSION_RECOMMENDED_OUTPUT_SIZE",
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	40 "DECOMPRESSION_RECOMMENDED_INPUT_SIZE",
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	41 "DECOMPRESSION_RECOMMENDED_OUTPUT_SIZE",
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	42 "MAGIC_NUMBER",
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	43 "BLOCKSIZELOG_MAX",
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	44 "BLOCKSIZE_MAX",
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	45 "WINDOWLOG_MIN",
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	46 "WINDOWLOG_MAX",
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	47 "CHAINLOG_MIN",
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	48 "CHAINLOG_MAX",
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	49 "HASHLOG_MIN",
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	50 "HASHLOG_MAX",
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	51 "HASHLOG3_MAX",
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	52 "MINMATCH_MIN",
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	53 "MINMATCH_MAX",
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	54 "SEARCHLOG_MIN",
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	55 "SEARCHLOG_MAX",
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	56 "SEARCHLENGTH_MIN",
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	57 "SEARCHLENGTH_MAX",
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	58 "TARGETLENGTH_MIN",
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	59 "TARGETLENGTH_MAX",
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	60 "LDM_MINMATCH_MIN",
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	61 "LDM_MINMATCH_MAX",
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	62 "LDM_BUCKETSIZELOG_MAX",
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	63 "STRATEGY_FAST",
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	64 "STRATEGY_DFAST",
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	65 "STRATEGY_GREEDY",
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	66 "STRATEGY_LAZY",
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	67 "STRATEGY_LAZY2",
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	68 "STRATEGY_BTLAZY2",
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	69 "STRATEGY_BTOPT",
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	70 "STRATEGY_BTULTRA",
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	71 "STRATEGY_BTULTRA2",
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	72 "DICT_TYPE_AUTO",
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	73 "DICT_TYPE_RAWCONTENT",
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	74 "DICT_TYPE_FULLDICT",
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	75 "FORMAT_ZSTD1",
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	76 "FORMAT_ZSTD1_MAGICLESS",
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	77 ]
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	78
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	79 import io
31799 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30924 diff changeset	80 import os
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	81 import sys
30444 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	82
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	83 from _zstd_cffi import (
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	84 ffi,
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	85 lib,
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	86 )
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	87
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	88 if sys.version_info[0] == 2:
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	89 bytes_type = str
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	90 int_type = long
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	91 else:
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	92 bytes_type = bytes
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	93 int_type = int
30444 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	94
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	95
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	96 COMPRESSION_RECOMMENDED_INPUT_SIZE = lib.ZSTD_CStreamInSize()
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	97 COMPRESSION_RECOMMENDED_OUTPUT_SIZE = lib.ZSTD_CStreamOutSize()
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	98 DECOMPRESSION_RECOMMENDED_INPUT_SIZE = lib.ZSTD_DStreamInSize()
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	99 DECOMPRESSION_RECOMMENDED_OUTPUT_SIZE = lib.ZSTD_DStreamOutSize()
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	100
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	101 new_nonzero = ffi.new_allocator(should_clear_after_alloc=False)
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	102
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	103
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	104 MAX_COMPRESSION_LEVEL = lib.ZSTD_maxCLevel()
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	105 MAGIC_NUMBER = lib.ZSTD_MAGICNUMBER
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	106 FRAME_HEADER = b"\x28\xb5\x2f\xfd"
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	107 CONTENTSIZE_UNKNOWN = lib.ZSTD_CONTENTSIZE_UNKNOWN
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	108 CONTENTSIZE_ERROR = lib.ZSTD_CONTENTSIZE_ERROR
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	109 ZSTD_VERSION = (
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	110 lib.ZSTD_VERSION_MAJOR,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	111 lib.ZSTD_VERSION_MINOR,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	112 lib.ZSTD_VERSION_RELEASE,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	113 )
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	114
40122 73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	115 BLOCKSIZELOG_MAX = lib.ZSTD_BLOCKSIZELOG_MAX
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	116 BLOCKSIZE_MAX = lib.ZSTD_BLOCKSIZE_MAX
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	117 WINDOWLOG_MIN = lib.ZSTD_WINDOWLOG_MIN
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	118 WINDOWLOG_MAX = lib.ZSTD_WINDOWLOG_MAX
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	119 CHAINLOG_MIN = lib.ZSTD_CHAINLOG_MIN
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	120 CHAINLOG_MAX = lib.ZSTD_CHAINLOG_MAX
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	121 HASHLOG_MIN = lib.ZSTD_HASHLOG_MIN
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	122 HASHLOG_MAX = lib.ZSTD_HASHLOG_MAX
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	123 HASHLOG3_MAX = lib.ZSTD_HASHLOG3_MAX
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	124 MINMATCH_MIN = lib.ZSTD_MINMATCH_MIN
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	125 MINMATCH_MAX = lib.ZSTD_MINMATCH_MAX
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	126 SEARCHLOG_MIN = lib.ZSTD_SEARCHLOG_MIN
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	127 SEARCHLOG_MAX = lib.ZSTD_SEARCHLOG_MAX
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	128 SEARCHLENGTH_MIN = lib.ZSTD_MINMATCH_MIN
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	129 SEARCHLENGTH_MAX = lib.ZSTD_MINMATCH_MAX
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	130 TARGETLENGTH_MIN = lib.ZSTD_TARGETLENGTH_MIN
40122 73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	131 TARGETLENGTH_MAX = lib.ZSTD_TARGETLENGTH_MAX
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	132 LDM_MINMATCH_MIN = lib.ZSTD_LDM_MINMATCH_MIN
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	133 LDM_MINMATCH_MAX = lib.ZSTD_LDM_MINMATCH_MAX
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	134 LDM_BUCKETSIZELOG_MAX = lib.ZSTD_LDM_BUCKETSIZELOG_MAX
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	135
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	136 STRATEGY_FAST = lib.ZSTD_fast
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	137 STRATEGY_DFAST = lib.ZSTD_dfast
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	138 STRATEGY_GREEDY = lib.ZSTD_greedy
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	139 STRATEGY_LAZY = lib.ZSTD_lazy
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	140 STRATEGY_LAZY2 = lib.ZSTD_lazy2
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	141 STRATEGY_BTLAZY2 = lib.ZSTD_btlazy2
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	142 STRATEGY_BTOPT = lib.ZSTD_btopt
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	143 STRATEGY_BTULTRA = lib.ZSTD_btultra
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	144 STRATEGY_BTULTRA2 = lib.ZSTD_btultra2
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	145
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	146 DICT_TYPE_AUTO = lib.ZSTD_dct_auto
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	147 DICT_TYPE_RAWCONTENT = lib.ZSTD_dct_rawContent
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	148 DICT_TYPE_FULLDICT = lib.ZSTD_dct_fullDict
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	149
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	150 FORMAT_ZSTD1 = lib.ZSTD_f_zstd1
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	151 FORMAT_ZSTD1_MAGICLESS = lib.ZSTD_f_zstd1_magicless
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	152
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	153 FLUSH_BLOCK = 0
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	154 FLUSH_FRAME = 1
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	155
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	156 COMPRESSOBJ_FLUSH_FINISH = 0
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	157 COMPRESSOBJ_FLUSH_BLOCK = 1
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	158
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	159
31799 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30924 diff changeset	160 def _cpu_count():
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30924 diff changeset	161 # os.cpu_count() was introducd in Python 3.4.
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30924 diff changeset	162 try:
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30924 diff changeset	163 return os.cpu_count() or 0
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30924 diff changeset	164 except AttributeError:
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30924 diff changeset	165 pass
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30924 diff changeset	166
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30924 diff changeset	167 # Linux.
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30924 diff changeset	168 try:
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30924 diff changeset	169 if sys.version_info[0] == 2:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	170 return os.sysconf(b"SC_NPROCESSORS_ONLN")
31799 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30924 diff changeset	171 else:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	172 return os.sysconf("SC_NPROCESSORS_ONLN")
31799 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30924 diff changeset	173 except (AttributeError, ValueError):
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30924 diff changeset	174 pass
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30924 diff changeset	175
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30924 diff changeset	176 # TODO implement on other platforms.
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30924 diff changeset	177 return 0
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30924 diff changeset	178
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30924 diff changeset	179
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	180 class ZstdError(Exception):
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	181 pass
30444 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	182
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	183
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	184 def _zstd_error(zresult):
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	185 # Resolves to bytes on Python 2 and 3. We use the string for formatting
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	186 # into error messages, which will be literal unicode. So convert it to
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	187 # unicode.
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	188 return ffi.string(lib.ZSTD_getErrorName(zresult)).decode("utf-8")
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	189
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	190
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	191 def _make_cctx_params(params):
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	192 res = lib.ZSTD_createCCtxParams()
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	193 if res == ffi.NULL:
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	194 raise MemoryError()
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	195
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	196 res = ffi.gc(res, lib.ZSTD_freeCCtxParams)
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	197
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	198 attrs = [
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	199 (lib.ZSTD_c_format, params.format),
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	200 (lib.ZSTD_c_compressionLevel, params.compression_level),
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	201 (lib.ZSTD_c_windowLog, params.window_log),
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	202 (lib.ZSTD_c_hashLog, params.hash_log),
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	203 (lib.ZSTD_c_chainLog, params.chain_log),
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	204 (lib.ZSTD_c_searchLog, params.search_log),
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	205 (lib.ZSTD_c_minMatch, params.min_match),
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	206 (lib.ZSTD_c_targetLength, params.target_length),
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	207 (lib.ZSTD_c_strategy, params.compression_strategy),
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	208 (lib.ZSTD_c_contentSizeFlag, params.write_content_size),
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	209 (lib.ZSTD_c_checksumFlag, params.write_checksum),
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	210 (lib.ZSTD_c_dictIDFlag, params.write_dict_id),
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	211 (lib.ZSTD_c_nbWorkers, params.threads),
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	212 (lib.ZSTD_c_jobSize, params.job_size),
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	213 (lib.ZSTD_c_overlapLog, params.overlap_log),
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	214 (lib.ZSTD_c_forceMaxWindow, params.force_max_window),
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	215 (lib.ZSTD_c_enableLongDistanceMatching, params.enable_ldm),
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	216 (lib.ZSTD_c_ldmHashLog, params.ldm_hash_log),
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	217 (lib.ZSTD_c_ldmMinMatch, params.ldm_min_match),
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	218 (lib.ZSTD_c_ldmBucketSizeLog, params.ldm_bucket_size_log),
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	219 (lib.ZSTD_c_ldmHashRateLog, params.ldm_hash_rate_log),
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	220 ]
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	221
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	222 for param, value in attrs:
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	223 _set_compression_parameter(res, param, value)
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	224
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	225 return res
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	226
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	227
52666 9bd6854aab86 pyupgrade: convert to new style classes Matt Harbison <matt_harbison@yahoo.com> parents: 51762 diff changeset	228 class ZstdCompressionParameters:
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	229 @staticmethod
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	230 def from_level(level, source_size=0, dict_size=0, **kwargs):
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	231 params = lib.ZSTD_getCParams(level, source_size, dict_size)
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	232
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	233 args = {
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	234 "window_log": "windowLog",
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	235 "chain_log": "chainLog",
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	236 "hash_log": "hashLog",
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	237 "search_log": "searchLog",
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	238 "min_match": "minMatch",
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	239 "target_length": "targetLength",
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	240 "compression_strategy": "strategy",
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	241 }
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	242
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	243 for arg, attr in args.items():
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	244 if arg not in kwargs:
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	245 kwargs[arg] = getattr(params, attr)
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	246
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	247 return ZstdCompressionParameters(**kwargs)
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	248
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	249 def __init__(
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	250 self,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	251 format=0,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	252 compression_level=0,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	253 window_log=0,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	254 hash_log=0,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	255 chain_log=0,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	256 search_log=0,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	257 min_match=0,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	258 target_length=0,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	259 strategy=-1,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	260 compression_strategy=-1,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	261 write_content_size=1,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	262 write_checksum=0,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	263 write_dict_id=0,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	264 job_size=0,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	265 overlap_log=-1,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	266 overlap_size_log=-1,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	267 force_max_window=0,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	268 enable_ldm=0,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	269 ldm_hash_log=0,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	270 ldm_min_match=0,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	271 ldm_bucket_size_log=0,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	272 ldm_hash_rate_log=-1,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	273 ldm_hash_every_log=-1,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	274 threads=0,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	275 ):
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	276 params = lib.ZSTD_createCCtxParams()
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	277 if params == ffi.NULL:
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	278 raise MemoryError()
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	279
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	280 params = ffi.gc(params, lib.ZSTD_freeCCtxParams)
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	281
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	282 self._params = params
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	283
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	284 if threads < 0:
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	285 threads = _cpu_count()
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	286
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	287 # We need to set ZSTD_c_nbWorkers before ZSTD_c_jobSize and ZSTD_c_overlapLog
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	288 # because setting ZSTD_c_nbWorkers resets the other parameters.
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	289 _set_compression_parameter(params, lib.ZSTD_c_nbWorkers, threads)
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	290
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	291 _set_compression_parameter(params, lib.ZSTD_c_format, format)
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	292 _set_compression_parameter(
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	293 params, lib.ZSTD_c_compressionLevel, compression_level
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	294 )
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	295 _set_compression_parameter(params, lib.ZSTD_c_windowLog, window_log)
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	296 _set_compression_parameter(params, lib.ZSTD_c_hashLog, hash_log)
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	297 _set_compression_parameter(params, lib.ZSTD_c_chainLog, chain_log)
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	298 _set_compression_parameter(params, lib.ZSTD_c_searchLog, search_log)
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	299 _set_compression_parameter(params, lib.ZSTD_c_minMatch, min_match)
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	300 _set_compression_parameter(
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	301 params, lib.ZSTD_c_targetLength, target_length
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	302 )
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	303
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	304 if strategy != -1 and compression_strategy != -1:
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	305 raise ValueError(
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	306 "cannot specify both compression_strategy and strategy"
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	307 )
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	308
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	309 if compression_strategy != -1:
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	310 strategy = compression_strategy
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	311 elif strategy == -1:
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	312 strategy = 0
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	313
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	314 _set_compression_parameter(params, lib.ZSTD_c_strategy, strategy)
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	315 _set_compression_parameter(
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	316 params, lib.ZSTD_c_contentSizeFlag, write_content_size
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	317 )
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	318 _set_compression_parameter(
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	319 params, lib.ZSTD_c_checksumFlag, write_checksum
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	320 )
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	321 _set_compression_parameter(params, lib.ZSTD_c_dictIDFlag, write_dict_id)
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	322 _set_compression_parameter(params, lib.ZSTD_c_jobSize, job_size)
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	323
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	324 if overlap_log != -1 and overlap_size_log != -1:
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	325 raise ValueError(
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	326 "cannot specify both overlap_log and overlap_size_log"
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	327 )
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	328
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	329 if overlap_size_log != -1:
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	330 overlap_log = overlap_size_log
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	331 elif overlap_log == -1:
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	332 overlap_log = 0
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	333
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	334 _set_compression_parameter(params, lib.ZSTD_c_overlapLog, overlap_log)
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	335 _set_compression_parameter(
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	336 params, lib.ZSTD_c_forceMaxWindow, force_max_window
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	337 )
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	338 _set_compression_parameter(
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	339 params, lib.ZSTD_c_enableLongDistanceMatching, enable_ldm
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	340 )
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	341 _set_compression_parameter(params, lib.ZSTD_c_ldmHashLog, ldm_hash_log)
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	342 _set_compression_parameter(
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	343 params, lib.ZSTD_c_ldmMinMatch, ldm_min_match
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	344 )
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	345 _set_compression_parameter(
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	346 params, lib.ZSTD_c_ldmBucketSizeLog, ldm_bucket_size_log
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	347 )
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	348
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	349 if ldm_hash_rate_log != -1 and ldm_hash_every_log != -1:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	350 raise ValueError(
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	351 "cannot specify both ldm_hash_rate_log and ldm_hash_every_log"
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	352 )
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	353
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	354 if ldm_hash_every_log != -1:
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	355 ldm_hash_rate_log = ldm_hash_every_log
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	356 elif ldm_hash_rate_log == -1:
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	357 ldm_hash_rate_log = 0
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	358
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	359 _set_compression_parameter(
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	360 params, lib.ZSTD_c_ldmHashRateLog, ldm_hash_rate_log
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	361 )
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	362
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	363 @property
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	364 def format(self):
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	365 return _get_compression_parameter(self._params, lib.ZSTD_c_format)
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	366
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	367 @property
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	368 def compression_level(self):
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	369 return _get_compression_parameter(
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	370 self._params, lib.ZSTD_c_compressionLevel
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	371 )
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	372
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	373 @property
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	374 def window_log(self):
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	375 return _get_compression_parameter(self._params, lib.ZSTD_c_windowLog)
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	376
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	377 @property
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	378 def hash_log(self):
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	379 return _get_compression_parameter(self._params, lib.ZSTD_c_hashLog)
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	380
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	381 @property
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	382 def chain_log(self):
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	383 return _get_compression_parameter(self._params, lib.ZSTD_c_chainLog)
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	384
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	385 @property
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	386 def search_log(self):
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	387 return _get_compression_parameter(self._params, lib.ZSTD_c_searchLog)
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	388
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	389 @property
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	390 def min_match(self):
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	391 return _get_compression_parameter(self._params, lib.ZSTD_c_minMatch)
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	392
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	393 @property
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	394 def target_length(self):
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	395 return _get_compression_parameter(self._params, lib.ZSTD_c_targetLength)
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	396
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	397 @property
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	398 def compression_strategy(self):
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	399 return _get_compression_parameter(self._params, lib.ZSTD_c_strategy)
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	400
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	401 @property
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	402 def write_content_size(self):
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	403 return _get_compression_parameter(
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	404 self._params, lib.ZSTD_c_contentSizeFlag
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	405 )
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	406
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	407 @property
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	408 def write_checksum(self):
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	409 return _get_compression_parameter(self._params, lib.ZSTD_c_checksumFlag)
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	410
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	411 @property
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	412 def write_dict_id(self):
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	413 return _get_compression_parameter(self._params, lib.ZSTD_c_dictIDFlag)
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	414
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	415 @property
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	416 def job_size(self):
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	417 return _get_compression_parameter(self._params, lib.ZSTD_c_jobSize)
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	418
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	419 @property
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	420 def overlap_log(self):
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	421 return _get_compression_parameter(self._params, lib.ZSTD_c_overlapLog)
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	422
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	423 @property
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	424 def overlap_size_log(self):
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	425 return self.overlap_log
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	426
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	427 @property
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	428 def force_max_window(self):
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	429 return _get_compression_parameter(
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	430 self._params, lib.ZSTD_c_forceMaxWindow
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	431 )
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	432
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	433 @property
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	434 def enable_ldm(self):
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	435 return _get_compression_parameter(
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	436 self._params, lib.ZSTD_c_enableLongDistanceMatching
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	437 )
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	438
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	439 @property
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	440 def ldm_hash_log(self):
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	441 return _get_compression_parameter(self._params, lib.ZSTD_c_ldmHashLog)
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	442
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	443 @property
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	444 def ldm_min_match(self):
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	445 return _get_compression_parameter(self._params, lib.ZSTD_c_ldmMinMatch)
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	446
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	447 @property
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	448 def ldm_bucket_size_log(self):
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	449 return _get_compression_parameter(
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	450 self._params, lib.ZSTD_c_ldmBucketSizeLog
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	451 )
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	452
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	453 @property
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	454 def ldm_hash_rate_log(self):
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	455 return _get_compression_parameter(
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	456 self._params, lib.ZSTD_c_ldmHashRateLog
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	457 )
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	458
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	459 @property
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	460 def ldm_hash_every_log(self):
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	461 return self.ldm_hash_rate_log
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	462
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	463 @property
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	464 def threads(self):
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	465 return _get_compression_parameter(self._params, lib.ZSTD_c_nbWorkers)
31799 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30924 diff changeset	466
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30924 diff changeset	467 def estimated_compression_context_size(self):
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	468 return lib.ZSTD_estimateCCtxSize_usingCCtxParams(self._params)
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	469
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	470
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	471 CompressionParameters = ZstdCompressionParameters
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	472
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	473
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	474 def estimate_decompression_context_size():
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	475 return lib.ZSTD_estimateDCtxSize()
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	476
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	477
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	478 def _set_compression_parameter(params, param, value):
42941 69de49c4e39c zstandard: vendor python-zstandard 0.12 Gregory Szorc <gregory.szorc@gmail.com> parents: 42070 diff changeset	479 zresult = lib.ZSTD_CCtxParams_setParameter(params, param, value)
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	480 if lib.ZSTD_isError(zresult):
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	481 raise ZstdError(
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	482 "unable to set compression context parameter: %s"
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	483 % _zstd_error(zresult)
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	484 )
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	485
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	486
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	487 def _get_compression_parameter(params, param):
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	488 result = ffi.new("int *")
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	489
42941 69de49c4e39c zstandard: vendor python-zstandard 0.12 Gregory Szorc <gregory.szorc@gmail.com> parents: 42070 diff changeset	490 zresult = lib.ZSTD_CCtxParams_getParameter(params, param, result)
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	491 if lib.ZSTD_isError(zresult):
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	492 raise ZstdError(
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	493 "unable to get compression context parameter: %s"
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	494 % _zstd_error(zresult)
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	495 )
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	496
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	497 return result[0]
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	498
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	499
52666 9bd6854aab86 pyupgrade: convert to new style classes Matt Harbison <matt_harbison@yahoo.com> parents: 51762 diff changeset	500 class ZstdCompressionWriter:
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	501 def __init__(
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	502 self, compressor, writer, source_size, write_size, write_return_read
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	503 ):
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	504 self._compressor = compressor
30444 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	505 self._writer = writer
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	506 self._write_size = write_size
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	507 self._write_return_read = bool(write_return_read)
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	508 self._entered = False
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	509 self._closed = False
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	510 self._bytes_compressed = 0
30444 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	511
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	512 self._dst_buffer = ffi.new("char[]", write_size)
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	513 self._out_buffer = ffi.new("ZSTD_outBuffer *")
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	514 self._out_buffer.dst = self._dst_buffer
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	515 self._out_buffer.size = len(self._dst_buffer)
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	516 self._out_buffer.pos = 0
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	517
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	518 zresult = lib.ZSTD_CCtx_setPledgedSrcSize(compressor._cctx, source_size)
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	519 if lib.ZSTD_isError(zresult):
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	520 raise ZstdError(
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	521 "error setting source size: %s" % _zstd_error(zresult)
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	522 )
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	523
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	524 def __enter__(self):
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	525 if self._closed:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	526 raise ValueError("stream is closed")
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	527
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	528 if self._entered:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	529 raise ZstdError("cannot __enter__ multiple times")
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	530
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	531 self._entered = True
30444 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	532 return self
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	533
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	534 def __exit__(self, exc_type, exc_value, exc_tb):
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	535 self._entered = False
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	536
30444 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	537 if not exc_type and not exc_value and not exc_tb:
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	538 self.close()
30444 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	539
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	540 self._compressor = None
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	541
30444 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	542 return False
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	543
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	544 def memory_size(self):
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	545 return lib.ZSTD_sizeof_CCtx(self._compressor._cctx)
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	546
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	547 def fileno(self):
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	548 f = getattr(self._writer, "fileno", None)
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	549 if f:
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	550 return f()
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	551 else:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	552 raise OSError("fileno not available on underlying writer")
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	553
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	554 def close(self):
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	555 if self._closed:
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	556 return
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	557
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	558 try:
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	559 self.flush(FLUSH_FRAME)
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	560 finally:
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	561 self._closed = True
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	562
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	563 # Call close() on underlying stream as well.
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	564 f = getattr(self._writer, "close", None)
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	565 if f:
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	566 f()
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	567
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	568 @property
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	569 def closed(self):
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	570 return self._closed
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	571
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	572 def isatty(self):
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	573 return False
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	574
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	575 def readable(self):
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	576 return False
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	577
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	578 def readline(self, size=-1):
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	579 raise io.UnsupportedOperation()
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	580
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	581 def readlines(self, hint=-1):
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	582 raise io.UnsupportedOperation()
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	583
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	584 def seek(self, offset, whence=None):
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	585 raise io.UnsupportedOperation()
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	586
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	587 def seekable(self):
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	588 return False
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	589
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	590 def truncate(self, size=None):
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	591 raise io.UnsupportedOperation()
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	592
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	593 def writable(self):
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	594 return True
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	595
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	596 def writelines(self, lines):
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	597 raise NotImplementedError("writelines() is not yet implemented")
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	598
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	599 def read(self, size=-1):
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	600 raise io.UnsupportedOperation()
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	601
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	602 def readall(self):
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	603 raise io.UnsupportedOperation()
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	604
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	605 def readinto(self, b):
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	606 raise io.UnsupportedOperation()
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	607
30444 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	608 def write(self, data):
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	609 if self._closed:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	610 raise ValueError("stream is closed")
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	611
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	612 total_write = 0
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	613
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	614 data_buffer = ffi.from_buffer(data)
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	615
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	616 in_buffer = ffi.new("ZSTD_inBuffer *")
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	617 in_buffer.src = data_buffer
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	618 in_buffer.size = len(data_buffer)
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	619 in_buffer.pos = 0
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	620
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	621 out_buffer = self._out_buffer
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	622 out_buffer.pos = 0
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	623
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	624 while in_buffer.pos < in_buffer.size:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	625 zresult = lib.ZSTD_compressStream2(
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	626 self._compressor._cctx,
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	627 out_buffer,
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	628 in_buffer,
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	629 lib.ZSTD_e_continue,
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	630 )
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	631 if lib.ZSTD_isError(zresult):
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	632 raise ZstdError(
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	633 "zstd compress error: %s" % _zstd_error(zresult)
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	634 )
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	635
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	636 if out_buffer.pos:
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	637 self._writer.write(
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	638 ffi.buffer(out_buffer.dst, out_buffer.pos)[:]
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	639 )
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	640 total_write += out_buffer.pos
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	641 self._bytes_compressed += out_buffer.pos
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	642 out_buffer.pos = 0
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	643
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	644 if self._write_return_read:
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	645 return in_buffer.pos
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	646 else:
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	647 return total_write
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	648
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	649 def flush(self, flush_mode=FLUSH_BLOCK):
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	650 if flush_mode == FLUSH_BLOCK:
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	651 flush = lib.ZSTD_e_flush
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	652 elif flush_mode == FLUSH_FRAME:
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	653 flush = lib.ZSTD_e_end
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	654 else:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	655 raise ValueError("unknown flush_mode: %r" % flush_mode)
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	656
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	657 if self._closed:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	658 raise ValueError("stream is closed")
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	659
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	660 total_write = 0
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	661
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	662 out_buffer = self._out_buffer
30444 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	663 out_buffer.pos = 0
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	664
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	665 in_buffer = ffi.new("ZSTD_inBuffer *")
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	666 in_buffer.src = ffi.NULL
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	667 in_buffer.size = 0
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	668 in_buffer.pos = 0
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	669
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	670 while True:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	671 zresult = lib.ZSTD_compressStream2(
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	672 self._compressor._cctx, out_buffer, in_buffer, flush
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	673 )
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	674 if lib.ZSTD_isError(zresult):
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	675 raise ZstdError(
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	676 "zstd compress error: %s" % _zstd_error(zresult)
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	677 )
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	678
40122 73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	679 if out_buffer.pos:
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	680 self._writer.write(
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	681 ffi.buffer(out_buffer.dst, out_buffer.pos)[:]
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	682 )
40122 73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	683 total_write += out_buffer.pos
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	684 self._bytes_compressed += out_buffer.pos
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	685 out_buffer.pos = 0
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	686
40122 73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	687 if not zresult:
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	688 break
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	689
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	690 return total_write
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	691
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	692 def tell(self):
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	693 return self._bytes_compressed
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	694
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	695
52666 9bd6854aab86 pyupgrade: convert to new style classes Matt Harbison <matt_harbison@yahoo.com> parents: 51762 diff changeset	696 class ZstdCompressionObj:
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	697 def compress(self, data):
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	698 if self._finished:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	699 raise ZstdError("cannot call compress() after compressor finished")
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	700
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	701 data_buffer = ffi.from_buffer(data)
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	702 source = ffi.new("ZSTD_inBuffer *")
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	703 source.src = data_buffer
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	704 source.size = len(data_buffer)
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	705 source.pos = 0
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	706
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	707 chunks = []
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	708
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	709 while source.pos < len(data):
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	710 zresult = lib.ZSTD_compressStream2(
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	711 self._compressor._cctx, self._out, source, lib.ZSTD_e_continue
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	712 )
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	713 if lib.ZSTD_isError(zresult):
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	714 raise ZstdError(
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	715 "zstd compress error: %s" % _zstd_error(zresult)
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	716 )
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	717
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	718 if self._out.pos:
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	719 chunks.append(ffi.buffer(self._out.dst, self._out.pos)[:])
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	720 self._out.pos = 0
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	721
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	722 return b"".join(chunks)
30444 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	723
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	724 def flush(self, flush_mode=COMPRESSOBJ_FLUSH_FINISH):
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	725 if flush_mode not in (
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	726 COMPRESSOBJ_FLUSH_FINISH,
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	727 COMPRESSOBJ_FLUSH_BLOCK,
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	728 ):
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	729 raise ValueError("flush mode not recognized")
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	730
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	731 if self._finished:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	732 raise ZstdError("compressor object already finished")
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	733
40122 73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	734 if flush_mode == COMPRESSOBJ_FLUSH_BLOCK:
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	735 z_flush_mode = lib.ZSTD_e_flush
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	736 elif flush_mode == COMPRESSOBJ_FLUSH_FINISH:
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	737 z_flush_mode = lib.ZSTD_e_end
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	738 self._finished = True
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	739 else:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	740 raise ZstdError("unhandled flush mode")
40122 73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	741
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	742 assert self._out.pos == 0
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	743
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	744 in_buffer = ffi.new("ZSTD_inBuffer *")
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	745 in_buffer.src = ffi.NULL
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	746 in_buffer.size = 0
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	747 in_buffer.pos = 0
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	748
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	749 chunks = []
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	750
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	751 while True:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	752 zresult = lib.ZSTD_compressStream2(
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	753 self._compressor._cctx, self._out, in_buffer, z_flush_mode
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	754 )
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	755 if lib.ZSTD_isError(zresult):
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	756 raise ZstdError(
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	757 "error ending compression stream: %s" % _zstd_error(zresult)
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	758 )
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	759
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	760 if self._out.pos:
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	761 chunks.append(ffi.buffer(self._out.dst, self._out.pos)[:])
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	762 self._out.pos = 0
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	763
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	764 if not zresult:
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	765 break
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	766
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	767 return b"".join(chunks)
30444 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	768
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	769
52666 9bd6854aab86 pyupgrade: convert to new style classes Matt Harbison <matt_harbison@yahoo.com> parents: 51762 diff changeset	770 class ZstdCompressionChunker:
40122 73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	771 def __init__(self, compressor, chunk_size):
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	772 self._compressor = compressor
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	773 self._out = ffi.new("ZSTD_outBuffer *")
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	774 self._dst_buffer = ffi.new("char[]", chunk_size)
40122 73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	775 self._out.dst = self._dst_buffer
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	776 self._out.size = chunk_size
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	777 self._out.pos = 0
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	778
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	779 self._in = ffi.new("ZSTD_inBuffer *")
40122 73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	780 self._in.src = ffi.NULL
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	781 self._in.size = 0
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	782 self._in.pos = 0
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	783 self._finished = False
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	784
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	785 def compress(self, data):
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	786 if self._finished:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	787 raise ZstdError("cannot call compress() after compression finished")
40122 73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	788
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	789 if self._in.src != ffi.NULL:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	790 raise ZstdError(
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	791 "cannot perform operation before consuming output "
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	792 "from previous operation"
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	793 )
40122 73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	794
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	795 data_buffer = ffi.from_buffer(data)
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	796
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	797 if not len(data_buffer):
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	798 return
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	799
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	800 self._in.src = data_buffer
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	801 self._in.size = len(data_buffer)
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	802 self._in.pos = 0
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	803
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	804 while self._in.pos < self._in.size:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	805 zresult = lib.ZSTD_compressStream2(
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	806 self._compressor._cctx, self._out, self._in, lib.ZSTD_e_continue
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	807 )
40122 73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	808
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	809 if self._in.pos == self._in.size:
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	810 self._in.src = ffi.NULL
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	811 self._in.size = 0
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	812 self._in.pos = 0
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	813
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	814 if lib.ZSTD_isError(zresult):
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	815 raise ZstdError(
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	816 "zstd compress error: %s" % _zstd_error(zresult)
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	817 )
40122 73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	818
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	819 if self._out.pos == self._out.size:
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	820 yield ffi.buffer(self._out.dst, self._out.pos)[:]
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	821 self._out.pos = 0
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	822
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	823 def flush(self):
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	824 if self._finished:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	825 raise ZstdError("cannot call flush() after compression finished")
40122 73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	826
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	827 if self._in.src != ffi.NULL:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	828 raise ZstdError(
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	829 "cannot call flush() before consuming output from "
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	830 "previous operation"
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	831 )
40122 73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	832
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	833 while True:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	834 zresult = lib.ZSTD_compressStream2(
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	835 self._compressor._cctx, self._out, self._in, lib.ZSTD_e_flush
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	836 )
40122 73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	837 if lib.ZSTD_isError(zresult):
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	838 raise ZstdError(
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	839 "zstd compress error: %s" % _zstd_error(zresult)
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	840 )
40122 73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	841
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	842 if self._out.pos:
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	843 yield ffi.buffer(self._out.dst, self._out.pos)[:]
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	844 self._out.pos = 0
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	845
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	846 if not zresult:
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	847 return
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	848
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	849 def finish(self):
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	850 if self._finished:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	851 raise ZstdError("cannot call finish() after compression finished")
40122 73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	852
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	853 if self._in.src != ffi.NULL:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	854 raise ZstdError(
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	855 "cannot call finish() before consuming output from "
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	856 "previous operation"
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	857 )
40122 73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	858
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	859 while True:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	860 zresult = lib.ZSTD_compressStream2(
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	861 self._compressor._cctx, self._out, self._in, lib.ZSTD_e_end
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	862 )
40122 73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	863 if lib.ZSTD_isError(zresult):
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	864 raise ZstdError(
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	865 "zstd compress error: %s" % _zstd_error(zresult)
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	866 )
40122 73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	867
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	868 if self._out.pos:
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	869 yield ffi.buffer(self._out.dst, self._out.pos)[:]
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	870 self._out.pos = 0
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	871
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	872 if not zresult:
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	873 self._finished = True
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	874 return
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	875
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	876
52666 9bd6854aab86 pyupgrade: convert to new style classes Matt Harbison <matt_harbison@yahoo.com> parents: 51762 diff changeset	877 class ZstdCompressionReader:
40122 73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	878 def __init__(self, compressor, source, read_size):
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	879 self._compressor = compressor
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	880 self._source = source
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	881 self._read_size = read_size
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	882 self._entered = False
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	883 self._closed = False
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	884 self._bytes_compressed = 0
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	885 self._finished_input = False
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	886 self._finished_output = False
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	887
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	888 self._in_buffer = ffi.new("ZSTD_inBuffer *")
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	889 # Holds a ref so backing bytes in self._in_buffer stay alive.
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	890 self._source_buffer = None
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	891
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	892 def __enter__(self):
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	893 if self._entered:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	894 raise ValueError("cannot __enter__ multiple times")
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	895
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	896 self._entered = True
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	897 return self
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	898
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	899 def __exit__(self, exc_type, exc_value, exc_tb):
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	900 self._entered = False
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	901 self._closed = True
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	902 self._source = None
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	903 self._compressor = None
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	904
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	905 return False
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	906
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	907 def readable(self):
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	908 return True
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	909
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	910 def writable(self):
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	911 return False
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	912
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	913 def seekable(self):
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	914 return False
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	915
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	916 def readline(self):
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	917 raise io.UnsupportedOperation()
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	918
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	919 def readlines(self):
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	920 raise io.UnsupportedOperation()
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	921
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	922 def write(self, data):
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	923 raise OSError("stream is not writable")
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	924
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	925 def writelines(self, ignored):
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	926 raise OSError("stream is not writable")
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	927
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	928 def isatty(self):
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	929 return False
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	930
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	931 def flush(self):
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	932 return None
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	933
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	934 def close(self):
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	935 self._closed = True
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	936 return None
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	937
40122 73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	938 @property
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	939 def closed(self):
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	940 return self._closed
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	941
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	942 def tell(self):
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	943 return self._bytes_compressed
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	944
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	945 def readall(self):
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	946 chunks = []
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	947
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	948 while True:
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	949 chunk = self.read(1048576)
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	950 if not chunk:
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	951 break
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	952
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	953 chunks.append(chunk)
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	954
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	955 return b"".join(chunks)
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	956
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	957 def __iter__(self):
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	958 raise io.UnsupportedOperation()
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	959
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	960 def __next__(self):
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	961 raise io.UnsupportedOperation()
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	962
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	963 next = __next__
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	964
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	965 def _read_input(self):
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	966 if self._finished_input:
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	967 return
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	968
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	969 if hasattr(self._source, "read"):
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	970 data = self._source.read(self._read_size)
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	971
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	972 if not data:
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	973 self._finished_input = True
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	974 return
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	975
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	976 self._source_buffer = ffi.from_buffer(data)
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	977 self._in_buffer.src = self._source_buffer
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	978 self._in_buffer.size = len(self._source_buffer)
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	979 self._in_buffer.pos = 0
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	980 else:
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	981 self._source_buffer = ffi.from_buffer(self._source)
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	982 self._in_buffer.src = self._source_buffer
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	983 self._in_buffer.size = len(self._source_buffer)
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	984 self._in_buffer.pos = 0
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	985
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	986 def _compress_into_buffer(self, out_buffer):
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	987 if self._in_buffer.pos >= self._in_buffer.size:
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	988 return
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	989
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	990 old_pos = out_buffer.pos
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	991
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	992 zresult = lib.ZSTD_compressStream2(
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	993 self._compressor._cctx,
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	994 out_buffer,
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	995 self._in_buffer,
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	996 lib.ZSTD_e_continue,
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	997 )
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	998
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	999 self._bytes_compressed += out_buffer.pos - old_pos
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1000
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1001 if self._in_buffer.pos == self._in_buffer.size:
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1002 self._in_buffer.src = ffi.NULL
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1003 self._in_buffer.pos = 0
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1004 self._in_buffer.size = 0
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1005 self._source_buffer = None
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1006
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1007 if not hasattr(self._source, "read"):
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1008 self._finished_input = True
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1009
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1010 if lib.ZSTD_isError(zresult):
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1011 raise ZstdError("zstd compress error: %s", _zstd_error(zresult))
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1012
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1013 return out_buffer.pos and out_buffer.pos == out_buffer.size
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1014
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1015 def read(self, size=-1):
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1016 if self._closed:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1017 raise ValueError("stream is closed")
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1018
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1019 if size < -1:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1020 raise ValueError("cannot read negative amounts less than -1")
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1021
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1022 if size == -1:
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1023 return self.readall()
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1024
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1025 if self._finished_output or size == 0:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1026 return b""
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1027
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1028 # Need a dedicated ref to dest buffer otherwise it gets collected.
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1029 dst_buffer = ffi.new("char[]", size)
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1030 out_buffer = ffi.new("ZSTD_outBuffer *")
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1031 out_buffer.dst = dst_buffer
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1032 out_buffer.size = size
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1033 out_buffer.pos = 0
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1034
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1035 if self._compress_into_buffer(out_buffer):
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1036 return ffi.buffer(out_buffer.dst, out_buffer.pos)[:]
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1037
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1038 while not self._finished_input:
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1039 self._read_input()
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1040
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1041 if self._compress_into_buffer(out_buffer):
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1042 return ffi.buffer(out_buffer.dst, out_buffer.pos)[:]
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1043
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1044 # EOF
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1045 old_pos = out_buffer.pos
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1046
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1047 zresult = lib.ZSTD_compressStream2(
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1048 self._compressor._cctx, out_buffer, self._in_buffer, lib.ZSTD_e_end
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1049 )
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1050
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1051 self._bytes_compressed += out_buffer.pos - old_pos
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1052
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1053 if lib.ZSTD_isError(zresult):
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1054 raise ZstdError(
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1055 "error ending compression stream: %s", _zstd_error(zresult)
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1056 )
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1057
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1058 if zresult == 0:
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1059 self._finished_output = True
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1060
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1061 return ffi.buffer(out_buffer.dst, out_buffer.pos)[:]
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1062
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1063 def read1(self, size=-1):
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1064 if self._closed:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1065 raise ValueError("stream is closed")
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1066
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1067 if size < -1:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1068 raise ValueError("cannot read negative amounts less than -1")
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1069
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1070 if self._finished_output or size == 0:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1071 return b""
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1072
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1073 # -1 returns arbitrary number of bytes.
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1074 if size == -1:
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1075 size = COMPRESSION_RECOMMENDED_OUTPUT_SIZE
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1076
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1077 dst_buffer = ffi.new("char[]", size)
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1078 out_buffer = ffi.new("ZSTD_outBuffer *")
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1079 out_buffer.dst = dst_buffer
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1080 out_buffer.size = size
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1081 out_buffer.pos = 0
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1082
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1083 # read1() dictates that we can perform at most 1 call to the
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1084 # underlying stream to get input. However, we can't satisfy this
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1085 # restriction with compression because not all input generates output.
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1086 # It is possible to perform a block flush in order to ensure output.
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1087 # But this may not be desirable behavior. So we allow multiple read()
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1088 # to the underlying stream. But unlike read(), we stop once we have
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1089 # any output.
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1090
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1091 self._compress_into_buffer(out_buffer)
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1092 if out_buffer.pos:
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1093 return ffi.buffer(out_buffer.dst, out_buffer.pos)[:]
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1094
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1095 while not self._finished_input:
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1096 self._read_input()
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1097
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1098 # If we've filled the output buffer, return immediately.
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1099 if self._compress_into_buffer(out_buffer):
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1100 return ffi.buffer(out_buffer.dst, out_buffer.pos)[:]
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1101
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1102 # If we've populated the output buffer and we're not at EOF,
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1103 # also return, as we've satisfied the read1() limits.
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1104 if out_buffer.pos and not self._finished_input:
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1105 return ffi.buffer(out_buffer.dst, out_buffer.pos)[:]
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1106
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1107 # Else if we're at EOS and we have room left in the buffer,
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1108 # fall through to below and try to add more data to the output.
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1109
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1110 # EOF.
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1111 old_pos = out_buffer.pos
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1112
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1113 zresult = lib.ZSTD_compressStream2(
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1114 self._compressor._cctx, out_buffer, self._in_buffer, lib.ZSTD_e_end
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1115 )
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1116
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1117 self._bytes_compressed += out_buffer.pos - old_pos
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1118
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1119 if lib.ZSTD_isError(zresult):
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1120 raise ZstdError(
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1121 "error ending compression stream: %s" % _zstd_error(zresult)
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1122 )
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1123
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1124 if zresult == 0:
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1125 self._finished_output = True
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1126
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1127 return ffi.buffer(out_buffer.dst, out_buffer.pos)[:]
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1128
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1129 def readinto(self, b):
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1130 if self._closed:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1131 raise ValueError("stream is closed")
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1132
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1133 if self._finished_output:
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1134 return 0
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1135
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1136 # TODO use writable=True once we require CFFI >= 1.12.
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1137 dest_buffer = ffi.from_buffer(b)
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1138 ffi.memmove(b, b"", 0)
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1139 out_buffer = ffi.new("ZSTD_outBuffer *")
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1140 out_buffer.dst = dest_buffer
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1141 out_buffer.size = len(dest_buffer)
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1142 out_buffer.pos = 0
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1143
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1144 if self._compress_into_buffer(out_buffer):
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1145 return out_buffer.pos
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1146
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1147 while not self._finished_input:
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1148 self._read_input()
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1149 if self._compress_into_buffer(out_buffer):
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1150 return out_buffer.pos
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1151
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1152 # EOF.
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1153 old_pos = out_buffer.pos
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1154 zresult = lib.ZSTD_compressStream2(
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1155 self._compressor._cctx, out_buffer, self._in_buffer, lib.ZSTD_e_end
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1156 )
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1157
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1158 self._bytes_compressed += out_buffer.pos - old_pos
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1159
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1160 if lib.ZSTD_isError(zresult):
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1161 raise ZstdError(
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1162 "error ending compression stream: %s", _zstd_error(zresult)
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1163 )
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1164
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1165 if zresult == 0:
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1166 self._finished_output = True
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1167
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1168 return out_buffer.pos
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1169
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1170 def readinto1(self, b):
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1171 if self._closed:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1172 raise ValueError("stream is closed")
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1173
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1174 if self._finished_output:
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1175 return 0
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1176
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1177 # TODO use writable=True once we require CFFI >= 1.12.
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1178 dest_buffer = ffi.from_buffer(b)
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1179 ffi.memmove(b, b"", 0)
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1180
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1181 out_buffer = ffi.new("ZSTD_outBuffer *")
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1182 out_buffer.dst = dest_buffer
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1183 out_buffer.size = len(dest_buffer)
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1184 out_buffer.pos = 0
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1185
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1186 self._compress_into_buffer(out_buffer)
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1187 if out_buffer.pos:
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1188 return out_buffer.pos
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1189
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1190 while not self._finished_input:
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1191 self._read_input()
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1192
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1193 if self._compress_into_buffer(out_buffer):
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1194 return out_buffer.pos
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1195
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1196 if out_buffer.pos and not self._finished_input:
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1197 return out_buffer.pos
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1198
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1199 # EOF.
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1200 old_pos = out_buffer.pos
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1201
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1202 zresult = lib.ZSTD_compressStream2(
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1203 self._compressor._cctx, out_buffer, self._in_buffer, lib.ZSTD_e_end
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1204 )
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1205
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1206 self._bytes_compressed += out_buffer.pos - old_pos
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1207
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1208 if lib.ZSTD_isError(zresult):
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1209 raise ZstdError(
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1210 "error ending compression stream: %s" % _zstd_error(zresult)
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1211 )
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1212
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1213 if zresult == 0:
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1214 self._finished_output = True
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1215
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1216 return out_buffer.pos
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1217
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1218
52666 9bd6854aab86 pyupgrade: convert to new style classes Matt Harbison <matt_harbison@yahoo.com> parents: 51762 diff changeset	1219 class ZstdCompressor:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1220 def __init__(
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1221 self,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1222 level=3,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1223 dict_data=None,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1224 compression_params=None,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1225 write_checksum=None,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1226 write_content_size=None,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1227 write_dict_id=None,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1228 threads=0,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1229 ):
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1230 if level > lib.ZSTD_maxCLevel():
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1231 raise ValueError(
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1232 "level must be less than %d" % lib.ZSTD_maxCLevel()
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1233 )
30444 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1234
31799 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30924 diff changeset	1235 if threads < 0:
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30924 diff changeset	1236 threads = _cpu_count()
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30924 diff changeset	1237
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1238 if compression_params and write_checksum is not None:
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1239 raise ValueError(
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1240 "cannot define compression_params and " "write_checksum"
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1241 )
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1242
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1243 if compression_params and write_content_size is not None:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1244 raise ValueError(
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1245 "cannot define compression_params and " "write_content_size"
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1246 )
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1247
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1248 if compression_params and write_dict_id is not None:
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1249 raise ValueError(
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1250 "cannot define compression_params and " "write_dict_id"
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1251 )
30444 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1252
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1253 if compression_params and threads:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1254 raise ValueError("cannot define compression_params and threads")
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1255
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1256 if compression_params:
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1257 self._params = _make_cctx_params(compression_params)
31799 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30924 diff changeset	1258 else:
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1259 if write_dict_id is None:
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1260 write_dict_id = True
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1261
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1262 params = lib.ZSTD_createCCtxParams()
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1263 if params == ffi.NULL:
31799 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30924 diff changeset	1264 raise MemoryError()
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30924 diff changeset	1265
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1266 self._params = ffi.gc(params, lib.ZSTD_freeCCtxParams)
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1267
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1268 _set_compression_parameter(
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1269 self._params, lib.ZSTD_c_compressionLevel, level
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1270 )
31799 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30924 diff changeset	1271
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1272 _set_compression_parameter(
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1273 self._params,
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1274 lib.ZSTD_c_contentSizeFlag,
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1275 write_content_size if write_content_size is not None else 1,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1276 )
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1277
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1278 _set_compression_parameter(
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1279 self._params,
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1280 lib.ZSTD_c_checksumFlag,
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1281 1 if write_checksum else 0,
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1282 )
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1283
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1284 _set_compression_parameter(
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1285 self._params, lib.ZSTD_c_dictIDFlag, 1 if write_dict_id else 0
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1286 )
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1287
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1288 if threads:
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1289 _set_compression_parameter(
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1290 self._params, lib.ZSTD_c_nbWorkers, threads
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1291 )
31799 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30924 diff changeset	1292
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1293 cctx = lib.ZSTD_createCCtx()
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1294 if cctx == ffi.NULL:
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1295 raise MemoryError()
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1296
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1297 self._cctx = cctx
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1298 self._dict_data = dict_data
31799 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30924 diff changeset	1299
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1300 # We defer setting up garbage collection until after calling
40122 73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	1301 # _setup_cctx() to ensure the memory size estimate is more accurate.
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1302 try:
40122 73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	1303 self._setup_cctx()
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1304 finally:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1305 self._cctx = ffi.gc(
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1306 cctx, lib.ZSTD_freeCCtx, size=lib.ZSTD_sizeof_CCtx(cctx)
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1307 )
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1308
40122 73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	1309 def _setup_cctx(self):
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1310 zresult = lib.ZSTD_CCtx_setParametersUsingCCtxParams(
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1311 self._cctx, self._params
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1312 )
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1313 if lib.ZSTD_isError(zresult):
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1314 raise ZstdError(
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1315 "could not set compression parameters: %s"
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1316 % _zstd_error(zresult)
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1317 )
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1318
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1319 dict_data = self._dict_data
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1320
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1321 if dict_data:
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1322 if dict_data._cdict:
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1323 zresult = lib.ZSTD_CCtx_refCDict(self._cctx, dict_data._cdict)
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1324 else:
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1325 zresult = lib.ZSTD_CCtx_loadDictionary_advanced(
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1326 self._cctx,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1327 dict_data.as_bytes(),
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1328 len(dict_data),
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1329 lib.ZSTD_dlm_byRef,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1330 dict_data._dict_type,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1331 )
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1332
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1333 if lib.ZSTD_isError(zresult):
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1334 raise ZstdError(
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1335 "could not load compression dictionary: %s"
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1336 % _zstd_error(zresult)
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1337 )
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1338
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1339 def memory_size(self):
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1340 return lib.ZSTD_sizeof_CCtx(self._cctx)
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1341
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1342 def compress(self, data):
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1343 lib.ZSTD_CCtx_reset(self._cctx, lib.ZSTD_reset_session_only)
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1344
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1345 data_buffer = ffi.from_buffer(data)
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1346
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1347 dest_size = lib.ZSTD_compressBound(len(data_buffer))
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1348 out = new_nonzero("char[]", dest_size)
30444 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1349
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1350 zresult = lib.ZSTD_CCtx_setPledgedSrcSize(self._cctx, len(data_buffer))
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1351 if lib.ZSTD_isError(zresult):
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1352 raise ZstdError(
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1353 "error setting source size: %s" % _zstd_error(zresult)
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1354 )
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1355
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1356 out_buffer = ffi.new("ZSTD_outBuffer *")
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1357 in_buffer = ffi.new("ZSTD_inBuffer *")
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1358
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1359 out_buffer.dst = out
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1360 out_buffer.size = dest_size
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1361 out_buffer.pos = 0
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1362
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1363 in_buffer.src = data_buffer
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1364 in_buffer.size = len(data_buffer)
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1365 in_buffer.pos = 0
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1366
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1367 zresult = lib.ZSTD_compressStream2(
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1368 self._cctx, out_buffer, in_buffer, lib.ZSTD_e_end
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1369 )
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1370
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1371 if lib.ZSTD_isError(zresult):
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1372 raise ZstdError("cannot compress: %s" % _zstd_error(zresult))
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1373 elif zresult:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1374 raise ZstdError("unexpected partial frame flush")
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1375
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1376 return ffi.buffer(out, out_buffer.pos)[:]
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1377
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1378 def compressobj(self, size=-1):
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1379 lib.ZSTD_CCtx_reset(self._cctx, lib.ZSTD_reset_session_only)
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1380
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1381 if size < 0:
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1382 size = lib.ZSTD_CONTENTSIZE_UNKNOWN
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1383
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1384 zresult = lib.ZSTD_CCtx_setPledgedSrcSize(self._cctx, size)
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1385 if lib.ZSTD_isError(zresult):
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1386 raise ZstdError(
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1387 "error setting source size: %s" % _zstd_error(zresult)
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1388 )
31799 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30924 diff changeset	1389
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1390 cobj = ZstdCompressionObj()
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1391 cobj._out = ffi.new("ZSTD_outBuffer *")
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1392 cobj._dst_buffer = ffi.new(
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1393 "char[]", COMPRESSION_RECOMMENDED_OUTPUT_SIZE
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1394 )
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1395 cobj._out.dst = cobj._dst_buffer
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1396 cobj._out.size = COMPRESSION_RECOMMENDED_OUTPUT_SIZE
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1397 cobj._out.pos = 0
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1398 cobj._compressor = self
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1399 cobj._finished = False
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1400
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1401 return cobj
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1402
40122 73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	1403 def chunker(self, size=-1, chunk_size=COMPRESSION_RECOMMENDED_OUTPUT_SIZE):
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1404 lib.ZSTD_CCtx_reset(self._cctx, lib.ZSTD_reset_session_only)
40122 73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	1405
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	1406 if size < 0:
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	1407 size = lib.ZSTD_CONTENTSIZE_UNKNOWN
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	1408
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	1409 zresult = lib.ZSTD_CCtx_setPledgedSrcSize(self._cctx, size)
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	1410 if lib.ZSTD_isError(zresult):
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1411 raise ZstdError(
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1412 "error setting source size: %s" % _zstd_error(zresult)
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1413 )
40122 73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	1414
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	1415 return ZstdCompressionChunker(self, chunk_size=chunk_size)
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	1416
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1417 def copy_stream(
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1418 self,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1419 ifh,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1420 ofh,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1421 size=-1,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1422 read_size=COMPRESSION_RECOMMENDED_INPUT_SIZE,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1423 write_size=COMPRESSION_RECOMMENDED_OUTPUT_SIZE,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1424 ):
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1425 if not hasattr(ifh, "read"):
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1426 raise ValueError("first argument must have a read() method")
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1427 if not hasattr(ofh, "write"):
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1428 raise ValueError("second argument must have a write() method")
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1429
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1430 lib.ZSTD_CCtx_reset(self._cctx, lib.ZSTD_reset_session_only)
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1431
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1432 if size < 0:
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1433 size = lib.ZSTD_CONTENTSIZE_UNKNOWN
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1434
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1435 zresult = lib.ZSTD_CCtx_setPledgedSrcSize(self._cctx, size)
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1436 if lib.ZSTD_isError(zresult):
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1437 raise ZstdError(
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1438 "error setting source size: %s" % _zstd_error(zresult)
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1439 )
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1440
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1441 in_buffer = ffi.new("ZSTD_inBuffer *")
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1442 out_buffer = ffi.new("ZSTD_outBuffer *")
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1443
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1444 dst_buffer = ffi.new("char[]", write_size)
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1445 out_buffer.dst = dst_buffer
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1446 out_buffer.size = write_size
30444 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1447 out_buffer.pos = 0
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1448
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1449 total_read, total_write = 0, 0
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1450
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1451 while True:
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1452 data = ifh.read(read_size)
30444 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1453 if not data:
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1454 break
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1455
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1456 data_buffer = ffi.from_buffer(data)
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1457 total_read += len(data_buffer)
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1458 in_buffer.src = data_buffer
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1459 in_buffer.size = len(data_buffer)
30444 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1460 in_buffer.pos = 0
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1461
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1462 while in_buffer.pos < in_buffer.size:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1463 zresult = lib.ZSTD_compressStream2(
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1464 self._cctx, out_buffer, in_buffer, lib.ZSTD_e_continue
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1465 )
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1466 if lib.ZSTD_isError(zresult):
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1467 raise ZstdError(
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1468 "zstd compress error: %s" % _zstd_error(zresult)
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1469 )
30444 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1470
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1471 if out_buffer.pos:
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1472 ofh.write(ffi.buffer(out_buffer.dst, out_buffer.pos))
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1473 total_write += out_buffer.pos
30444 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1474 out_buffer.pos = 0
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1475
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1476 # We've finished reading. Flush the compressor.
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1477 while True:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1478 zresult = lib.ZSTD_compressStream2(
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1479 self._cctx, out_buffer, in_buffer, lib.ZSTD_e_end
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1480 )
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1481 if lib.ZSTD_isError(zresult):
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1482 raise ZstdError(
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1483 "error ending compression stream: %s" % _zstd_error(zresult)
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1484 )
30444 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1485
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1486 if out_buffer.pos:
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1487 ofh.write(ffi.buffer(out_buffer.dst, out_buffer.pos))
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1488 total_write += out_buffer.pos
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1489 out_buffer.pos = 0
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1490
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1491 if zresult == 0:
30444 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1492 break
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1493
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1494 return total_read, total_write
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1495
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1496 def stream_reader(
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1497 self, source, size=-1, read_size=COMPRESSION_RECOMMENDED_INPUT_SIZE
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1498 ):
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1499 lib.ZSTD_CCtx_reset(self._cctx, lib.ZSTD_reset_session_only)
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1500
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1501 try:
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1502 size = len(source)
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1503 except Exception:
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1504 pass
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1505
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1506 if size < 0:
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1507 size = lib.ZSTD_CONTENTSIZE_UNKNOWN
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1508
40122 73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	1509 zresult = lib.ZSTD_CCtx_setPledgedSrcSize(self._cctx, size)
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	1510 if lib.ZSTD_isError(zresult):
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1511 raise ZstdError(
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1512 "error setting source size: %s" % _zstd_error(zresult)
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1513 )
40122 73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	1514
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1515 return ZstdCompressionReader(self, source, read_size)
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1516
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1517 def stream_writer(
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1518 self,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1519 writer,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1520 size=-1,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1521 write_size=COMPRESSION_RECOMMENDED_OUTPUT_SIZE,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1522 write_return_read=False,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1523 ):
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1524 if not hasattr(writer, "write"):
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1525 raise ValueError("must pass an object with a write() method")
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1526
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1527 lib.ZSTD_CCtx_reset(self._cctx, lib.ZSTD_reset_session_only)
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1528
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1529 if size < 0:
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1530 size = lib.ZSTD_CONTENTSIZE_UNKNOWN
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1531
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1532 return ZstdCompressionWriter(
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1533 self, writer, size, write_size, write_return_read
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1534 )
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1535
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1536 write_to = stream_writer
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1537
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1538 def read_to_iter(
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1539 self,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1540 reader,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1541 size=-1,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1542 read_size=COMPRESSION_RECOMMENDED_INPUT_SIZE,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1543 write_size=COMPRESSION_RECOMMENDED_OUTPUT_SIZE,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1544 ):
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1545 if hasattr(reader, "read"):
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1546 have_read = True
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1547 elif hasattr(reader, "__getitem__"):
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1548 have_read = False
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1549 buffer_offset = 0
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1550 size = len(reader)
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1551 else:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1552 raise ValueError(
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1553 "must pass an object with a read() method or "
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1554 "conforms to buffer protocol"
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1555 )
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1556
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1557 lib.ZSTD_CCtx_reset(self._cctx, lib.ZSTD_reset_session_only)
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1558
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1559 if size < 0:
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1560 size = lib.ZSTD_CONTENTSIZE_UNKNOWN
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1561
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1562 zresult = lib.ZSTD_CCtx_setPledgedSrcSize(self._cctx, size)
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1563 if lib.ZSTD_isError(zresult):
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1564 raise ZstdError(
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1565 "error setting source size: %s" % _zstd_error(zresult)
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1566 )
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1567
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1568 in_buffer = ffi.new("ZSTD_inBuffer *")
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1569 out_buffer = ffi.new("ZSTD_outBuffer *")
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1570
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1571 in_buffer.src = ffi.NULL
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1572 in_buffer.size = 0
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1573 in_buffer.pos = 0
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1574
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1575 dst_buffer = ffi.new("char[]", write_size)
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1576 out_buffer.dst = dst_buffer
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1577 out_buffer.size = write_size
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1578 out_buffer.pos = 0
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1579
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1580 while True:
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1581 # We should never have output data sitting around after a previous
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1582 # iteration.
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1583 assert out_buffer.pos == 0
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1584
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1585 # Collect input data.
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1586 if have_read:
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1587 read_result = reader.read(read_size)
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1588 else:
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1589 remaining = len(reader) - buffer_offset
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1590 slice_size = min(remaining, read_size)
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1591 read_result = reader[buffer_offset : buffer_offset + slice_size]
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1592 buffer_offset += slice_size
30444 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1593
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1594 # No new input data. Break out of the read loop.
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1595 if not read_result:
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1596 break
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1597
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1598 # Feed all read data into the compressor and emit output until
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1599 # exhausted.
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1600 read_buffer = ffi.from_buffer(read_result)
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1601 in_buffer.src = read_buffer
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1602 in_buffer.size = len(read_buffer)
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1603 in_buffer.pos = 0
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1604
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1605 while in_buffer.pos < in_buffer.size:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1606 zresult = lib.ZSTD_compressStream2(
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1607 self._cctx, out_buffer, in_buffer, lib.ZSTD_e_continue
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1608 )
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1609 if lib.ZSTD_isError(zresult):
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1610 raise ZstdError(
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1611 "zstd compress error: %s" % _zstd_error(zresult)
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1612 )
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1613
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1614 if out_buffer.pos:
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1615 data = ffi.buffer(out_buffer.dst, out_buffer.pos)[:]
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1616 out_buffer.pos = 0
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1617 yield data
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1618
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1619 assert out_buffer.pos == 0
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1620
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1621 # And repeat the loop to collect more data.
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1622 continue
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1623
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1624 # If we get here, input is exhausted. End the stream and emit what
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1625 # remains.
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1626 while True:
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1627 assert out_buffer.pos == 0
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1628 zresult = lib.ZSTD_compressStream2(
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1629 self._cctx, out_buffer, in_buffer, lib.ZSTD_e_end
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1630 )
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1631 if lib.ZSTD_isError(zresult):
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1632 raise ZstdError(
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1633 "error ending compression stream: %s" % _zstd_error(zresult)
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1634 )
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1635
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1636 if out_buffer.pos:
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1637 data = ffi.buffer(out_buffer.dst, out_buffer.pos)[:]
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1638 out_buffer.pos = 0
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1639 yield data
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1640
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1641 if zresult == 0:
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1642 break
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1643
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1644 read_from = read_to_iter
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1645
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1646 def frame_progression(self):
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1647 progression = lib.ZSTD_getFrameProgression(self._cctx)
31799 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30924 diff changeset	1648
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1649 return progression.ingested, progression.consumed, progression.produced
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1650
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1651
52666 9bd6854aab86 pyupgrade: convert to new style classes Matt Harbison <matt_harbison@yahoo.com> parents: 51762 diff changeset	1652 class FrameParameters:
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1653 def __init__(self, fparams):
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1654 self.content_size = fparams.frameContentSize
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1655 self.window_size = fparams.windowSize
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1656 self.dict_id = fparams.dictID
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1657 self.has_checksum = bool(fparams.checksumFlag)
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1658
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1659
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1660 def frame_content_size(data):
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1661 data_buffer = ffi.from_buffer(data)
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1662
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1663 size = lib.ZSTD_getFrameContentSize(data_buffer, len(data_buffer))
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1664
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1665 if size == lib.ZSTD_CONTENTSIZE_ERROR:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1666 raise ZstdError("error when determining content size")
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1667 elif size == lib.ZSTD_CONTENTSIZE_UNKNOWN:
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1668 return -1
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1669 else:
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1670 return size
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1671
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1672
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1673 def frame_header_size(data):
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1674 data_buffer = ffi.from_buffer(data)
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1675
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1676 zresult = lib.ZSTD_frameHeaderSize(data_buffer, len(data_buffer))
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1677 if lib.ZSTD_isError(zresult):
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1678 raise ZstdError(
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1679 "could not determine frame header size: %s" % _zstd_error(zresult)
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1680 )
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1681
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1682 return zresult
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1683
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1684
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1685 def get_frame_parameters(data):
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1686 params = ffi.new("ZSTD_frameHeader *")
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1687
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1688 data_buffer = ffi.from_buffer(data)
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1689 zresult = lib.ZSTD_getFrameHeader(params, data_buffer, len(data_buffer))
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1690 if lib.ZSTD_isError(zresult):
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1691 raise ZstdError(
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1692 "cannot get frame parameters: %s" % _zstd_error(zresult)
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1693 )
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1694
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1695 if zresult:
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1696 raise ZstdError(
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1697 "not enough data for frame parameters; need %d bytes" % zresult
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1698 )
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1699
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1700 return FrameParameters(params[0])
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1701
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1702
52666 9bd6854aab86 pyupgrade: convert to new style classes Matt Harbison <matt_harbison@yahoo.com> parents: 51762 diff changeset	1703 class ZstdCompressionDict:
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1704 def __init__(self, data, dict_type=DICT_TYPE_AUTO, k=0, d=0):
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1705 assert isinstance(data, bytes_type)
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1706 self._data = data
31799 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30924 diff changeset	1707 self.k = k
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30924 diff changeset	1708 self.d = d
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1709
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1710 if dict_type not in (
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1711 DICT_TYPE_AUTO,
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1712 DICT_TYPE_RAWCONTENT,
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1713 DICT_TYPE_FULLDICT,
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1714 ):
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1715 raise ValueError(
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1716 "invalid dictionary load mode: %d; must use "
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1717 "DICT_TYPE_* constants"
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1718 )
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1719
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1720 self._dict_type = dict_type
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1721 self._cdict = None
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1722
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1723 def __len__(self):
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1724 return len(self._data)
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1725
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1726 def dict_id(self):
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1727 return int_type(lib.ZDICT_getDictID(self._data, len(self._data)))
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1728
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1729 def as_bytes(self):
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1730 return self._data
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1731
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1732 def precompute_compress(self, level=0, compression_params=None):
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1733 if level and compression_params:
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1734 raise ValueError(
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1735 "must only specify one of level or " "compression_params"
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1736 )
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1737
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1738 if not level and not compression_params:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1739 raise ValueError("must specify one of level or compression_params")
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1740
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1741 if level:
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1742 cparams = lib.ZSTD_getCParams(level, 0, len(self._data))
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1743 else:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1744 cparams = ffi.new("ZSTD_compressionParameters")
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1745 cparams.chainLog = compression_params.chain_log
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1746 cparams.hashLog = compression_params.hash_log
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1747 cparams.minMatch = compression_params.min_match
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1748 cparams.searchLog = compression_params.search_log
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1749 cparams.strategy = compression_params.compression_strategy
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1750 cparams.targetLength = compression_params.target_length
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1751 cparams.windowLog = compression_params.window_log
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1752
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1753 cdict = lib.ZSTD_createCDict_advanced(
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1754 self._data,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1755 len(self._data),
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1756 lib.ZSTD_dlm_byRef,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1757 self._dict_type,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1758 cparams,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1759 lib.ZSTD_defaultCMem,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1760 )
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1761 if cdict == ffi.NULL:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1762 raise ZstdError("unable to precompute dictionary")
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1763
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1764 self._cdict = ffi.gc(
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1765 cdict, lib.ZSTD_freeCDict, size=lib.ZSTD_sizeof_CDict(cdict)
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1766 )
31799 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30924 diff changeset	1767
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1768 @property
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1769 def _ddict(self):
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1770 ddict = lib.ZSTD_createDDict_advanced(
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1771 self._data,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1772 len(self._data),
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1773 lib.ZSTD_dlm_byRef,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1774 self._dict_type,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1775 lib.ZSTD_defaultCMem,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1776 )
31799 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30924 diff changeset	1777
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1778 if ddict == ffi.NULL:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1779 raise ZstdError("could not create decompression dict")
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1780
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1781 ddict = ffi.gc(
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1782 ddict, lib.ZSTD_freeDDict, size=lib.ZSTD_sizeof_DDict(ddict)
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1783 )
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1784 self.__dict__["_ddict"] = ddict
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1785
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1786 return ddict
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1787
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1788
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1789 def train_dictionary(
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1790 dict_size,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1791 samples,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1792 k=0,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1793 d=0,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1794 notifications=0,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1795 dict_id=0,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1796 level=0,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1797 steps=0,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1798 threads=0,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1799 ):
31799 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30924 diff changeset	1800 if not isinstance(samples, list):
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1801 raise TypeError("samples must be a list")
31799 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30924 diff changeset	1802
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30924 diff changeset	1803 if threads < 0:
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30924 diff changeset	1804 threads = _cpu_count()
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30924 diff changeset	1805
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30924 diff changeset	1806 total_size = sum(map(len, samples))
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30924 diff changeset	1807
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1808 samples_buffer = new_nonzero("char[]", total_size)
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1809 sample_sizes = new_nonzero("size_t[]", len(samples))
31799 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30924 diff changeset	1810
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30924 diff changeset	1811 offset = 0
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30924 diff changeset	1812 for i, sample in enumerate(samples):
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30924 diff changeset	1813 if not isinstance(sample, bytes_type):
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1814 raise ValueError("samples must be bytes")
31799 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30924 diff changeset	1815
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30924 diff changeset	1816 l = len(sample)
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30924 diff changeset	1817 ffi.memmove(samples_buffer + offset, sample, l)
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30924 diff changeset	1818 offset += l
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30924 diff changeset	1819 sample_sizes[i] = l
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30924 diff changeset	1820
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1821 dict_data = new_nonzero("char[]", dict_size)
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1822
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1823 dparams = ffi.new("ZDICT_cover_params_t *")[0]
31799 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30924 diff changeset	1824 dparams.k = k
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30924 diff changeset	1825 dparams.d = d
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30924 diff changeset	1826 dparams.steps = steps
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30924 diff changeset	1827 dparams.nbThreads = threads
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1828 dparams.zParams.notificationLevel = notifications
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1829 dparams.zParams.dictID = dict_id
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1830 dparams.zParams.compressionLevel = level
31799 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30924 diff changeset	1831
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1832 if (
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1833 not dparams.k
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1834 and not dparams.d
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1835 and not dparams.steps
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1836 and not dparams.nbThreads
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1837 and not dparams.zParams.notificationLevel
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1838 and not dparams.zParams.dictID
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1839 and not dparams.zParams.compressionLevel
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1840 ):
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1841 zresult = lib.ZDICT_trainFromBuffer(
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1842 ffi.addressof(dict_data),
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1843 dict_size,
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1844 ffi.addressof(samples_buffer),
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1845 ffi.addressof(sample_sizes, 0),
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1846 len(samples),
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1847 )
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1848 elif dparams.steps or dparams.nbThreads:
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1849 zresult = lib.ZDICT_optimizeTrainFromBuffer_cover(
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1850 ffi.addressof(dict_data),
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1851 dict_size,
31799 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30924 diff changeset	1852 ffi.addressof(samples_buffer),
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1853 ffi.addressof(sample_sizes, 0),
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1854 len(samples),
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1855 ffi.addressof(dparams),
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1856 )
31799 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30924 diff changeset	1857 else:
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1858 zresult = lib.ZDICT_trainFromBuffer_cover(
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1859 ffi.addressof(dict_data),
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1860 dict_size,
31799 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30924 diff changeset	1861 ffi.addressof(samples_buffer),
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1862 ffi.addressof(sample_sizes, 0),
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1863 len(samples),
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1864 dparams,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1865 )
31799 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30924 diff changeset	1866
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30924 diff changeset	1867 if lib.ZDICT_isError(zresult):
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1868 msg = ffi.string(lib.ZDICT_getErrorName(zresult)).decode("utf-8")
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1869 raise ZstdError("cannot train dict: %s" % msg)
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1870
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1871 return ZstdCompressionDict(
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1872 ffi.buffer(dict_data, zresult)[:],
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1873 dict_type=DICT_TYPE_FULLDICT,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1874 k=dparams.k,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1875 d=dparams.d,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1876 )
31799 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30924 diff changeset	1877
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30924 diff changeset	1878
52666 9bd6854aab86 pyupgrade: convert to new style classes Matt Harbison <matt_harbison@yahoo.com> parents: 51762 diff changeset	1879 class ZstdDecompressionObj:
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1880 def __init__(self, decompressor, write_size):
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1881 self._decompressor = decompressor
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1882 self._write_size = write_size
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1883 self._finished = False
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1884
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1885 def decompress(self, data):
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1886 if self._finished:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1887 raise ZstdError("cannot use a decompressobj multiple times")
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1888
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1889 in_buffer = ffi.new("ZSTD_inBuffer *")
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1890 out_buffer = ffi.new("ZSTD_outBuffer *")
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1891
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1892 data_buffer = ffi.from_buffer(data)
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1893
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1894 if len(data_buffer) == 0:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1895 return b""
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1896
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1897 in_buffer.src = data_buffer
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1898 in_buffer.size = len(data_buffer)
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1899 in_buffer.pos = 0
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1900
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1901 dst_buffer = ffi.new("char[]", self._write_size)
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1902 out_buffer.dst = dst_buffer
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1903 out_buffer.size = len(dst_buffer)
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1904 out_buffer.pos = 0
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1905
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1906 chunks = []
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1907
40122 73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	1908 while True:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1909 zresult = lib.ZSTD_decompressStream(
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1910 self._decompressor._dctx, out_buffer, in_buffer
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1911 )
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1912 if lib.ZSTD_isError(zresult):
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1913 raise ZstdError(
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1914 "zstd decompressor error: %s" % _zstd_error(zresult)
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	1915 )
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1916
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1917 if zresult == 0:
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1918 self._finished = True
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1919 self._decompressor = None
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1920
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1921 if out_buffer.pos:
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1922 chunks.append(ffi.buffer(out_buffer.dst, out_buffer.pos)[:])
40122 73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	1923
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1924 if zresult == 0 or (
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1925 in_buffer.pos == in_buffer.size and out_buffer.pos == 0
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1926 ):
40122 73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	1927 break
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	1928
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	1929 out_buffer.pos = 0
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1930
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1931 return b"".join(chunks)
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	1932
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1933 def flush(self, length=0):
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1934 pass
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1935
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1936
52666 9bd6854aab86 pyupgrade: convert to new style classes Matt Harbison <matt_harbison@yahoo.com> parents: 51762 diff changeset	1937 class ZstdDecompressionReader:
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1938 def __init__(self, decompressor, source, read_size, read_across_frames):
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1939 self._decompressor = decompressor
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1940 self._source = source
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1941 self._read_size = read_size
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1942 self._read_across_frames = bool(read_across_frames)
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1943 self._entered = False
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1944 self._closed = False
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1945 self._bytes_decompressed = 0
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1946 self._finished_input = False
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1947 self._finished_output = False
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1948 self._in_buffer = ffi.new("ZSTD_inBuffer *")
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1949 # Holds a ref to self._in_buffer.src.
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1950 self._source_buffer = None
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1951
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1952 def __enter__(self):
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1953 if self._entered:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	1954 raise ValueError("cannot __enter__ multiple times")
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1955
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1956 self._entered = True
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1957 return self
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1958
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1959 def __exit__(self, exc_type, exc_value, exc_tb):
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1960 self._entered = False
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1961 self._closed = True
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1962 self._source = None
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1963 self._decompressor = None
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1964
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1965 return False
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1966
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1967 def readable(self):
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1968 return True
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1969
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1970 def writable(self):
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1971 return False
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1972
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1973 def seekable(self):
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1974 return True
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1975
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1976 def readline(self):
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1977 raise io.UnsupportedOperation()
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1978
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1979 def readlines(self):
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	1980 raise io.UnsupportedOperation()
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1981
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1982 def write(self, data):
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1983 raise io.UnsupportedOperation()
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1984
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1985 def writelines(self, lines):
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1986 raise io.UnsupportedOperation()
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1987
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1988 def isatty(self):
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1989 return False
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1990
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1991 def flush(self):
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1992 return None
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1993
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1994 def close(self):
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1995 self._closed = True
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1996 return None
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1997
40122 73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	1998 @property
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	1999 def closed(self):
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2000 return self._closed
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2001
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2002 def tell(self):
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2003 return self._bytes_decompressed
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2004
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2005 def readall(self):
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2006 chunks = []
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2007
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2008 while True:
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2009 chunk = self.read(1048576)
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2010 if not chunk:
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2011 break
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2012
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2013 chunks.append(chunk)
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2014
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2015 return b"".join(chunks)
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2016
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2017 def __iter__(self):
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2018 raise io.UnsupportedOperation()
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2019
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2020 def __next__(self):
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2021 raise io.UnsupportedOperation()
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2022
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2023 next = __next__
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2024
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2025 def _read_input(self):
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2026 # We have data left over in the input buffer. Use it.
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2027 if self._in_buffer.pos < self._in_buffer.size:
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2028 return
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2029
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2030 # All input data exhausted. Nothing to do.
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2031 if self._finished_input:
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2032 return
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2033
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2034 # Else populate the input buffer from our source.
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2035 if hasattr(self._source, "read"):
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2036 data = self._source.read(self._read_size)
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2037
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2038 if not data:
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2039 self._finished_input = True
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2040 return
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2041
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2042 self._source_buffer = ffi.from_buffer(data)
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2043 self._in_buffer.src = self._source_buffer
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2044 self._in_buffer.size = len(self._source_buffer)
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2045 self._in_buffer.pos = 0
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2046 else:
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2047 self._source_buffer = ffi.from_buffer(self._source)
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2048 self._in_buffer.src = self._source_buffer
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2049 self._in_buffer.size = len(self._source_buffer)
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2050 self._in_buffer.pos = 0
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2051
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2052 def _decompress_into_buffer(self, out_buffer):
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2053 """Decompress available input into an output buffer.
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2054
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2055 Returns True if data in output buffer should be emitted.
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2056 """
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2057 zresult = lib.ZSTD_decompressStream(
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2058 self._decompressor._dctx, out_buffer, self._in_buffer
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2059 )
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2060
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2061 if self._in_buffer.pos == self._in_buffer.size:
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2062 self._in_buffer.src = ffi.NULL
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2063 self._in_buffer.pos = 0
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2064 self._in_buffer.size = 0
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2065 self._source_buffer = None
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2066
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2067 if not hasattr(self._source, "read"):
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2068 self._finished_input = True
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2069
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2070 if lib.ZSTD_isError(zresult):
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2071 raise ZstdError("zstd decompress error: %s" % _zstd_error(zresult))
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2072
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2073 # Emit data if there is data AND either:
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2074 # a) output buffer is full (read amount is satisfied)
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2075 # b) we're at end of a frame and not in frame spanning mode
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2076 return out_buffer.pos and (
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2077 out_buffer.pos == out_buffer.size
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2078 or zresult == 0
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2079 and not self._read_across_frames
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2080 )
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2081
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2082 def read(self, size=-1):
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2083 if self._closed:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2084 raise ValueError("stream is closed")
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2085
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2086 if size < -1:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2087 raise ValueError("cannot read negative amounts less than -1")
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2088
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2089 if size == -1:
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2090 # This is recursive. But it gets the job done.
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2091 return self.readall()
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2092
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2093 if self._finished_output or size == 0:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2094 return b""
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2095
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2096 # We /could/ call into readinto() here. But that introduces more
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2097 # overhead.
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2098 dst_buffer = ffi.new("char[]", size)
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2099 out_buffer = ffi.new("ZSTD_outBuffer *")
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2100 out_buffer.dst = dst_buffer
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2101 out_buffer.size = size
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2102 out_buffer.pos = 0
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2103
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2104 self._read_input()
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2105 if self._decompress_into_buffer(out_buffer):
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2106 self._bytes_decompressed += out_buffer.pos
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2107 return ffi.buffer(out_buffer.dst, out_buffer.pos)[:]
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2108
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2109 while not self._finished_input:
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2110 self._read_input()
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2111 if self._decompress_into_buffer(out_buffer):
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2112 self._bytes_decompressed += out_buffer.pos
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2113 return ffi.buffer(out_buffer.dst, out_buffer.pos)[:]
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2114
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2115 self._bytes_decompressed += out_buffer.pos
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2116 return ffi.buffer(out_buffer.dst, out_buffer.pos)[:]
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2117
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2118 def readinto(self, b):
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2119 if self._closed:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2120 raise ValueError("stream is closed")
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2121
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2122 if self._finished_output:
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2123 return 0
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2124
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2125 # TODO use writable=True once we require CFFI >= 1.12.
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2126 dest_buffer = ffi.from_buffer(b)
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2127 ffi.memmove(b, b"", 0)
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2128 out_buffer = ffi.new("ZSTD_outBuffer *")
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2129 out_buffer.dst = dest_buffer
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2130 out_buffer.size = len(dest_buffer)
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2131 out_buffer.pos = 0
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2132
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2133 self._read_input()
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2134 if self._decompress_into_buffer(out_buffer):
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2135 self._bytes_decompressed += out_buffer.pos
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2136 return out_buffer.pos
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2137
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2138 while not self._finished_input:
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2139 self._read_input()
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2140 if self._decompress_into_buffer(out_buffer):
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2141 self._bytes_decompressed += out_buffer.pos
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2142 return out_buffer.pos
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2143
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2144 self._bytes_decompressed += out_buffer.pos
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2145 return out_buffer.pos
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2146
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2147 def read1(self, size=-1):
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2148 if self._closed:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2149 raise ValueError("stream is closed")
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2150
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2151 if size < -1:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2152 raise ValueError("cannot read negative amounts less than -1")
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2153
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2154 if self._finished_output or size == 0:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2155 return b""
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2156
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2157 # -1 returns arbitrary number of bytes.
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2158 if size == -1:
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2159 size = DECOMPRESSION_RECOMMENDED_OUTPUT_SIZE
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2160
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2161 dst_buffer = ffi.new("char[]", size)
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2162 out_buffer = ffi.new("ZSTD_outBuffer *")
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2163 out_buffer.dst = dst_buffer
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2164 out_buffer.size = size
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2165 out_buffer.pos = 0
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2166
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2167 # read1() dictates that we can perform at most 1 call to underlying
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2168 # stream to get input. However, we can't satisfy this restriction with
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2169 # decompression because not all input generates output. So we allow
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2170 # multiple read(). But unlike read(), we stop once we have any output.
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2171 while not self._finished_input:
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2172 self._read_input()
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2173 self._decompress_into_buffer(out_buffer)
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2174
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2175 if out_buffer.pos:
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2176 break
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2177
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2178 self._bytes_decompressed += out_buffer.pos
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2179 return ffi.buffer(out_buffer.dst, out_buffer.pos)[:]
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2180
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2181 def readinto1(self, b):
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2182 if self._closed:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2183 raise ValueError("stream is closed")
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2184
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2185 if self._finished_output:
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2186 return 0
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2187
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2188 # TODO use writable=True once we require CFFI >= 1.12.
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2189 dest_buffer = ffi.from_buffer(b)
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2190 ffi.memmove(b, b"", 0)
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2191
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2192 out_buffer = ffi.new("ZSTD_outBuffer *")
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2193 out_buffer.dst = dest_buffer
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2194 out_buffer.size = len(dest_buffer)
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2195 out_buffer.pos = 0
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2196
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2197 while not self._finished_input and not self._finished_output:
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2198 self._read_input()
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2199 self._decompress_into_buffer(out_buffer)
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2200
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2201 if out_buffer.pos:
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2202 break
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2203
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2204 self._bytes_decompressed += out_buffer.pos
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2205 return out_buffer.pos
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2206
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2207 def seek(self, pos, whence=os.SEEK_SET):
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2208 if self._closed:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2209 raise ValueError("stream is closed")
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2210
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2211 read_amount = 0
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2212
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2213 if whence == os.SEEK_SET:
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2214 if pos < 0:
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	2215 raise ValueError(
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	2216 "cannot seek to negative position with SEEK_SET"
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	2217 )
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2218
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2219 if pos < self._bytes_decompressed:
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	2220 raise ValueError(
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	2221 "cannot seek zstd decompression stream " "backwards"
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	2222 )
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2223
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2224 read_amount = pos - self._bytes_decompressed
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2225
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2226 elif whence == os.SEEK_CUR:
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2227 if pos < 0:
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	2228 raise ValueError(
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	2229 "cannot seek zstd decompression stream " "backwards"
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	2230 )
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2231
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2232 read_amount = pos
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2233 elif whence == os.SEEK_END:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2234 raise ValueError(
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2235 "zstd decompression streams cannot be seeked " "with SEEK_END"
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2236 )
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2237
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2238 while read_amount:
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	2239 result = self.read(
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	2240 min(read_amount, DECOMPRESSION_RECOMMENDED_OUTPUT_SIZE)
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	2241 )
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2242
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2243 if not result:
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2244 break
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2245
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2246 read_amount -= len(result)
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2247
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2248 return self._bytes_decompressed
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2249
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2250
52666 9bd6854aab86 pyupgrade: convert to new style classes Matt Harbison <matt_harbison@yahoo.com> parents: 51762 diff changeset	2251 class ZstdDecompressionWriter:
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2252 def __init__(self, decompressor, writer, write_size, write_return_read):
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2253 decompressor._ensure_dctx()
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2254
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2255 self._decompressor = decompressor
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2256 self._writer = writer
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2257 self._write_size = write_size
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2258 self._write_return_read = bool(write_return_read)
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2259 self._entered = False
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2260 self._closed = False
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2261
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2262 def __enter__(self):
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2263 if self._closed:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2264 raise ValueError("stream is closed")
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2265
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2266 if self._entered:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2267 raise ZstdError("cannot __enter__ multiple times")
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2268
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2269 self._entered = True
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2270
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2271 return self
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2272
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2273 def __exit__(self, exc_type, exc_value, exc_tb):
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2274 self._entered = False
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2275 self.close()
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2276
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2277 def memory_size(self):
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2278 return lib.ZSTD_sizeof_DCtx(self._decompressor._dctx)
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2279
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2280 def close(self):
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2281 if self._closed:
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2282 return
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2283
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2284 try:
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2285 self.flush()
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2286 finally:
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2287 self._closed = True
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2288
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2289 f = getattr(self._writer, "close", None)
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2290 if f:
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2291 f()
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2292
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2293 @property
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2294 def closed(self):
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2295 return self._closed
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2296
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2297 def fileno(self):
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2298 f = getattr(self._writer, "fileno", None)
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2299 if f:
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2300 return f()
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2301 else:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2302 raise OSError("fileno not available on underlying writer")
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2303
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2304 def flush(self):
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2305 if self._closed:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2306 raise ValueError("stream is closed")
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2307
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2308 f = getattr(self._writer, "flush", None)
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2309 if f:
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2310 return f()
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2311
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2312 def isatty(self):
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2313 return False
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2314
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2315 def readable(self):
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2316 return False
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2317
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2318 def readline(self, size=-1):
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2319 raise io.UnsupportedOperation()
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2320
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2321 def readlines(self, hint=-1):
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2322 raise io.UnsupportedOperation()
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2323
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2324 def seek(self, offset, whence=None):
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2325 raise io.UnsupportedOperation()
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2326
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2327 def seekable(self):
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2328 return False
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2329
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2330 def tell(self):
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2331 raise io.UnsupportedOperation()
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2332
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2333 def truncate(self, size=None):
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2334 raise io.UnsupportedOperation()
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2335
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2336 def writable(self):
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2337 return True
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2338
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2339 def writelines(self, lines):
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2340 raise io.UnsupportedOperation()
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2341
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2342 def read(self, size=-1):
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2343 raise io.UnsupportedOperation()
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2344
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2345 def readall(self):
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2346 raise io.UnsupportedOperation()
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2347
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2348 def readinto(self, b):
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2349 raise io.UnsupportedOperation()
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2350
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2351 def write(self, data):
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2352 if self._closed:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2353 raise ValueError("stream is closed")
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2354
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2355 total_write = 0
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2356
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2357 in_buffer = ffi.new("ZSTD_inBuffer *")
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2358 out_buffer = ffi.new("ZSTD_outBuffer *")
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2359
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2360 data_buffer = ffi.from_buffer(data)
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2361 in_buffer.src = data_buffer
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2362 in_buffer.size = len(data_buffer)
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2363 in_buffer.pos = 0
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2364
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2365 dst_buffer = ffi.new("char[]", self._write_size)
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2366 out_buffer.dst = dst_buffer
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2367 out_buffer.size = len(dst_buffer)
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2368 out_buffer.pos = 0
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2369
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2370 dctx = self._decompressor._dctx
31799 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30924 diff changeset	2371
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2372 while in_buffer.pos < in_buffer.size:
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2373 zresult = lib.ZSTD_decompressStream(dctx, out_buffer, in_buffer)
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2374 if lib.ZSTD_isError(zresult):
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	2375 raise ZstdError(
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	2376 "zstd decompress error: %s" % _zstd_error(zresult)
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	2377 )
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2378
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2379 if out_buffer.pos:
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	2380 self._writer.write(
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	2381 ffi.buffer(out_buffer.dst, out_buffer.pos)[:]
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	2382 )
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2383 total_write += out_buffer.pos
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2384 out_buffer.pos = 0
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2385
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2386 if self._write_return_read:
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2387 return in_buffer.pos
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2388 else:
675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2389 return total_write
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2390
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2391
52666 9bd6854aab86 pyupgrade: convert to new style classes Matt Harbison <matt_harbison@yahoo.com> parents: 51762 diff changeset	2392 class ZstdDecompressor:
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2393 def __init__(self, dict_data=None, max_window_size=0, format=FORMAT_ZSTD1):
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2394 self._dict_data = dict_data
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2395 self._max_window_size = max_window_size
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2396 self._format = format
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2397
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2398 dctx = lib.ZSTD_createDCtx()
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2399 if dctx == ffi.NULL:
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2400 raise MemoryError()
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2401
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2402 self._dctx = dctx
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2403
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2404 # Defer setting up garbage collection until full state is loaded so
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2405 # the memory size is more accurate.
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2406 try:
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2407 self._ensure_dctx()
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2408 finally:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2409 self._dctx = ffi.gc(
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2410 dctx, lib.ZSTD_freeDCtx, size=lib.ZSTD_sizeof_DCtx(dctx)
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2411 )
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2412
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2413 def memory_size(self):
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2414 return lib.ZSTD_sizeof_DCtx(self._dctx)
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2415
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2416 def decompress(self, data, max_output_size=0):
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2417 self._ensure_dctx()
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2418
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2419 data_buffer = ffi.from_buffer(data)
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2420
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	2421 output_size = lib.ZSTD_getFrameContentSize(
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	2422 data_buffer, len(data_buffer)
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	2423 )
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2424
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2425 if output_size == lib.ZSTD_CONTENTSIZE_ERROR:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2426 raise ZstdError("error determining content size from frame header")
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2427 elif output_size == 0:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2428 return b""
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2429 elif output_size == lib.ZSTD_CONTENTSIZE_UNKNOWN:
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2430 if not max_output_size:
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	2431 raise ZstdError(
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	2432 "could not determine content size in frame header"
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	2433 )
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2434
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2435 result_buffer = ffi.new("char[]", max_output_size)
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2436 result_size = max_output_size
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2437 output_size = 0
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2438 else:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2439 result_buffer = ffi.new("char[]", output_size)
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2440 result_size = output_size
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2441
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2442 out_buffer = ffi.new("ZSTD_outBuffer *")
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2443 out_buffer.dst = result_buffer
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2444 out_buffer.size = result_size
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2445 out_buffer.pos = 0
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2446
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2447 in_buffer = ffi.new("ZSTD_inBuffer *")
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2448 in_buffer.src = data_buffer
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2449 in_buffer.size = len(data_buffer)
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2450 in_buffer.pos = 0
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2451
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2452 zresult = lib.ZSTD_decompressStream(self._dctx, out_buffer, in_buffer)
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2453 if lib.ZSTD_isError(zresult):
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2454 raise ZstdError("decompression error: %s" % _zstd_error(zresult))
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2455 elif zresult:
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	2456 raise ZstdError(
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	2457 "decompression error: did not decompress full frame"
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	2458 )
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2459 elif output_size and out_buffer.pos != output_size:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2460 raise ZstdError(
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2461 "decompression error: decompressed %d bytes; expected %d"
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2462 % (zresult, output_size)
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2463 )
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2464
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2465 return ffi.buffer(result_buffer, out_buffer.pos)[:]
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2466
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2467 def stream_reader(
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2468 self,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2469 source,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2470 read_size=DECOMPRESSION_RECOMMENDED_INPUT_SIZE,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2471 read_across_frames=False,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2472 ):
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2473 self._ensure_dctx()
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	2474 return ZstdDecompressionReader(
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	2475 self, source, read_size, read_across_frames
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	2476 )
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2477
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2478 def decompressobj(self, write_size=DECOMPRESSION_RECOMMENDED_OUTPUT_SIZE):
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2479 if write_size < 1:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2480 raise ValueError("write_size must be positive")
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2481
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2482 self._ensure_dctx()
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2483 return ZstdDecompressionObj(self, write_size=write_size)
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2484
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2485 def read_to_iter(
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2486 self,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2487 reader,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2488 read_size=DECOMPRESSION_RECOMMENDED_INPUT_SIZE,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2489 write_size=DECOMPRESSION_RECOMMENDED_OUTPUT_SIZE,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2490 skip_bytes=0,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2491 ):
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2492 if skip_bytes >= read_size:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2493 raise ValueError("skip_bytes must be smaller than read_size")
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2494
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2495 if hasattr(reader, "read"):
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2496 have_read = True
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2497 elif hasattr(reader, "__getitem__"):
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2498 have_read = False
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2499 buffer_offset = 0
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2500 size = len(reader)
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2501 else:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2502 raise ValueError(
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2503 "must pass an object with a read() method or "
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2504 "conforms to buffer protocol"
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2505 )
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2506
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2507 if skip_bytes:
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2508 if have_read:
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2509 reader.read(skip_bytes)
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2510 else:
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2511 if skip_bytes > size:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2512 raise ValueError("skip_bytes larger than first input chunk")
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2513
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2514 buffer_offset = skip_bytes
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2515
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2516 self._ensure_dctx()
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2517
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2518 in_buffer = ffi.new("ZSTD_inBuffer *")
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2519 out_buffer = ffi.new("ZSTD_outBuffer *")
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2520
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2521 dst_buffer = ffi.new("char[]", write_size)
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2522 out_buffer.dst = dst_buffer
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2523 out_buffer.size = len(dst_buffer)
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2524 out_buffer.pos = 0
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2525
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2526 while True:
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2527 assert out_buffer.pos == 0
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2528
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2529 if have_read:
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2530 read_result = reader.read(read_size)
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2531 else:
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2532 remaining = size - buffer_offset
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2533 slice_size = min(remaining, read_size)
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2534 read_result = reader[buffer_offset : buffer_offset + slice_size]
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2535 buffer_offset += slice_size
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2536
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2537 # No new input. Break out of read loop.
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2538 if not read_result:
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2539 break
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2540
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2541 # Feed all read data into decompressor and emit output until
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2542 # exhausted.
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2543 read_buffer = ffi.from_buffer(read_result)
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2544 in_buffer.src = read_buffer
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2545 in_buffer.size = len(read_buffer)
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2546 in_buffer.pos = 0
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2547
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2548 while in_buffer.pos < in_buffer.size:
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2549 assert out_buffer.pos == 0
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2550
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	2551 zresult = lib.ZSTD_decompressStream(
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	2552 self._dctx, out_buffer, in_buffer
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	2553 )
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2554 if lib.ZSTD_isError(zresult):
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	2555 raise ZstdError(
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	2556 "zstd decompress error: %s" % _zstd_error(zresult)
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	2557 )
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2558
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2559 if out_buffer.pos:
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2560 data = ffi.buffer(out_buffer.dst, out_buffer.pos)[:]
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2561 out_buffer.pos = 0
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2562 yield data
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2563
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2564 if zresult == 0:
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2565 return
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2566
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2567 # Repeat loop to collect more input data.
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2568 continue
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2569
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2570 # If we get here, input is exhausted.
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2571
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2572 read_from = read_to_iter
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2573
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2574 def stream_writer(
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2575 self,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2576 writer,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2577 write_size=DECOMPRESSION_RECOMMENDED_OUTPUT_SIZE,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2578 write_return_read=False,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2579 ):
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2580 if not hasattr(writer, "write"):
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2581 raise ValueError("must pass an object with a write() method")
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2582
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	2583 return ZstdDecompressionWriter(
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	2584 self, writer, write_size, write_return_read
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	2585 )
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2586
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2587 write_to = stream_writer
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2588
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2589 def copy_stream(
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2590 self,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2591 ifh,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2592 ofh,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2593 read_size=DECOMPRESSION_RECOMMENDED_INPUT_SIZE,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2594 write_size=DECOMPRESSION_RECOMMENDED_OUTPUT_SIZE,
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2595 ):
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2596 if not hasattr(ifh, "read"):
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2597 raise ValueError("first argument must have a read() method")
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2598 if not hasattr(ofh, "write"):
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2599 raise ValueError("second argument must have a write() method")
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2600
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2601 self._ensure_dctx()
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2602
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2603 in_buffer = ffi.new("ZSTD_inBuffer *")
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2604 out_buffer = ffi.new("ZSTD_outBuffer *")
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2605
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2606 dst_buffer = ffi.new("char[]", write_size)
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2607 out_buffer.dst = dst_buffer
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2608 out_buffer.size = write_size
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2609 out_buffer.pos = 0
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2610
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2611 total_read, total_write = 0, 0
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2612
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2613 # Read all available input.
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2614 while True:
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2615 data = ifh.read(read_size)
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2616 if not data:
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2617 break
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2618
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2619 data_buffer = ffi.from_buffer(data)
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2620 total_read += len(data_buffer)
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2621 in_buffer.src = data_buffer
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2622 in_buffer.size = len(data_buffer)
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2623 in_buffer.pos = 0
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2624
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2625 # Flush all read data to output.
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2626 while in_buffer.pos < in_buffer.size:
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	2627 zresult = lib.ZSTD_decompressStream(
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	2628 self._dctx, out_buffer, in_buffer
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	2629 )
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2630 if lib.ZSTD_isError(zresult):
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2631 raise ZstdError(
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2632 "zstd decompressor error: %s" % _zstd_error(zresult)
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2633 )
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2634
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2635 if out_buffer.pos:
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2636 ofh.write(ffi.buffer(out_buffer.dst, out_buffer.pos))
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2637 total_write += out_buffer.pos
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2638 out_buffer.pos = 0
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2639
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2640 # Continue loop to keep reading.
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2641
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2642 return total_read, total_write
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2643
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2644 def decompress_content_dict_chain(self, frames):
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2645 if not isinstance(frames, list):
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2646 raise TypeError("argument must be a list")
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2647
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2648 if not frames:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2649 raise ValueError("empty input chain")
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2650
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2651 # First chunk should not be using a dictionary. We handle it specially.
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2652 chunk = frames[0]
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2653 if not isinstance(chunk, bytes_type):
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2654 raise ValueError("chunk 0 must be bytes")
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2655
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2656 # All chunks should be zstd frames and should have content size set.
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2657 chunk_buffer = ffi.from_buffer(chunk)
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2658 params = ffi.new("ZSTD_frameHeader *")
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	2659 zresult = lib.ZSTD_getFrameHeader(
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	2660 params, chunk_buffer, len(chunk_buffer)
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	2661 )
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2662 if lib.ZSTD_isError(zresult):
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2663 raise ValueError("chunk 0 is not a valid zstd frame")
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2664 elif zresult:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2665 raise ValueError("chunk 0 is too small to contain a zstd frame")
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2666
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2667 if params.frameContentSize == lib.ZSTD_CONTENTSIZE_UNKNOWN:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2668 raise ValueError("chunk 0 missing content size in frame")
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2669
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2670 self._ensure_dctx(load_dict=False)
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2671
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2672 last_buffer = ffi.new("char[]", params.frameContentSize)
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2673
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2674 out_buffer = ffi.new("ZSTD_outBuffer *")
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2675 out_buffer.dst = last_buffer
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2676 out_buffer.size = len(last_buffer)
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2677 out_buffer.pos = 0
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2678
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2679 in_buffer = ffi.new("ZSTD_inBuffer *")
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2680 in_buffer.src = chunk_buffer
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2681 in_buffer.size = len(chunk_buffer)
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2682 in_buffer.pos = 0
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2683
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2684 zresult = lib.ZSTD_decompressStream(self._dctx, out_buffer, in_buffer)
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2685 if lib.ZSTD_isError(zresult):
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	2686 raise ZstdError(
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	2687 "could not decompress chunk 0: %s" % _zstd_error(zresult)
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	2688 )
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2689 elif zresult:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2690 raise ZstdError("chunk 0 did not decompress full frame")
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2691
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2692 # Special case of chain length of 1
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2693 if len(frames) == 1:
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2694 return ffi.buffer(last_buffer, len(last_buffer))[:]
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2695
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2696 i = 1
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2697 while i < len(frames):
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2698 chunk = frames[i]
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2699 if not isinstance(chunk, bytes_type):
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2700 raise ValueError("chunk %d must be bytes" % i)
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2701
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2702 chunk_buffer = ffi.from_buffer(chunk)
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	2703 zresult = lib.ZSTD_getFrameHeader(
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	2704 params, chunk_buffer, len(chunk_buffer)
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	2705 )
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2706 if lib.ZSTD_isError(zresult):
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2707 raise ValueError("chunk %d is not a valid zstd frame" % i)
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2708 elif zresult:
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	2709 raise ValueError(
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	2710 "chunk %d is too small to contain a zstd frame" % i
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	2711 )
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2712
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2713 if params.frameContentSize == lib.ZSTD_CONTENTSIZE_UNKNOWN:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2714 raise ValueError("chunk %d missing content size in frame" % i)
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2715
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2716 dest_buffer = ffi.new("char[]", params.frameContentSize)
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2717
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2718 out_buffer.dst = dest_buffer
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2719 out_buffer.size = len(dest_buffer)
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2720 out_buffer.pos = 0
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2721
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2722 in_buffer.src = chunk_buffer
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2723 in_buffer.size = len(chunk_buffer)
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2724 in_buffer.pos = 0
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2725
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	2726 zresult = lib.ZSTD_decompressStream(
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	2727 self._dctx, out_buffer, in_buffer
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	2728 )
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2729 if lib.ZSTD_isError(zresult):
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2730 raise ZstdError(
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2731 "could not decompress chunk %d: %s" % _zstd_error(zresult)
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2732 )
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2733 elif zresult:
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2734 raise ZstdError("chunk %d did not decompress full frame" % i)
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2735
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2736 last_buffer = dest_buffer
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2737 i += 1
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2738
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2739 return ffi.buffer(last_buffer, len(last_buffer))[:]
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2740
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2741 def _ensure_dctx(self, load_dict=True):
42070 675775c33ab6 zstandard: vendor python-zstandard 0.11 Gregory Szorc <gregory.szorc@gmail.com> parents: 40122 diff changeset	2742 lib.ZSTD_DCtx_reset(self._dctx, lib.ZSTD_reset_session_only)
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2743
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2744 if self._max_window_size:
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	2745 zresult = lib.ZSTD_DCtx_setMaxWindowSize(
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	2746 self._dctx, self._max_window_size
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	2747 )
31799 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30924 diff changeset	2748 if lib.ZSTD_isError(zresult):
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2749 raise ZstdError(
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2750 "unable to set max window size: %s" % _zstd_error(zresult)
de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2751 )
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2752
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2753 zresult = lib.ZSTD_DCtx_setFormat(self._dctx, self._format)
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2754 if lib.ZSTD_isError(zresult):
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	2755 raise ZstdError(
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	2756 "unable to set decoding format: %s" % _zstd_error(zresult)
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	2757 )
30924 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30444 diff changeset	2758
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2759 if self._dict_data and load_dict:
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2760 zresult = lib.ZSTD_DCtx_refDDict(self._dctx, self._dict_data._ddict)
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31799 diff changeset	2761 if lib.ZSTD_isError(zresult):
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2762 raise ZstdError(
44232 5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	2763 "unable to reference prepared dictionary: %s"
5e84a96d865b python-zstandard: blacken at 80 characters Gregory Szorc <gregory.szorc@gmail.com> parents: 43999 diff changeset	2764 % _zstd_error(zresult)
43999 de7838053207 zstandard: vendor python-zstandard 0.13.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 42941 diff changeset	2765 )

Mercurial > public > mercurial-scm > hg-stable

annotate contrib/python-zstandard/zstandard/cffi.py @ 53040:cdd7bf612c7b stable tip