mercurial-scm/hg: contrib/python-zstandard/c-ext/compressor.c annotate

annotate contrib/python-zstandard/c-ext/compressor.c @ 30895:c32454d69b85

zstd: vendor python-zstandard 0.7.0 Commit 3054ae3a66112970a091d3939fee32c2d0c1a23e from https://github.com/indygreg/python-zstandard is imported without modifications (other than removing unwanted files). The vendored zstd library within has been upgraded from 1.1.2 to 1.1.3. This version introduced new APIs for threads, thread pools, multi-threaded compression, and a new dictionary builder (COVER). These features are not yet used by python-zstandard (or Mercurial for that matter). However, that will likely change in the next python-zstandard release (and I think there are opportunities for Mercurial to take advantage of the multi-threaded APIs). Relevant to Mercurial, the CFFI bindings are now fully implemented. This means zstd should "just work" with PyPy (although I haven't tried). The python-zstandard test suite also runs all tests against both the C extension and CFFI bindings to ensure feature parity. There is also a "decompress_content_dict_chain()" API. This was derived from discussions with Yann Collet on list about alternate ways of encoding delta chains. The change most relevant to Mercurial is a performance enhancement in the simple decompression API to reuse a data structure across operations. This makes decompression of multiple inputs significantly faster. (This scenario occurs when reading revlog delta chains, for example.) Using python-zstandard's bench.py to measure the performance difference... On changelog chunks in the mozilla-unified repo: decompress discrete decompress() reuse zctx 1.262243 wall; 1.260000 CPU; 1.260000 user; 0.000000 sys 170.43 MB/s (best of 3) 0.949106 wall; 0.950000 CPU; 0.950000 user; 0.000000 sys 226.66 MB/s (best of 4) decompress discrete dict decompress() reuse zctx 0.692170 wall; 0.690000 CPU; 0.690000 user; 0.000000 sys 310.80 MB/s (best of 5) 0.437088 wall; 0.440000 CPU; 0.440000 user; 0.000000 sys 492.17 MB/s (best of 7) On manifest chunks in the mozilla-unified repo: decompress discrete decompress() reuse zctx 1.367284 wall; 1.370000 CPU; 1.370000 user; 0.000000 sys 274.01 MB/s (best of 3) 1.086831 wall; 1.080000 CPU; 1.080000 user; 0.000000 sys 344.72 MB/s (best of 3) decompress discrete dict decompress() reuse zctx 0.993272 wall; 0.990000 CPU; 0.990000 user; 0.000000 sys 377.19 MB/s (best of 3) 0.678651 wall; 0.680000 CPU; 0.680000 user; 0.000000 sys 552.06 MB/s (best of 5) That should make reads on zstd revlogs a bit faster ;) # no-check-commit

author	Gregory Szorc <gregory.szorc@gmail.com>
date	Tue, 07 Feb 2017 23:24:47 -0800
parents	08fa3a76a080
children	e0dc40530c5a

rev	line source
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1 /**
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	2 * Copyright (c) 2016-present, Gregory Szorc
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	3 * All rights reserved.
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	4 *
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	5 * This software may be modified and distributed under the terms
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	6 * of the BSD license. See the LICENSE file for details.
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	7 */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	8
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	9 #include "python-zstandard.h"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	10
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	11 extern PyObject* ZstdError;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	12
30822 b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	13 int populate_cdict(ZstdCompressor* compressor, void* dictData, size_t dictSize, ZSTD_parameters* zparams) {
b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	14 ZSTD_customMem zmem;
b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	15 assert(!compressor->cdict);
b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	16 Py_BEGIN_ALLOW_THREADS
b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	17 memset(&zmem, 0, sizeof(zmem));
b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	18 compressor->cdict = ZSTD_createCDict_advanced(compressor->dict->dictData,
30895 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30830 diff changeset	19 compressor->dict->dictSize, 1, *zparams, zmem);
30822 b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	20 Py_END_ALLOW_THREADS
b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	21
b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	22 if (!compressor->cdict) {
b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	23 PyErr_SetString(ZstdError, "could not create compression dictionary");
b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	24 return 1;
b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	25 }
b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	26
b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	27 return 0;
b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	28 }
b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	29
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	30 /**
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	31 * Initialize a zstd CStream from a ZstdCompressor instance.
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	32 *
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	33 * Returns a ZSTD_CStream on success or NULL on failure. If NULL, a Python
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	34 * exception will be set.
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	35 */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	36 ZSTD_CStream* CStream_from_ZstdCompressor(ZstdCompressor* compressor, Py_ssize_t sourceSize) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	37 ZSTD_CStream* cstream;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	38 ZSTD_parameters zparams;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	39 void* dictData = NULL;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	40 size_t dictSize = 0;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	41 size_t zresult;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	42
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	43 cstream = ZSTD_createCStream();
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	44 if (!cstream) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	45 PyErr_SetString(ZstdError, "cannot create CStream");
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	46 return NULL;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	47 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	48
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	49 if (compressor->dict) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	50 dictData = compressor->dict->dictData;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	51 dictSize = compressor->dict->dictSize;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	52 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	53
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	54 memset(&zparams, 0, sizeof(zparams));
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	55 if (compressor->cparams) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	56 ztopy_compression_parameters(compressor->cparams, &zparams.cParams);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	57 /* Do NOT call ZSTD_adjustCParams() here because the compression params
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	58 come from the user. */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	59 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	60 else {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	61 zparams.cParams = ZSTD_getCParams(compressor->compressionLevel, sourceSize, dictSize);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	62 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	63
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	64 zparams.fParams = compressor->fparams;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	65
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	66 zresult = ZSTD_initCStream_advanced(cstream, dictData, dictSize, zparams, sourceSize);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	67
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	68 if (ZSTD_isError(zresult)) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	69 ZSTD_freeCStream(cstream);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	70 PyErr_Format(ZstdError, "cannot init CStream: %s", ZSTD_getErrorName(zresult));
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	71 return NULL;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	72 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	73
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	74 return cstream;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	75 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	76
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	77 PyDoc_STRVAR(ZstdCompressor__doc__,
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	78 "ZstdCompressor(level=None, dict_data=None, compression_params=None)\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	79 "\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	80 "Create an object used to perform Zstandard compression.\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	81 "\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	82 "An instance can compress data various ways. Instances can be used multiple\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	83 "times. Each compression operation will use the compression parameters\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	84 "defined at construction time.\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	85 "\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	86 "Compression can be configured via the following names arguments:\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	87 "\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	88 "level\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	89 " Integer compression level.\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	90 "dict_data\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	91 " A ``ZstdCompressionDict`` to be used to compress with dictionary data.\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	92 "compression_params\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	93 " A ``CompressionParameters`` instance defining low-level compression"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	94 " parameters. If defined, this will overwrite the ``level`` argument.\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	95 "write_checksum\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	96 " If True, a 4 byte content checksum will be written with the compressed\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	97 " data, allowing the decompressor to perform content verification.\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	98 "write_content_size\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	99 " If True, the decompressed content size will be included in the header of\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	100 " the compressed data. This data will only be written if the compressor\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	101 " knows the size of the input data.\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	102 "write_dict_id\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	103 " Determines whether the dictionary ID will be written into the compressed\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	104 " data. Defaults to True. Only adds content to the compressed data if\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	105 " a dictionary is being used.\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	106 );
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	107
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	108 static int ZstdCompressor_init(ZstdCompressor* self, PyObject* args, PyObject* kwargs) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	109 static char* kwlist[] = {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	110 "level",
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	111 "dict_data",
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	112 "compression_params",
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	113 "write_checksum",
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	114 "write_content_size",
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	115 "write_dict_id",
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	116 NULL
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	117 };
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	118
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	119 int level = 3;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	120 ZstdCompressionDict* dict = NULL;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	121 CompressionParametersObject* params = NULL;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	122 PyObject* writeChecksum = NULL;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	123 PyObject* writeContentSize = NULL;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	124 PyObject* writeDictID = NULL;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	125
30822 b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	126 self->cctx = NULL;
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	127 self->dict = NULL;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	128 self->cparams = NULL;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	129 self->cdict = NULL;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	130
30895 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30830 diff changeset	131 if (!PyArg_ParseTupleAndKeywords(args, kwargs, "\|iO!O!OOO:ZstdCompressor",
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30830 diff changeset	132 kwlist, &level, &ZstdCompressionDictType, &dict,
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	133 &CompressionParametersType, &params,
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	134 &writeChecksum, &writeContentSize, &writeDictID)) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	135 return -1;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	136 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	137
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	138 if (level < 1) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	139 PyErr_SetString(PyExc_ValueError, "level must be greater than 0");
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	140 return -1;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	141 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	142
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	143 if (level > ZSTD_maxCLevel()) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	144 PyErr_Format(PyExc_ValueError, "level must be less than %d",
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	145 ZSTD_maxCLevel() + 1);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	146 return -1;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	147 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	148
30822 b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	149 /* We create a ZSTD_CCtx for reuse among multiple operations to reduce the
b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	150 overhead of each compression operation. */
b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	151 self->cctx = ZSTD_createCCtx();
b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	152 if (!self->cctx) {
b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	153 PyErr_NoMemory();
b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	154 return -1;
b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	155 }
b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	156
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	157 self->compressionLevel = level;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	158
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	159 if (dict) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	160 self->dict = dict;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	161 Py_INCREF(dict);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	162 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	163
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	164 if (params) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	165 self->cparams = params;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	166 Py_INCREF(params);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	167 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	168
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	169 memset(&self->fparams, 0, sizeof(self->fparams));
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	170
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	171 if (writeChecksum && PyObject_IsTrue(writeChecksum)) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	172 self->fparams.checksumFlag = 1;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	173 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	174 if (writeContentSize && PyObject_IsTrue(writeContentSize)) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	175 self->fparams.contentSizeFlag = 1;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	176 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	177 if (writeDictID && PyObject_Not(writeDictID)) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	178 self->fparams.noDictIDFlag = 1;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	179 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	180
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	181 return 0;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	182 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	183
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	184 static void ZstdCompressor_dealloc(ZstdCompressor* self) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	185 Py_XDECREF(self->cparams);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	186 Py_XDECREF(self->dict);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	187
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	188 if (self->cdict) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	189 ZSTD_freeCDict(self->cdict);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	190 self->cdict = NULL;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	191 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	192
30822 b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	193 if (self->cctx) {
b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	194 ZSTD_freeCCtx(self->cctx);
b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	195 self->cctx = NULL;
b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	196 }
b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	197
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	198 PyObject_Del(self);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	199 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	200
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	201 PyDoc_STRVAR(ZstdCompressor_copy_stream__doc__,
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	202 "copy_stream(ifh, ofh[, size=0, read_size=default, write_size=default])\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	203 "compress data between streams\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	204 "\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	205 "Data will be read from ``ifh``, compressed, and written to ``ofh``.\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	206 "``ifh`` must have a ``read(size)`` method. ``ofh`` must have a ``write(data)``\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	207 "method.\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	208 "\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	209 "An optional ``size`` argument specifies the size of the source stream.\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	210 "If defined, compression parameters will be tuned based on the size.\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	211 "\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	212 "Optional arguments ``read_size`` and ``write_size`` define the chunk sizes\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	213 "of ``read()`` and ``write()`` operations, respectively. By default, they use\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	214 "the default compression stream input and output sizes, respectively.\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	215 );
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	216
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	217 static PyObject* ZstdCompressor_copy_stream(ZstdCompressor* self, PyObject* args, PyObject* kwargs) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	218 static char* kwlist[] = {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	219 "ifh",
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	220 "ofh",
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	221 "size",
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	222 "read_size",
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	223 "write_size",
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	224 NULL
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	225 };
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	226
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	227 PyObject* source;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	228 PyObject* dest;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	229 Py_ssize_t sourceSize = 0;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	230 size_t inSize = ZSTD_CStreamInSize();
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	231 size_t outSize = ZSTD_CStreamOutSize();
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	232 ZSTD_CStream* cstream;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	233 ZSTD_inBuffer input;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	234 ZSTD_outBuffer output;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	235 Py_ssize_t totalRead = 0;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	236 Py_ssize_t totalWrite = 0;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	237 char* readBuffer;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	238 Py_ssize_t readSize;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	239 PyObject* readResult;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	240 PyObject* res = NULL;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	241 size_t zresult;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	242 PyObject* writeResult;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	243 PyObject* totalReadPy;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	244 PyObject* totalWritePy;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	245
30895 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30830 diff changeset	246 if (!PyArg_ParseTupleAndKeywords(args, kwargs, "OO\|nkk:copy_stream", kwlist,
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30830 diff changeset	247 &source, &dest, &sourceSize, &inSize, &outSize)) {
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	248 return NULL;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	249 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	250
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	251 if (!PyObject_HasAttrString(source, "read")) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	252 PyErr_SetString(PyExc_ValueError, "first argument must have a read() method");
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	253 return NULL;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	254 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	255
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	256 if (!PyObject_HasAttrString(dest, "write")) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	257 PyErr_SetString(PyExc_ValueError, "second argument must have a write() method");
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	258 return NULL;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	259 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	260
30830 08fa3a76a080 zstd: prevent potential free() of uninitialized memory Gregory Szorc <gregory.szorc@gmail.com> parents: 30822 diff changeset	261 /* Prevent free on uninitialized memory in finally. */
08fa3a76a080 zstd: prevent potential free() of uninitialized memory Gregory Szorc <gregory.szorc@gmail.com> parents: 30822 diff changeset	262 output.dst = NULL;
08fa3a76a080 zstd: prevent potential free() of uninitialized memory Gregory Szorc <gregory.szorc@gmail.com> parents: 30822 diff changeset	263
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	264 cstream = CStream_from_ZstdCompressor(self, sourceSize);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	265 if (!cstream) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	266 res = NULL;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	267 goto finally;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	268 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	269
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	270 output.dst = PyMem_Malloc(outSize);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	271 if (!output.dst) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	272 PyErr_NoMemory();
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	273 res = NULL;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	274 goto finally;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	275 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	276 output.size = outSize;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	277 output.pos = 0;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	278
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	279 while (1) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	280 /* Try to read from source stream. */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	281 readResult = PyObject_CallMethod(source, "read", "n", inSize);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	282 if (!readResult) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	283 PyErr_SetString(ZstdError, "could not read() from source");
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	284 goto finally;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	285 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	286
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	287 PyBytes_AsStringAndSize(readResult, &readBuffer, &readSize);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	288
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	289 /* If no data was read, we're at EOF. */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	290 if (0 == readSize) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	291 break;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	292 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	293
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	294 totalRead += readSize;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	295
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	296 /* Send data to compressor */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	297 input.src = readBuffer;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	298 input.size = readSize;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	299 input.pos = 0;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	300
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	301 while (input.pos < input.size) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	302 Py_BEGIN_ALLOW_THREADS
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	303 zresult = ZSTD_compressStream(cstream, &output, &input);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	304 Py_END_ALLOW_THREADS
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	305
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	306 if (ZSTD_isError(zresult)) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	307 res = NULL;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	308 PyErr_Format(ZstdError, "zstd compress error: %s", ZSTD_getErrorName(zresult));
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	309 goto finally;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	310 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	311
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	312 if (output.pos) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	313 #if PY_MAJOR_VERSION >= 3
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	314 writeResult = PyObject_CallMethod(dest, "write", "y#",
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	315 #else
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	316 writeResult = PyObject_CallMethod(dest, "write", "s#",
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	317 #endif
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	318 output.dst, output.pos);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	319 Py_XDECREF(writeResult);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	320 totalWrite += output.pos;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	321 output.pos = 0;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	322 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	323 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	324 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	325
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	326 /* We've finished reading. Now flush the compressor stream. */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	327 while (1) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	328 zresult = ZSTD_endStream(cstream, &output);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	329 if (ZSTD_isError(zresult)) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	330 PyErr_Format(ZstdError, "error ending compression stream: %s",
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	331 ZSTD_getErrorName(zresult));
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	332 res = NULL;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	333 goto finally;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	334 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	335
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	336 if (output.pos) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	337 #if PY_MAJOR_VERSION >= 3
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	338 writeResult = PyObject_CallMethod(dest, "write", "y#",
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	339 #else
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	340 writeResult = PyObject_CallMethod(dest, "write", "s#",
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	341 #endif
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	342 output.dst, output.pos);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	343 totalWrite += output.pos;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	344 Py_XDECREF(writeResult);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	345 output.pos = 0;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	346 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	347
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	348 if (!zresult) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	349 break;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	350 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	351 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	352
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	353 ZSTD_freeCStream(cstream);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	354 cstream = NULL;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	355
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	356 totalReadPy = PyLong_FromSsize_t(totalRead);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	357 totalWritePy = PyLong_FromSsize_t(totalWrite);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	358 res = PyTuple_Pack(2, totalReadPy, totalWritePy);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	359 Py_DecRef(totalReadPy);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	360 Py_DecRef(totalWritePy);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	361
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	362 finally:
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	363 if (output.dst) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	364 PyMem_Free(output.dst);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	365 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	366
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	367 if (cstream) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	368 ZSTD_freeCStream(cstream);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	369 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	370
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	371 return res;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	372 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	373
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	374 PyDoc_STRVAR(ZstdCompressor_compress__doc__,
30822 b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	375 "compress(data, allow_empty=False)\n"
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	376 "\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	377 "Compress data in a single operation.\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	378 "\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	379 "This is the simplest mechanism to perform compression: simply pass in a\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	380 "value and get a compressed value back. It is almost the most prone to abuse.\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	381 "The input and output values must fit in memory, so passing in very large\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	382 "values can result in excessive memory usage. For this reason, one of the\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	383 "streaming based APIs is preferred for larger values.\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	384 );
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	385
30822 b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	386 static PyObject* ZstdCompressor_compress(ZstdCompressor* self, PyObject* args, PyObject* kwargs) {
b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	387 static char* kwlist[] = {
b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	388 "data",
b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	389 "allow_empty",
b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	390 NULL
b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	391 };
b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	392
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	393 const char* source;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	394 Py_ssize_t sourceSize;
30822 b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	395 PyObject* allowEmpty = NULL;
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	396 size_t destSize;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	397 PyObject* output;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	398 char* dest;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	399 void* dictData = NULL;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	400 size_t dictSize = 0;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	401 size_t zresult;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	402 ZSTD_parameters zparams;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	403
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	404 #if PY_MAJOR_VERSION >= 3
30895 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30830 diff changeset	405 if (!PyArg_ParseTupleAndKeywords(args, kwargs, "y#\|O:compress",
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	406 #else
30895 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30830 diff changeset	407 if (!PyArg_ParseTupleAndKeywords(args, kwargs, "s#\|O:compress",
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	408 #endif
30822 b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	409 kwlist, &source, &sourceSize, &allowEmpty)) {
b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	410 return NULL;
b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	411 }
b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	412
b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	413 /* Limitation in zstd C API doesn't let decompression side distinguish
b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	414 between content size of 0 and unknown content size. This can make round
b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	415 tripping via Python difficult. Until this is fixed, require a flag
b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	416 to fire the footgun.
b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	417 https://github.com/indygreg/python-zstandard/issues/11 */
b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	418 if (0 == sourceSize && self->fparams.contentSizeFlag
b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	419 && (!allowEmpty \|\| PyObject_Not(allowEmpty))) {
b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	420 PyErr_SetString(PyExc_ValueError, "cannot write empty inputs when writing content sizes");
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	421 return NULL;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	422 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	423
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	424 destSize = ZSTD_compressBound(sourceSize);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	425 output = PyBytes_FromStringAndSize(NULL, destSize);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	426 if (!output) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	427 return NULL;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	428 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	429
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	430 dest = PyBytes_AsString(output);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	431
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	432 if (self->dict) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	433 dictData = self->dict->dictData;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	434 dictSize = self->dict->dictSize;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	435 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	436
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	437 memset(&zparams, 0, sizeof(zparams));
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	438 if (!self->cparams) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	439 zparams.cParams = ZSTD_getCParams(self->compressionLevel, sourceSize, dictSize);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	440 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	441 else {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	442 ztopy_compression_parameters(self->cparams, &zparams.cParams);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	443 /* Do NOT call ZSTD_adjustCParams() here because the compression params
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	444 come from the user. */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	445 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	446
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	447 zparams.fParams = self->fparams;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	448
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	449 /* The raw dict data has to be processed before it can be used. Since this
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	450 adds overhead - especially if multiple dictionary compression operations
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	451 are performed on the same ZstdCompressor instance - we create a
30822 b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	452 ZSTD_CDict once and reuse it for all operations.
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	453
30822 b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	454 Note: the compression parameters used for the first invocation (possibly
b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	455 derived from the source size) will be reused on all subsequent invocations.
b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	456 https://github.com/facebook/zstd/issues/358 contains more info. We could
b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	457 potentially add an argument somewhere to control this behavior.
b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	458 */
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	459 if (dictData && !self->cdict) {
30822 b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	460 if (populate_cdict(self, dictData, dictSize, &zparams)) {
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	461 Py_DECREF(output);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	462 return NULL;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	463 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	464 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	465
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	466 Py_BEGIN_ALLOW_THREADS
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	467 /* By avoiding ZSTD_compress(), we don't necessarily write out content
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	468 size. This means the argument to ZstdCompressor to control frame
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	469 parameters is honored. */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	470 if (self->cdict) {
30822 b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	471 zresult = ZSTD_compress_usingCDict(self->cctx, dest, destSize,
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	472 source, sourceSize, self->cdict);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	473 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	474 else {
30822 b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	475 zresult = ZSTD_compress_advanced(self->cctx, dest, destSize,
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	476 source, sourceSize, dictData, dictSize, zparams);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	477 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	478 Py_END_ALLOW_THREADS
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	479
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	480 if (ZSTD_isError(zresult)) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	481 PyErr_Format(ZstdError, "cannot compress: %s", ZSTD_getErrorName(zresult));
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	482 Py_CLEAR(output);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	483 return NULL;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	484 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	485 else {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	486 Py_SIZE(output) = zresult;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	487 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	488
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	489 return output;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	490 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	491
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	492 PyDoc_STRVAR(ZstdCompressionObj__doc__,
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	493 "compressobj()\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	494 "\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	495 "Return an object exposing ``compress(data)`` and ``flush()`` methods.\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	496 "\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	497 "The returned object exposes an API similar to ``zlib.compressobj`` and\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	498 "``bz2.BZ2Compressor`` so that callers can swap in the zstd compressor\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	499 "without changing how compression is performed.\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	500 );
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	501
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	502 static ZstdCompressionObj* ZstdCompressor_compressobj(ZstdCompressor* self, PyObject* args, PyObject* kwargs) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	503 static char* kwlist[] = {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	504 "size",
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	505 NULL
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	506 };
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	507
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	508 Py_ssize_t inSize = 0;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	509 size_t outSize = ZSTD_CStreamOutSize();
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	510 ZstdCompressionObj* result = PyObject_New(ZstdCompressionObj, &ZstdCompressionObjType);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	511 if (!result) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	512 return NULL;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	513 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	514
30895 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30830 diff changeset	515 if (!PyArg_ParseTupleAndKeywords(args, kwargs, "\|n:compressobj", kwlist, &inSize)) {
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	516 return NULL;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	517 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	518
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	519 result->cstream = CStream_from_ZstdCompressor(self, inSize);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	520 if (!result->cstream) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	521 Py_DECREF(result);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	522 return NULL;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	523 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	524
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	525 result->output.dst = PyMem_Malloc(outSize);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	526 if (!result->output.dst) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	527 PyErr_NoMemory();
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	528 Py_DECREF(result);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	529 return NULL;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	530 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	531 result->output.size = outSize;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	532 result->output.pos = 0;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	533
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	534 result->compressor = self;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	535 Py_INCREF(result->compressor);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	536
30822 b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	537 result->finished = 0;
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	538
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	539 return result;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	540 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	541
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	542 PyDoc_STRVAR(ZstdCompressor_read_from__doc__,
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	543 "read_from(reader, [size=0, read_size=default, write_size=default])\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	544 "Read uncompress data from a reader and return an iterator\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	545 "\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	546 "Returns an iterator of compressed data produced from reading from ``reader``.\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	547 "\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	548 "Uncompressed data will be obtained from ``reader`` by calling the\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	549 "``read(size)`` method of it. The source data will be streamed into a\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	550 "compressor. As compressed data is available, it will be exposed to the\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	551 "iterator.\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	552 "\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	553 "Data is read from the source in chunks of ``read_size``. Compressed chunks\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	554 "are at most ``write_size`` bytes. Both values default to the zstd input and\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	555 "and output defaults, respectively.\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	556 "\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	557 "The caller is partially in control of how fast data is fed into the\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	558 "compressor by how it consumes the returned iterator. The compressor will\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	559 "not consume from the reader unless the caller consumes from the iterator.\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	560 );
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	561
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	562 static ZstdCompressorIterator* ZstdCompressor_read_from(ZstdCompressor* self, PyObject* args, PyObject* kwargs) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	563 static char* kwlist[] = {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	564 "reader",
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	565 "size",
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	566 "read_size",
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	567 "write_size",
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	568 NULL
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	569 };
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	570
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	571 PyObject* reader;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	572 Py_ssize_t sourceSize = 0;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	573 size_t inSize = ZSTD_CStreamInSize();
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	574 size_t outSize = ZSTD_CStreamOutSize();
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	575 ZstdCompressorIterator* result;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	576
30895 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30830 diff changeset	577 if (!PyArg_ParseTupleAndKeywords(args, kwargs, "O\|nkk:read_from", kwlist,
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30830 diff changeset	578 &reader, &sourceSize, &inSize, &outSize)) {
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	579 return NULL;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	580 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	581
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	582 result = PyObject_New(ZstdCompressorIterator, &ZstdCompressorIteratorType);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	583 if (!result) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	584 return NULL;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	585 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	586
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	587 result->compressor = NULL;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	588 result->reader = NULL;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	589 result->buffer = NULL;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	590 result->cstream = NULL;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	591 result->input.src = NULL;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	592 result->output.dst = NULL;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	593 result->readResult = NULL;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	594
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	595 if (PyObject_HasAttrString(reader, "read")) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	596 result->reader = reader;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	597 Py_INCREF(result->reader);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	598 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	599 else if (1 == PyObject_CheckBuffer(reader)) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	600 result->buffer = PyMem_Malloc(sizeof(Py_buffer));
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	601 if (!result->buffer) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	602 goto except;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	603 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	604
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	605 memset(result->buffer, 0, sizeof(Py_buffer));
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	606
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	607 if (0 != PyObject_GetBuffer(reader, result->buffer, PyBUF_CONTIG_RO)) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	608 goto except;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	609 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	610
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	611 result->bufferOffset = 0;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	612 sourceSize = result->buffer->len;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	613 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	614 else {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	615 PyErr_SetString(PyExc_ValueError,
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	616 "must pass an object with a read() method or conforms to buffer protocol");
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	617 goto except;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	618 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	619
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	620 result->compressor = self;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	621 Py_INCREF(result->compressor);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	622
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	623 result->sourceSize = sourceSize;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	624 result->cstream = CStream_from_ZstdCompressor(self, sourceSize);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	625 if (!result->cstream) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	626 goto except;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	627 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	628
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	629 result->inSize = inSize;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	630 result->outSize = outSize;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	631
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	632 result->output.dst = PyMem_Malloc(outSize);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	633 if (!result->output.dst) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	634 PyErr_NoMemory();
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	635 goto except;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	636 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	637 result->output.size = outSize;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	638 result->output.pos = 0;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	639
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	640 result->input.src = NULL;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	641 result->input.size = 0;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	642 result->input.pos = 0;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	643
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	644 result->finishedInput = 0;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	645 result->finishedOutput = 0;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	646
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	647 goto finally;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	648
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	649 except:
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	650 if (result->cstream) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	651 ZSTD_freeCStream(result->cstream);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	652 result->cstream = NULL;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	653 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	654
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	655 Py_DecRef((PyObject*)result->compressor);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	656 Py_DecRef(result->reader);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	657
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	658 Py_DECREF(result);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	659 result = NULL;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	660
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	661 finally:
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	662 return result;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	663 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	664
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	665 PyDoc_STRVAR(ZstdCompressor_write_to___doc__,
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	666 "Create a context manager to write compressed data to an object.\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	667 "\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	668 "The passed object must have a ``write()`` method.\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	669 "\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	670 "The caller feeds input data to the object by calling ``compress(data)``.\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	671 "Compressed data is written to the argument given to this function.\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	672 "\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	673 "The function takes an optional ``size`` argument indicating the total size\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	674 "of the eventual input. If specified, the size will influence compression\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	675 "parameter tuning and could result in the size being written into the\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	676 "header of the compressed data.\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	677 "\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	678 "An optional ``write_size`` argument is also accepted. It defines the maximum\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	679 "byte size of chunks fed to ``write()``. By default, it uses the zstd default\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	680 "for a compressor output stream.\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	681 );
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	682
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	683 static ZstdCompressionWriter* ZstdCompressor_write_to(ZstdCompressor* self, PyObject* args, PyObject* kwargs) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	684 static char* kwlist[] = {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	685 "writer",
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	686 "size",
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	687 "write_size",
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	688 NULL
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	689 };
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	690
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	691 PyObject* writer;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	692 ZstdCompressionWriter* result;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	693 Py_ssize_t sourceSize = 0;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	694 size_t outSize = ZSTD_CStreamOutSize();
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	695
30895 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30830 diff changeset	696 if (!PyArg_ParseTupleAndKeywords(args, kwargs, "O\|nk:write_to", kwlist,
c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30830 diff changeset	697 &writer, &sourceSize, &outSize)) {
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	698 return NULL;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	699 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	700
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	701 if (!PyObject_HasAttrString(writer, "write")) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	702 PyErr_SetString(PyExc_ValueError, "must pass an object with a write() method");
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	703 return NULL;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	704 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	705
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	706 result = PyObject_New(ZstdCompressionWriter, &ZstdCompressionWriterType);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	707 if (!result) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	708 return NULL;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	709 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	710
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	711 result->compressor = self;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	712 Py_INCREF(result->compressor);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	713
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	714 result->writer = writer;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	715 Py_INCREF(result->writer);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	716
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	717 result->sourceSize = sourceSize;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	718
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	719 result->outSize = outSize;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	720
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	721 result->entered = 0;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	722 result->cstream = NULL;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	723
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	724 return result;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	725 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	726
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	727 static PyMethodDef ZstdCompressor_methods[] = {
30822 b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	728 { "compress", (PyCFunction)ZstdCompressor_compress,
b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	729 METH_VARARGS \| METH_KEYWORDS, ZstdCompressor_compress__doc__ },
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	730 { "compressobj", (PyCFunction)ZstdCompressor_compressobj,
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	731 METH_VARARGS \| METH_KEYWORDS, ZstdCompressionObj__doc__ },
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	732 { "copy_stream", (PyCFunction)ZstdCompressor_copy_stream,
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	733 METH_VARARGS \| METH_KEYWORDS, ZstdCompressor_copy_stream__doc__ },
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	734 { "read_from", (PyCFunction)ZstdCompressor_read_from,
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	735 METH_VARARGS \| METH_KEYWORDS, ZstdCompressor_read_from__doc__ },
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	736 { "write_to", (PyCFunction)ZstdCompressor_write_to,
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	737 METH_VARARGS \| METH_KEYWORDS, ZstdCompressor_write_to___doc__ },
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	738 { NULL, NULL }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	739 };
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	740
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	741 PyTypeObject ZstdCompressorType = {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	742 PyVarObject_HEAD_INIT(NULL, 0)
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	743 "zstd.ZstdCompressor", /* tp_name */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	744 sizeof(ZstdCompressor), /* tp_basicsize */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	745 0, /* tp_itemsize */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	746 (destructor)ZstdCompressor_dealloc, /* tp_dealloc */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	747 0, /* tp_print */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	748 0, /* tp_getattr */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	749 0, /* tp_setattr */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	750 0, /* tp_compare */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	751 0, /* tp_repr */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	752 0, /* tp_as_number */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	753 0, /* tp_as_sequence */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	754 0, /* tp_as_mapping */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	755 0, /* tp_hash */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	756 0, /* tp_call */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	757 0, /* tp_str */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	758 0, /* tp_getattro */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	759 0, /* tp_setattro */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	760 0, /* tp_as_buffer */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	761 Py_TPFLAGS_DEFAULT \| Py_TPFLAGS_BASETYPE, /* tp_flags */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	762 ZstdCompressor__doc__, /* tp_doc */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	763 0, /* tp_traverse */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	764 0, /* tp_clear */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	765 0, /* tp_richcompare */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	766 0, /* tp_weaklistoffset */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	767 0, /* tp_iter */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	768 0, /* tp_iternext */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	769 ZstdCompressor_methods, /* tp_methods */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	770 0, /* tp_members */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	771 0, /* tp_getset */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	772 0, /* tp_base */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	773 0, /* tp_dict */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	774 0, /* tp_descr_get */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	775 0, /* tp_descr_set */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	776 0, /* tp_dictoffset */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	777 (initproc)ZstdCompressor_init, /* tp_init */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	778 0, /* tp_alloc */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	779 PyType_GenericNew, /* tp_new */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	780 };
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	781
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	782 void compressor_module_init(PyObject* mod) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	783 Py_TYPE(&ZstdCompressorType) = &PyType_Type;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	784 if (PyType_Ready(&ZstdCompressorType) < 0) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	785 return;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	786 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	787
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	788 Py_INCREF((PyObject*)&ZstdCompressorType);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	789 PyModule_AddObject(mod, "ZstdCompressor",
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	790 (PyObject*)&ZstdCompressorType);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	791 }

Mercurial > public > mercurial-scm > hg

annotate contrib/python-zstandard/c-ext/compressor.c @ 30895:c32454d69b85