mercurial-scm/hg: contrib/python-zstandard/c-ext/compressor.c annotate

annotate contrib/python-zstandard/c-ext/compressor.c @ 40121:73fef626dae3

zstandard: vendor python-zstandard 0.10.1 This was just released. The upstream source distribution from PyPI was extracted. Unwanted files were removed. The clang-format ignore list was updated to reflect the new source of files. setup.py was updated to pass a new argument to python-zstandard's function for returning an Extension instance. Upstream had to change to use relative paths because Python 3.7's packaging doesn't seem to like absolute paths when defining sources, includes, etc. The default relative path calculation is relative to setup_zstd.py which is different from the directory of Mercurial's setup.py. The project contains a vendored copy of zstandard 1.3.6. The old version was 1.3.4. The API should be backwards compatible and nothing in core should need adjusted. However, there is a new "chunker" API that we may find useful in places where we want to emit compressed chunks of a fixed size. There are a pair of bug fixes in 0.10.0 with regards to compressobj() and decompressobj() when block flushing is used. I actually found these bugs when introducing these APIs in Mercurial! But existing Mercurial code is not affected because we don't perform block flushing. # no-check-commit because 3rd party code has different style guidelines Differential Revision: https://phab.mercurial-scm.org/D4911

author	Gregory Szorc <gregory.szorc@gmail.com>
date	Mon, 08 Oct 2018 16:27:40 -0700
parents	b1fb341d8a61
children	675775c33ab6

rev	line source
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1 /**
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	2 * Copyright (c) 2016-present, Gregory Szorc
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	3 * All rights reserved.
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	4 *
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	5 * This software may be modified and distributed under the terms
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	6 * of the BSD license. See the LICENSE file for details.
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	7 */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	8
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	9 #include "python-zstandard.h"
31796 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	10 #include "pool.h"
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	11
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	12 extern PyObject* ZstdError;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	13
40121 73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	14 int setup_cctx(ZstdCompressor* compressor) {
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	15 size_t zresult;
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	16
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	17 assert(compressor);
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	18 assert(compressor->cctx);
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	19 assert(compressor->params);
31796 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	20
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	21 zresult = ZSTD_CCtx_setParametersUsingCCtxParams(compressor->cctx, compressor->params);
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	22 if (ZSTD_isError(zresult)) {
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	23 PyErr_Format(ZstdError, "could not set compression parameters: %s",
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	24 ZSTD_getErrorName(zresult));
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	25 return 1;
31796 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	26 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	27
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	28 if (compressor->dict) {
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	29 if (compressor->dict->cdict) {
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	30 zresult = ZSTD_CCtx_refCDict(compressor->cctx, compressor->dict->cdict);
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	31 }
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	32 else {
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	33 zresult = ZSTD_CCtx_loadDictionary_advanced(compressor->cctx,
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	34 compressor->dict->dictData, compressor->dict->dictSize,
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	35 ZSTD_dlm_byRef, compressor->dict->dictType);
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	36 }
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	37 if (ZSTD_isError(zresult)) {
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	38 PyErr_Format(ZstdError, "could not load compression dictionary: %s",
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	39 ZSTD_getErrorName(zresult));
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	40 return 1;
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	41 }
30822 b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	42 }
b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	43
b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	44 return 0;
b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	45 }
b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	46
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	47 static PyObject* frame_progression(ZSTD_CCtx* cctx) {
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	48 PyObject* result = NULL;
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	49 PyObject* value;
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	50 ZSTD_frameProgression progression;
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	51
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	52 result = PyTuple_New(3);
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	53 if (!result) {
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	54 return NULL;
31796 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	55 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	56
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	57 progression = ZSTD_getFrameProgression(cctx);
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	58
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	59 value = PyLong_FromUnsignedLongLong(progression.ingested);
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	60 if (!value) {
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	61 Py_DECREF(result);
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	62 return NULL;
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	63 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	64
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	65 PyTuple_SET_ITEM(result, 0, value);
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	66
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	67 value = PyLong_FromUnsignedLongLong(progression.consumed);
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	68 if (!value) {
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	69 Py_DECREF(result);
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	70 return NULL;
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	71 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	72
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	73 PyTuple_SET_ITEM(result, 1, value);
31796 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	74
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	75 value = PyLong_FromUnsignedLongLong(progression.produced);
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	76 if (!value) {
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	77 Py_DECREF(result);
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	78 return NULL;
31796 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	79 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	80
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	81 PyTuple_SET_ITEM(result, 2, value);
31796 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	82
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	83 return result;
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	84 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	85
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	86 PyDoc_STRVAR(ZstdCompressor__doc__,
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	87 "ZstdCompressor(level=None, dict_data=None, compression_params=None)\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	88 "\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	89 "Create an object used to perform Zstandard compression.\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	90 "\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	91 "An instance can compress data various ways. Instances can be used multiple\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	92 "times. Each compression operation will use the compression parameters\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	93 "defined at construction time.\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	94 "\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	95 "Compression can be configured via the following names arguments:\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	96 "\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	97 "level\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	98 " Integer compression level.\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	99 "dict_data\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	100 " A ``ZstdCompressionDict`` to be used to compress with dictionary data.\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	101 "compression_params\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	102 " A ``CompressionParameters`` instance defining low-level compression"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	103 " parameters. If defined, this will overwrite the ``level`` argument.\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	104 "write_checksum\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	105 " If True, a 4 byte content checksum will be written with the compressed\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	106 " data, allowing the decompressor to perform content verification.\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	107 "write_content_size\n"
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	108 " If True (the default), the decompressed content size will be included in\n"
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	109 " the header of the compressed data. This data will only be written if the\n"
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	110 " compressor knows the size of the input data.\n"
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	111 "write_dict_id\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	112 " Determines whether the dictionary ID will be written into the compressed\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	113 " data. Defaults to True. Only adds content to the compressed data if\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	114 " a dictionary is being used.\n"
31796 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	115 "threads\n"
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	116 " Number of threads to use to compress data concurrently. When set,\n"
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	117 " compression operations are performed on multiple threads. The default\n"
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	118 " value (0) disables multi-threaded compression. A value of ``-1`` means to\n"
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	119 " set the number of threads to the number of detected logical CPUs.\n"
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	120 );
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	121
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	122 static int ZstdCompressor_init(ZstdCompressor* self, PyObject* args, PyObject* kwargs) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	123 static char* kwlist[] = {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	124 "level",
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	125 "dict_data",
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	126 "compression_params",
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	127 "write_checksum",
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	128 "write_content_size",
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	129 "write_dict_id",
31796 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	130 "threads",
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	131 NULL
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	132 };
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	133
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	134 int level = 3;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	135 ZstdCompressionDict* dict = NULL;
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	136 ZstdCompressionParametersObject* params = NULL;
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	137 PyObject* writeChecksum = NULL;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	138 PyObject* writeContentSize = NULL;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	139 PyObject* writeDictID = NULL;
31796 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	140 int threads = 0;
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	141
31796 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	142 if (!PyArg_ParseTupleAndKeywords(args, kwargs, "\|iO!O!OOOi:ZstdCompressor",
30895 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30830 diff changeset	143 kwlist, &level, &ZstdCompressionDictType, &dict,
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	144 &ZstdCompressionParametersType, &params,
31796 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	145 &writeChecksum, &writeContentSize, &writeDictID, &threads)) {
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	146 return -1;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	147 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	148
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	149 if (level > ZSTD_maxCLevel()) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	150 PyErr_Format(PyExc_ValueError, "level must be less than %d",
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	151 ZSTD_maxCLevel() + 1);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	152 return -1;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	153 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	154
31796 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	155 if (threads < 0) {
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	156 threads = cpu_count();
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	157 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	158
30822 b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	159 /* We create a ZSTD_CCtx for reuse among multiple operations to reduce the
b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	160 overhead of each compression operation. */
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	161 self->cctx = ZSTD_createCCtx();
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	162 if (!self->cctx) {
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	163 PyErr_NoMemory();
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	164 return -1;
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	165 }
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	166
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	167 /* TODO stuff the original parameters away somewhere so we can reset later. This
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	168 will allow us to do things like automatically adjust cparams based on input
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	169 size (assuming zstd isn't doing that internally). */
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	170
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	171 self->params = ZSTD_createCCtxParams();
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	172 if (!self->params) {
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	173 PyErr_NoMemory();
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	174 return -1;
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	175 }
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	176
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	177 if (params && writeChecksum) {
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	178 PyErr_SetString(PyExc_ValueError,
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	179 "cannot define compression_params and write_checksum");
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	180 return -1;
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	181 }
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	182
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	183 if (params && writeContentSize) {
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	184 PyErr_SetString(PyExc_ValueError,
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	185 "cannot define compression_params and write_content_size");
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	186 return -1;
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	187 }
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	188
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	189 if (params && writeDictID) {
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	190 PyErr_SetString(PyExc_ValueError,
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	191 "cannot define compression_params and write_dict_id");
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	192 return -1;
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	193 }
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	194
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	195 if (params && threads) {
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	196 PyErr_SetString(PyExc_ValueError,
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	197 "cannot define compression_params and threads");
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	198 return -1;
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	199 }
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	200
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	201 if (params) {
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	202 if (set_parameters(self->params, params)) {
31796 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	203 return -1;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	204 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	205 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	206 else {
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	207 if (set_parameter(self->params, ZSTD_p_compressionLevel, level)) {
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	208 return -1;
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	209 }
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	210
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	211 if (set_parameter(self->params, ZSTD_p_contentSizeFlag,
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	212 writeContentSize ? PyObject_IsTrue(writeContentSize) : 1)) {
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	213 return -1;
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	214 }
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	215
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	216 if (set_parameter(self->params, ZSTD_p_checksumFlag,
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	217 writeChecksum ? PyObject_IsTrue(writeChecksum) : 0)) {
31796 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	218 return -1;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	219 }
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	220
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	221 if (set_parameter(self->params, ZSTD_p_dictIDFlag,
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	222 writeDictID ? PyObject_IsTrue(writeDictID) : 1)) {
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	223 return -1;
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	224 }
30822 b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	225
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	226 if (threads) {
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	227 if (set_parameter(self->params, ZSTD_p_nbWorkers, threads)) {
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	228 return -1;
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	229 }
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	230 }
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	231 }
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	232
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	233 if (dict) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	234 self->dict = dict;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	235 Py_INCREF(dict);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	236 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	237
40121 73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	238 if (setup_cctx(self)) {
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	239 return -1;
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	240 }
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	241
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	242 return 0;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	243 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	244
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	245 static void ZstdCompressor_dealloc(ZstdCompressor* self) {
30822 b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	246 if (self->cctx) {
b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	247 ZSTD_freeCCtx(self->cctx);
b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	248 self->cctx = NULL;
b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	249 }
b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	250
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	251 if (self->params) {
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	252 ZSTD_freeCCtxParams(self->params);
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	253 self->params = NULL;
31796 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	254 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	255
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	256 Py_XDECREF(self->dict);
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	257 PyObject_Del(self);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	258 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	259
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	260 PyDoc_STRVAR(ZstdCompressor_memory_size__doc__,
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	261 "memory_size()\n"
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	262 "\n"
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	263 "Obtain the memory usage of this compressor, in bytes.\n"
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	264 );
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	265
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	266 static PyObject* ZstdCompressor_memory_size(ZstdCompressor* self) {
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	267 if (self->cctx) {
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	268 return PyLong_FromSize_t(ZSTD_sizeof_CCtx(self->cctx));
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	269 }
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	270 else {
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	271 PyErr_SetString(ZstdError, "no compressor context found; this should never happen");
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	272 return NULL;
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	273 }
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	274 }
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	275
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	276 PyDoc_STRVAR(ZstdCompressor_frame_progression__doc__,
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	277 "frame_progression()\n"
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	278 "\n"
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	279 "Return information on how much work the compressor has done.\n"
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	280 "\n"
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	281 "Returns a 3-tuple of (ingested, consumed, produced).\n"
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	282 );
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	283
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	284 static PyObject* ZstdCompressor_frame_progression(ZstdCompressor* self) {
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	285 return frame_progression(self->cctx);
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	286 }
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	287
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	288 PyDoc_STRVAR(ZstdCompressor_copy_stream__doc__,
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	289 "copy_stream(ifh, ofh[, size=0, read_size=default, write_size=default])\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	290 "compress data between streams\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	291 "\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	292 "Data will be read from ``ifh``, compressed, and written to ``ofh``.\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	293 "``ifh`` must have a ``read(size)`` method. ``ofh`` must have a ``write(data)``\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	294 "method.\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	295 "\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	296 "An optional ``size`` argument specifies the size of the source stream.\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	297 "If defined, compression parameters will be tuned based on the size.\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	298 "\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	299 "Optional arguments ``read_size`` and ``write_size`` define the chunk sizes\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	300 "of ``read()`` and ``write()`` operations, respectively. By default, they use\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	301 "the default compression stream input and output sizes, respectively.\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	302 );
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	303
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	304 static PyObject* ZstdCompressor_copy_stream(ZstdCompressor* self, PyObject* args, PyObject* kwargs) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	305 static char* kwlist[] = {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	306 "ifh",
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	307 "ofh",
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	308 "size",
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	309 "read_size",
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	310 "write_size",
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	311 NULL
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	312 };
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	313
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	314 PyObject* source;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	315 PyObject* dest;
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	316 unsigned long long sourceSize = ZSTD_CONTENTSIZE_UNKNOWN;
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	317 size_t inSize = ZSTD_CStreamInSize();
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	318 size_t outSize = ZSTD_CStreamOutSize();
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	319 ZSTD_inBuffer input;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	320 ZSTD_outBuffer output;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	321 Py_ssize_t totalRead = 0;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	322 Py_ssize_t totalWrite = 0;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	323 char* readBuffer;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	324 Py_ssize_t readSize;
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	325 PyObject* readResult = NULL;
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	326 PyObject* res = NULL;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	327 size_t zresult;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	328 PyObject* writeResult;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	329 PyObject* totalReadPy;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	330 PyObject* totalWritePy;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	331
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	332 if (!PyArg_ParseTupleAndKeywords(args, kwargs, "OO\|Kkk:copy_stream", kwlist,
30895 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30830 diff changeset	333 &source, &dest, &sourceSize, &inSize, &outSize)) {
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	334 return NULL;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	335 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	336
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	337 if (!PyObject_HasAttrString(source, "read")) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	338 PyErr_SetString(PyExc_ValueError, "first argument must have a read() method");
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	339 return NULL;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	340 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	341
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	342 if (!PyObject_HasAttrString(dest, "write")) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	343 PyErr_SetString(PyExc_ValueError, "second argument must have a write() method");
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	344 return NULL;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	345 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	346
40121 73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	347 ZSTD_CCtx_reset(self->cctx);
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	348
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	349 zresult = ZSTD_CCtx_setPledgedSrcSize(self->cctx, sourceSize);
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	350 if (ZSTD_isError(zresult)) {
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	351 PyErr_Format(ZstdError, "error setting source size: %s",
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	352 ZSTD_getErrorName(zresult));
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	353 return NULL;
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	354 }
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	355
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	356 /* Prevent free on uninitialized memory in finally. */
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	357 output.dst = PyMem_Malloc(outSize);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	358 if (!output.dst) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	359 PyErr_NoMemory();
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	360 res = NULL;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	361 goto finally;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	362 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	363 output.size = outSize;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	364 output.pos = 0;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	365
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	366 input.src = NULL;
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	367 input.size = 0;
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	368 input.pos = 0;
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	369
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	370 while (1) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	371 /* Try to read from source stream. */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	372 readResult = PyObject_CallMethod(source, "read", "n", inSize);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	373 if (!readResult) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	374 PyErr_SetString(ZstdError, "could not read() from source");
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	375 goto finally;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	376 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	377
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	378 PyBytes_AsStringAndSize(readResult, &readBuffer, &readSize);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	379
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	380 /* If no data was read, we're at EOF. */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	381 if (0 == readSize) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	382 break;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	383 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	384
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	385 totalRead += readSize;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	386
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	387 /* Send data to compressor */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	388 input.src = readBuffer;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	389 input.size = readSize;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	390 input.pos = 0;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	391
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	392 while (input.pos < input.size) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	393 Py_BEGIN_ALLOW_THREADS
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	394 zresult = ZSTD_compress_generic(self->cctx, &output, &input, ZSTD_e_continue);
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	395 Py_END_ALLOW_THREADS
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	396
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	397 if (ZSTD_isError(zresult)) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	398 res = NULL;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	399 PyErr_Format(ZstdError, "zstd compress error: %s", ZSTD_getErrorName(zresult));
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	400 goto finally;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	401 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	402
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	403 if (output.pos) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	404 #if PY_MAJOR_VERSION >= 3
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	405 writeResult = PyObject_CallMethod(dest, "write", "y#",
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	406 #else
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	407 writeResult = PyObject_CallMethod(dest, "write", "s#",
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	408 #endif
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	409 output.dst, output.pos);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	410 Py_XDECREF(writeResult);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	411 totalWrite += output.pos;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	412 output.pos = 0;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	413 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	414 }
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	415
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	416 Py_CLEAR(readResult);
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	417 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	418
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	419 /* We've finished reading. Now flush the compressor stream. */
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	420 assert(input.pos == input.size);
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	421
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	422 while (1) {
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	423 Py_BEGIN_ALLOW_THREADS
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	424 zresult = ZSTD_compress_generic(self->cctx, &output, &input, ZSTD_e_end);
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	425 Py_END_ALLOW_THREADS
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	426
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	427 if (ZSTD_isError(zresult)) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	428 PyErr_Format(ZstdError, "error ending compression stream: %s",
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	429 ZSTD_getErrorName(zresult));
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	430 res = NULL;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	431 goto finally;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	432 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	433
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	434 if (output.pos) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	435 #if PY_MAJOR_VERSION >= 3
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	436 writeResult = PyObject_CallMethod(dest, "write", "y#",
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	437 #else
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	438 writeResult = PyObject_CallMethod(dest, "write", "s#",
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	439 #endif
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	440 output.dst, output.pos);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	441 totalWrite += output.pos;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	442 Py_XDECREF(writeResult);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	443 output.pos = 0;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	444 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	445
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	446 if (!zresult) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	447 break;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	448 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	449 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	450
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	451 totalReadPy = PyLong_FromSsize_t(totalRead);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	452 totalWritePy = PyLong_FromSsize_t(totalWrite);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	453 res = PyTuple_Pack(2, totalReadPy, totalWritePy);
31796 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	454 Py_DECREF(totalReadPy);
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	455 Py_DECREF(totalWritePy);
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	456
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	457 finally:
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	458 if (output.dst) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	459 PyMem_Free(output.dst);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	460 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	461
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	462 Py_XDECREF(readResult);
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	463
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	464 return res;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	465 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	466
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	467 PyDoc_STRVAR(ZstdCompressor_stream_reader__doc__,
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	468 "stream_reader(source, [size=0])\n"
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	469 "\n"
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	470 "Obtain an object that behaves like an I/O stream.\n"
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	471 "\n"
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	472 "The source object can be any object with a ``read(size)`` method\n"
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	473 "or an object that conforms to the buffer protocol.\n"
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	474 );
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	475
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	476 static ZstdCompressionReader* ZstdCompressor_stream_reader(ZstdCompressor* self, PyObject* args, PyObject* kwargs) {
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	477 static char* kwlist[] = {
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	478 "source",
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	479 "size",
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	480 "read_size",
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	481 NULL
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	482 };
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	483
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	484 PyObject* source;
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	485 unsigned long long sourceSize = ZSTD_CONTENTSIZE_UNKNOWN;
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	486 size_t readSize = ZSTD_CStreamInSize();
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	487 ZstdCompressionReader* result = NULL;
40121 73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	488 size_t zresult;
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	489
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	490 if (!PyArg_ParseTupleAndKeywords(args, kwargs, "O\|Kk:stream_reader", kwlist,
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	491 &source, &sourceSize, &readSize)) {
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	492 return NULL;
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	493 }
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	494
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	495 result = (ZstdCompressionReader)PyObject_CallObject((PyObject)&ZstdCompressionReaderType, NULL);
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	496 if (!result) {
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	497 return NULL;
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	498 }
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	499
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	500 if (PyObject_HasAttrString(source, "read")) {
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	501 result->reader = source;
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	502 Py_INCREF(source);
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	503 result->readSize = readSize;
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	504 }
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	505 else if (1 == PyObject_CheckBuffer(source)) {
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	506 if (0 != PyObject_GetBuffer(source, &result->buffer, PyBUF_CONTIG_RO)) {
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	507 goto except;
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	508 }
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	509
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	510 assert(result->buffer.len >= 0);
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	511
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	512 sourceSize = result->buffer.len;
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	513 }
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	514 else {
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	515 PyErr_SetString(PyExc_TypeError,
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	516 "must pass an object with a read() method or that conforms to the buffer protocol");
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	517 goto except;
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	518 }
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	519
40121 73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	520 ZSTD_CCtx_reset(self->cctx);
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	521
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	522 zresult = ZSTD_CCtx_setPledgedSrcSize(self->cctx, sourceSize);
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	523 if (ZSTD_isError(zresult)) {
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	524 PyErr_Format(ZstdError, "error setting source source: %s",
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	525 ZSTD_getErrorName(zresult));
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	526 goto except;
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	527 }
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	528
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	529 result->compressor = self;
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	530 Py_INCREF(self);
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	531
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	532 return result;
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	533
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	534 except:
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	535 Py_CLEAR(result);
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	536
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	537 return NULL;
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	538 }
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	539
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	540 PyDoc_STRVAR(ZstdCompressor_compress__doc__,
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	541 "compress(data)\n"
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	542 "\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	543 "Compress data in a single operation.\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	544 "\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	545 "This is the simplest mechanism to perform compression: simply pass in a\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	546 "value and get a compressed value back. It is almost the most prone to abuse.\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	547 "The input and output values must fit in memory, so passing in very large\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	548 "values can result in excessive memory usage. For this reason, one of the\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	549 "streaming based APIs is preferred for larger values.\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	550 );
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	551
30822 b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	552 static PyObject* ZstdCompressor_compress(ZstdCompressor* self, PyObject* args, PyObject* kwargs) {
b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	553 static char* kwlist[] = {
b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	554 "data",
b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	555 NULL
b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	556 };
b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	557
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	558 Py_buffer source;
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	559 size_t destSize;
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	560 PyObject* output = NULL;
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	561 size_t zresult;
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	562 ZSTD_outBuffer outBuffer;
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	563 ZSTD_inBuffer inBuffer;
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	564
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	565 #if PY_MAJOR_VERSION >= 3
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	566 if (!PyArg_ParseTupleAndKeywords(args, kwargs, "y*\|O:compress",
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	567 #else
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	568 if (!PyArg_ParseTupleAndKeywords(args, kwargs, "s*\|O:compress",
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	569 #endif
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	570 kwlist, &source)) {
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	571 return NULL;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	572 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	573
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	574 if (!PyBuffer_IsContiguous(&source, 'C') \|\| source.ndim > 1) {
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	575 PyErr_SetString(PyExc_ValueError,
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	576 "data buffer should be contiguous and have at most one dimension");
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	577 goto finally;
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	578 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	579
40121 73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	580 ZSTD_CCtx_reset(self->cctx);
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	581
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	582 destSize = ZSTD_compressBound(source.len);
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	583 output = PyBytes_FromStringAndSize(NULL, destSize);
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	584 if (!output) {
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	585 goto finally;
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	586 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	587
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	588 zresult = ZSTD_CCtx_setPledgedSrcSize(self->cctx, source.len);
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	589 if (ZSTD_isError(zresult)) {
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	590 PyErr_Format(ZstdError, "error setting source size: %s",
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	591 ZSTD_getErrorName(zresult));
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	592 Py_CLEAR(output);
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	593 goto finally;
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	594 }
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	595
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	596 inBuffer.src = source.buf;
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	597 inBuffer.size = source.len;
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	598 inBuffer.pos = 0;
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	599
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	600 outBuffer.dst = PyBytes_AsString(output);
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	601 outBuffer.size = destSize;
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	602 outBuffer.pos = 0;
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	603
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	604 Py_BEGIN_ALLOW_THREADS
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	605 /* By avoiding ZSTD_compress(), we don't necessarily write out content
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	606 size. This means the argument to ZstdCompressor to control frame
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	607 parameters is honored. */
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	608 zresult = ZSTD_compress_generic(self->cctx, &outBuffer, &inBuffer, ZSTD_e_end);
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	609 Py_END_ALLOW_THREADS
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	610
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	611 if (ZSTD_isError(zresult)) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	612 PyErr_Format(ZstdError, "cannot compress: %s", ZSTD_getErrorName(zresult));
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	613 Py_CLEAR(output);
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	614 goto finally;
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	615 }
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	616 else if (zresult) {
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	617 PyErr_SetString(ZstdError, "unexpected partial frame flush");
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	618 Py_CLEAR(output);
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	619 goto finally;
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	620 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	621
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	622 Py_SIZE(output) = outBuffer.pos;
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	623
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	624 finally:
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	625 PyBuffer_Release(&source);
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	626 return output;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	627 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	628
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	629 PyDoc_STRVAR(ZstdCompressionObj__doc__,
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	630 "compressobj()\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	631 "\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	632 "Return an object exposing ``compress(data)`` and ``flush()`` methods.\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	633 "\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	634 "The returned object exposes an API similar to ``zlib.compressobj`` and\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	635 "``bz2.BZ2Compressor`` so that callers can swap in the zstd compressor\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	636 "without changing how compression is performed.\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	637 );
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	638
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	639 static ZstdCompressionObj* ZstdCompressor_compressobj(ZstdCompressor* self, PyObject* args, PyObject* kwargs) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	640 static char* kwlist[] = {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	641 "size",
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	642 NULL
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	643 };
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	644
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	645 unsigned long long inSize = ZSTD_CONTENTSIZE_UNKNOWN;
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	646 size_t outSize = ZSTD_CStreamOutSize();
31796 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	647 ZstdCompressionObj* result = NULL;
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	648 size_t zresult;
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	649
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	650 if (!PyArg_ParseTupleAndKeywords(args, kwargs, "\|K:compressobj", kwlist, &inSize)) {
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	651 return NULL;
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	652 }
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	653
40121 73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	654 ZSTD_CCtx_reset(self->cctx);
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	655
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	656 zresult = ZSTD_CCtx_setPledgedSrcSize(self->cctx, inSize);
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	657 if (ZSTD_isError(zresult)) {
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	658 PyErr_Format(ZstdError, "error setting source size: %s",
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	659 ZSTD_getErrorName(zresult));
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	660 return NULL;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	661 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	662
31796 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	663 result = (ZstdCompressionObj)PyObject_CallObject((PyObject)&ZstdCompressionObjType, NULL);
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	664 if (!result) {
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	665 return NULL;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	666 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	667
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	668 result->output.dst = PyMem_Malloc(outSize);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	669 if (!result->output.dst) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	670 PyErr_NoMemory();
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	671 Py_DECREF(result);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	672 return NULL;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	673 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	674 result->output.size = outSize;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	675 result->compressor = self;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	676 Py_INCREF(result->compressor);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	677
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	678 return result;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	679 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	680
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	681 PyDoc_STRVAR(ZstdCompressor_read_to_iter__doc__,
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	682 "read_to_iter(reader, [size=0, read_size=default, write_size=default])\n"
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	683 "Read uncompressed data from a reader and return an iterator\n"
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	684 "\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	685 "Returns an iterator of compressed data produced from reading from ``reader``.\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	686 "\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	687 "Uncompressed data will be obtained from ``reader`` by calling the\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	688 "``read(size)`` method of it. The source data will be streamed into a\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	689 "compressor. As compressed data is available, it will be exposed to the\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	690 "iterator.\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	691 "\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	692 "Data is read from the source in chunks of ``read_size``. Compressed chunks\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	693 "are at most ``write_size`` bytes. Both values default to the zstd input and\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	694 "and output defaults, respectively.\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	695 "\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	696 "The caller is partially in control of how fast data is fed into the\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	697 "compressor by how it consumes the returned iterator. The compressor will\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	698 "not consume from the reader unless the caller consumes from the iterator.\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	699 );
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	700
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	701 static ZstdCompressorIterator* ZstdCompressor_read_to_iter(ZstdCompressor* self, PyObject* args, PyObject* kwargs) {
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	702 static char* kwlist[] = {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	703 "reader",
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	704 "size",
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	705 "read_size",
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	706 "write_size",
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	707 NULL
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	708 };
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	709
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	710 PyObject* reader;
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	711 unsigned long long sourceSize = ZSTD_CONTENTSIZE_UNKNOWN;
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	712 size_t inSize = ZSTD_CStreamInSize();
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	713 size_t outSize = ZSTD_CStreamOutSize();
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	714 ZstdCompressorIterator* result;
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	715 size_t zresult;
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	716
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	717 if (!PyArg_ParseTupleAndKeywords(args, kwargs, "O\|Kkk:read_to_iter", kwlist,
30895 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30830 diff changeset	718 &reader, &sourceSize, &inSize, &outSize)) {
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	719 return NULL;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	720 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	721
31796 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	722 result = (ZstdCompressorIterator)PyObject_CallObject((PyObject)&ZstdCompressorIteratorType, NULL);
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	723 if (!result) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	724 return NULL;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	725 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	726 if (PyObject_HasAttrString(reader, "read")) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	727 result->reader = reader;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	728 Py_INCREF(result->reader);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	729 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	730 else if (1 == PyObject_CheckBuffer(reader)) {
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	731 if (0 != PyObject_GetBuffer(reader, &result->buffer, PyBUF_CONTIG_RO)) {
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	732 goto except;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	733 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	734
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	735 sourceSize = result->buffer.len;
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	736 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	737 else {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	738 PyErr_SetString(PyExc_ValueError,
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	739 "must pass an object with a read() method or conforms to buffer protocol");
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	740 goto except;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	741 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	742
40121 73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	743 ZSTD_CCtx_reset(self->cctx);
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	744
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	745 zresult = ZSTD_CCtx_setPledgedSrcSize(self->cctx, sourceSize);
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	746 if (ZSTD_isError(zresult)) {
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	747 PyErr_Format(ZstdError, "error setting source size: %s",
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	748 ZSTD_getErrorName(zresult));
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	749 return NULL;
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	750 }
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	751
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	752 result->compressor = self;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	753 Py_INCREF(result->compressor);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	754
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	755 result->inSize = inSize;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	756 result->outSize = outSize;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	757
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	758 result->output.dst = PyMem_Malloc(outSize);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	759 if (!result->output.dst) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	760 PyErr_NoMemory();
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	761 goto except;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	762 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	763 result->output.size = outSize;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	764
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	765 goto finally;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	766
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	767 except:
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	768 Py_CLEAR(result);
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	769
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	770 finally:
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	771 return result;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	772 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	773
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	774 PyDoc_STRVAR(ZstdCompressor_stream_writer___doc__,
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	775 "Create a context manager to write compressed data to an object.\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	776 "\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	777 "The passed object must have a ``write()`` method.\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	778 "\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	779 "The caller feeds input data to the object by calling ``compress(data)``.\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	780 "Compressed data is written to the argument given to this function.\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	781 "\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	782 "The function takes an optional ``size`` argument indicating the total size\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	783 "of the eventual input. If specified, the size will influence compression\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	784 "parameter tuning and could result in the size being written into the\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	785 "header of the compressed data.\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	786 "\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	787 "An optional ``write_size`` argument is also accepted. It defines the maximum\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	788 "byte size of chunks fed to ``write()``. By default, it uses the zstd default\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	789 "for a compressor output stream.\n"
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	790 );
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	791
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	792 static ZstdCompressionWriter* ZstdCompressor_stream_writer(ZstdCompressor* self, PyObject* args, PyObject* kwargs) {
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	793 static char* kwlist[] = {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	794 "writer",
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	795 "size",
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	796 "write_size",
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	797 NULL
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	798 };
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	799
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	800 PyObject* writer;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	801 ZstdCompressionWriter* result;
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	802 unsigned long long sourceSize = ZSTD_CONTENTSIZE_UNKNOWN;
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	803 size_t outSize = ZSTD_CStreamOutSize();
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	804
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	805 if (!PyArg_ParseTupleAndKeywords(args, kwargs, "O\|Kk:stream_writer", kwlist,
30895 c32454d69b85 zstd: vendor python-zstandard 0.7.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30830 diff changeset	806 &writer, &sourceSize, &outSize)) {
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	807 return NULL;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	808 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	809
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	810 if (!PyObject_HasAttrString(writer, "write")) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	811 PyErr_SetString(PyExc_ValueError, "must pass an object with a write() method");
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	812 return NULL;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	813 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	814
40121 73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	815 ZSTD_CCtx_reset(self->cctx);
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	816
31796 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	817 result = (ZstdCompressionWriter)PyObject_CallObject((PyObject)&ZstdCompressionWriterType, NULL);
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	818 if (!result) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	819 return NULL;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	820 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	821
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	822 result->compressor = self;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	823 Py_INCREF(result->compressor);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	824
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	825 result->writer = writer;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	826 Py_INCREF(result->writer);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	827
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	828 result->sourceSize = sourceSize;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	829 result->outSize = outSize;
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	830 result->bytesCompressed = 0;
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	831
31796 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	832 return result;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	833 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	834
40121 73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	835 PyDoc_STRVAR(ZstdCompressor_chunker__doc__,
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	836 "Create an object for iterative compressing to same-sized chunks.\n"
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	837 );
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	838
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	839 static ZstdCompressionChunker* ZstdCompressor_chunker(ZstdCompressor* self, PyObject* args, PyObject* kwargs) {
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	840 static char* kwlist[] = {
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	841 "size",
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	842 "chunk_size",
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	843 NULL
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	844 };
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	845
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	846 unsigned long long sourceSize = ZSTD_CONTENTSIZE_UNKNOWN;
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	847 size_t chunkSize = ZSTD_CStreamOutSize();
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	848 ZstdCompressionChunker* chunker;
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	849 size_t zresult;
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	850
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	851 if (!PyArg_ParseTupleAndKeywords(args, kwargs, "\|Kk:chunker", kwlist,
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	852 &sourceSize, &chunkSize)) {
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	853 return NULL;
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	854 }
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	855
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	856 ZSTD_CCtx_reset(self->cctx);
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	857
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	858 zresult = ZSTD_CCtx_setPledgedSrcSize(self->cctx, sourceSize);
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	859 if (ZSTD_isError(zresult)) {
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	860 PyErr_Format(ZstdError, "error setting source size: %s",
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	861 ZSTD_getErrorName(zresult));
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	862 return NULL;
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	863 }
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	864
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	865 chunker = (ZstdCompressionChunker)PyObject_CallObject((PyObject)&ZstdCompressionChunkerType, NULL);
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	866 if (!chunker) {
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	867 return NULL;
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	868 }
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	869
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	870 chunker->output.dst = PyMem_Malloc(chunkSize);
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	871 if (!chunker->output.dst) {
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	872 PyErr_NoMemory();
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	873 Py_DECREF(chunker);
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	874 return NULL;
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	875 }
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	876 chunker->output.size = chunkSize;
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	877 chunker->output.pos = 0;
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	878
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	879 chunker->compressor = self;
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	880 Py_INCREF(chunker->compressor);
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	881
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	882 chunker->chunkSize = chunkSize;
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	883
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	884 return chunker;
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	885 }
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	886
31796 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	887 typedef struct {
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	888 void* sourceData;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	889 size_t sourceSize;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	890 } DataSource;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	891
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	892 typedef struct {
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	893 DataSource* sources;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	894 Py_ssize_t sourcesSize;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	895 unsigned long long totalSourceSize;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	896 } DataSources;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	897
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	898 typedef struct {
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	899 void* dest;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	900 Py_ssize_t destSize;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	901 BufferSegment* segments;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	902 Py_ssize_t segmentsSize;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	903 } DestBuffer;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	904
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	905 typedef enum {
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	906 WorkerError_none = 0,
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	907 WorkerError_zstd = 1,
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	908 WorkerError_no_memory = 2,
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	909 WorkerError_nospace = 3,
31796 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	910 } WorkerError;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	911
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	912 /**
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	913 * Holds state for an individual worker performing multi_compress_to_buffer work.
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	914 */
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	915 typedef struct {
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	916 /* Used for compression. */
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	917 ZSTD_CCtx* cctx;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	918
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	919 /* What to compress. */
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	920 DataSource* sources;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	921 Py_ssize_t sourcesSize;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	922 Py_ssize_t startOffset;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	923 Py_ssize_t endOffset;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	924 unsigned long long totalSourceSize;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	925
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	926 /* Result storage. */
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	927 DestBuffer* destBuffers;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	928 Py_ssize_t destCount;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	929
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	930 /* Error tracking. */
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	931 WorkerError error;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	932 size_t zresult;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	933 Py_ssize_t errorOffset;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	934 } WorkerState;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	935
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	936 static void compress_worker(WorkerState* state) {
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	937 Py_ssize_t inputOffset = state->startOffset;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	938 Py_ssize_t remainingItems = state->endOffset - state->startOffset + 1;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	939 Py_ssize_t currentBufferStartOffset = state->startOffset;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	940 size_t zresult;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	941 void* newDest;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	942 size_t allocationSize;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	943 size_t boundSize;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	944 Py_ssize_t destOffset = 0;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	945 DataSource* sources = state->sources;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	946 DestBuffer* destBuffer;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	947
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	948 assert(!state->destBuffers);
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	949 assert(0 == state->destCount);
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	950
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	951 /*
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	952 * The total size of the compressed data is unknown until we actually
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	953 * compress data. That means we can't pre-allocate the exact size we need.
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	954 *
31796 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	955 * There is a cost to every allocation and reallocation. So, it is in our
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	956 * interest to minimize the number of allocations.
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	957 *
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	958 * There is also a cost to too few allocations. If allocations are too
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	959 * large they may fail. If buffers are shared and all inputs become
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	960 * irrelevant at different lifetimes, then a reference to one segment
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	961 * in the buffer will keep the entire buffer alive. This leads to excessive
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	962 * memory usage.
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	963 *
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	964 * Our current strategy is to assume a compression ratio of 16:1 and
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	965 * allocate buffers of that size, rounded up to the nearest power of 2
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	966 * (because computers like round numbers). That ratio is greater than what
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	967 * most inputs achieve. This is by design: we don't want to over-allocate.
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	968 * But we don't want to under-allocate and lead to too many buffers either.
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	969 */
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	970
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	971 state->destCount = 1;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	972
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	973 state->destBuffers = calloc(1, sizeof(DestBuffer));
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	974 if (NULL == state->destBuffers) {
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	975 state->error = WorkerError_no_memory;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	976 return;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	977 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	978
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	979 destBuffer = &state->destBuffers[state->destCount - 1];
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	980
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	981 /*
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	982 * Rather than track bounds and grow the segments buffer, allocate space
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	983 * to hold remaining items then truncate when we're done with it.
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	984 */
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	985 destBuffer->segments = calloc(remainingItems, sizeof(BufferSegment));
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	986 if (NULL == destBuffer->segments) {
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	987 state->error = WorkerError_no_memory;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	988 return;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	989 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	990
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	991 destBuffer->segmentsSize = remainingItems;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	992
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	993 assert(state->totalSourceSize <= SIZE_MAX);
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	994 allocationSize = roundpow2((size_t)state->totalSourceSize >> 4);
31796 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	995
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	996 /* If the maximum size of the output is larger than that, round up. */
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	997 boundSize = ZSTD_compressBound(sources[inputOffset].sourceSize);
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	998
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	999 if (boundSize > allocationSize) {
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1000 allocationSize = roundpow2(boundSize);
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1001 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1002
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1003 destBuffer->dest = malloc(allocationSize);
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1004 if (NULL == destBuffer->dest) {
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1005 state->error = WorkerError_no_memory;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1006 return;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1007 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1008
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1009 destBuffer->destSize = allocationSize;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1010
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1011 for (inputOffset = state->startOffset; inputOffset <= state->endOffset; inputOffset++) {
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1012 void* source = sources[inputOffset].sourceData;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1013 size_t sourceSize = sources[inputOffset].sourceSize;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1014 size_t destAvailable;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1015 void* dest;
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1016 ZSTD_outBuffer opOutBuffer;
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1017 ZSTD_inBuffer opInBuffer;
31796 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1018
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1019 destAvailable = destBuffer->destSize - destOffset;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1020 boundSize = ZSTD_compressBound(sourceSize);
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1021
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1022 /*
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1023 * Not enough space in current buffer to hold largest compressed output.
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1024 * So allocate and switch to a new output buffer.
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1025 */
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1026 if (boundSize > destAvailable) {
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1027 /*
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1028 * The downsizing of the existing buffer is optional. It should be cheap
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1029 * (unlike growing). So we just do it.
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1030 */
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1031 if (destAvailable) {
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1032 newDest = realloc(destBuffer->dest, destOffset);
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1033 if (NULL == newDest) {
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1034 state->error = WorkerError_no_memory;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1035 return;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1036 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1037
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1038 destBuffer->dest = newDest;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1039 destBuffer->destSize = destOffset;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1040 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1041
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1042 /* Truncate segments buffer. */
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1043 newDest = realloc(destBuffer->segments,
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1044 (inputOffset - currentBufferStartOffset + 1) * sizeof(BufferSegment));
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1045 if (NULL == newDest) {
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1046 state->error = WorkerError_no_memory;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1047 return;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1048 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1049
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1050 destBuffer->segments = newDest;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1051 destBuffer->segmentsSize = inputOffset - currentBufferStartOffset;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1052
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1053 /* Grow space for new struct. */
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1054 /* TODO consider over-allocating so we don't do this every time. */
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1055 newDest = realloc(state->destBuffers, (state->destCount + 1) * sizeof(DestBuffer));
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1056 if (NULL == newDest) {
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1057 state->error = WorkerError_no_memory;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1058 return;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1059 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1060
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1061 state->destBuffers = newDest;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1062 state->destCount++;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1063
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1064 destBuffer = &state->destBuffers[state->destCount - 1];
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1065
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1066 /* Don't take any chances with non-NULL pointers. */
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1067 memset(destBuffer, 0, sizeof(DestBuffer));
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1068
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1069 /**
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1070 * We could dynamically update allocation size based on work done so far.
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1071 * For now, keep is simple.
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1072 */
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1073 assert(state->totalSourceSize <= SIZE_MAX);
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1074 allocationSize = roundpow2((size_t)state->totalSourceSize >> 4);
31796 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1075
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1076 if (boundSize > allocationSize) {
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1077 allocationSize = roundpow2(boundSize);
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1078 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1079
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1080 destBuffer->dest = malloc(allocationSize);
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1081 if (NULL == destBuffer->dest) {
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1082 state->error = WorkerError_no_memory;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1083 return;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1084 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1085
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1086 destBuffer->destSize = allocationSize;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1087 destAvailable = allocationSize;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1088 destOffset = 0;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1089
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1090 destBuffer->segments = calloc(remainingItems, sizeof(BufferSegment));
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1091 if (NULL == destBuffer->segments) {
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1092 state->error = WorkerError_no_memory;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1093 return;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1094 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1095
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1096 destBuffer->segmentsSize = remainingItems;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1097 currentBufferStartOffset = inputOffset;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1098 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1099
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1100 dest = (char*)destBuffer->dest + destOffset;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1101
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1102 opInBuffer.src = source;
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1103 opInBuffer.size = sourceSize;
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1104 opInBuffer.pos = 0;
31796 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1105
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1106 opOutBuffer.dst = dest;
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1107 opOutBuffer.size = destAvailable;
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1108 opOutBuffer.pos = 0;
31796 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1109
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1110 zresult = ZSTD_CCtx_setPledgedSrcSize(state->cctx, sourceSize);
31796 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1111 if (ZSTD_isError(zresult)) {
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1112 state->error = WorkerError_zstd;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1113 state->zresult = zresult;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1114 state->errorOffset = inputOffset;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1115 break;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1116 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1117
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1118 zresult = ZSTD_compress_generic(state->cctx, &opOutBuffer, &opInBuffer, ZSTD_e_end);
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1119 if (ZSTD_isError(zresult)) {
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1120 state->error = WorkerError_zstd;
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1121 state->zresult = zresult;
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1122 state->errorOffset = inputOffset;
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1123 break;
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1124 }
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1125 else if (zresult) {
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1126 state->error = WorkerError_nospace;
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1127 state->errorOffset = inputOffset;
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1128 break;
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1129 }
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1130
31796 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1131 destBuffer->segments[inputOffset - currentBufferStartOffset].offset = destOffset;
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1132 destBuffer->segments[inputOffset - currentBufferStartOffset].length = opOutBuffer.pos;
31796 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1133
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1134 destOffset += opOutBuffer.pos;
31796 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1135 remainingItems--;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1136 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1137
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1138 if (destBuffer->destSize > destOffset) {
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1139 newDest = realloc(destBuffer->dest, destOffset);
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1140 if (NULL == newDest) {
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1141 state->error = WorkerError_no_memory;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1142 return;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1143 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1144
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1145 destBuffer->dest = newDest;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1146 destBuffer->destSize = destOffset;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1147 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1148 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1149
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1150 ZstdBufferWithSegmentsCollection* compress_from_datasources(ZstdCompressor* compressor,
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1151 DataSources* sources, Py_ssize_t threadCount) {
31796 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1152 unsigned long long bytesPerWorker;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1153 POOL_ctx* pool = NULL;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1154 WorkerState* workerStates = NULL;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1155 Py_ssize_t i;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1156 unsigned long long workerBytes = 0;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1157 Py_ssize_t workerStartOffset = 0;
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1158 Py_ssize_t currentThread = 0;
31796 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1159 int errored = 0;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1160 Py_ssize_t segmentsCount = 0;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1161 Py_ssize_t segmentIndex;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1162 PyObject* segmentsArg = NULL;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1163 ZstdBufferWithSegments* buffer;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1164 ZstdBufferWithSegmentsCollection* result = NULL;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1165
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1166 assert(sources->sourcesSize > 0);
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1167 assert(sources->totalSourceSize > 0);
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1168 assert(threadCount >= 1);
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1169
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1170 /* More threads than inputs makes no sense. */
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1171 threadCount = sources->sourcesSize < threadCount ? sources->sourcesSize
31796 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1172 : threadCount;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1173
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1174 /* TODO lower thread count when input size is too small and threads would add
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1175 overhead. */
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1176
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1177 workerStates = PyMem_Malloc(threadCount * sizeof(WorkerState));
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1178 if (NULL == workerStates) {
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1179 PyErr_NoMemory();
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1180 goto finally;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1181 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1182
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1183 memset(workerStates, 0, threadCount * sizeof(WorkerState));
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1184
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1185 if (threadCount > 1) {
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1186 pool = POOL_create(threadCount, 1);
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1187 if (NULL == pool) {
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1188 PyErr_SetString(ZstdError, "could not initialize zstd thread pool");
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1189 goto finally;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1190 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1191 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1192
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1193 bytesPerWorker = sources->totalSourceSize / threadCount;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1194
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1195 for (i = 0; i < threadCount; i++) {
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1196 size_t zresult;
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1197
31796 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1198 workerStates[i].cctx = ZSTD_createCCtx();
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1199 if (!workerStates[i].cctx) {
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1200 PyErr_NoMemory();
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1201 goto finally;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1202 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1203
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1204 zresult = ZSTD_CCtx_setParametersUsingCCtxParams(workerStates[i].cctx,
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1205 compressor->params);
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1206 if (ZSTD_isError(zresult)) {
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1207 PyErr_Format(ZstdError, "could not set compression parameters: %s",
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1208 ZSTD_getErrorName(zresult));
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1209 goto finally;
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1210 }
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1211
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1212 if (compressor->dict) {
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1213 if (compressor->dict->cdict) {
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1214 zresult = ZSTD_CCtx_refCDict(workerStates[i].cctx, compressor->dict->cdict);
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1215 }
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1216 else {
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1217 zresult = ZSTD_CCtx_loadDictionary_advanced(
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1218 workerStates[i].cctx,
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1219 compressor->dict->dictData,
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1220 compressor->dict->dictSize,
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1221 ZSTD_dlm_byRef,
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1222 compressor->dict->dictType);
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1223 }
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1224
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1225 if (ZSTD_isError(zresult)) {
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1226 PyErr_Format(ZstdError, "could not load compression dictionary: %s",
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1227 ZSTD_getErrorName(zresult));
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1228 goto finally;
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1229 }
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1230
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1231 }
31796 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1232
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1233 workerStates[i].sources = sources->sources;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1234 workerStates[i].sourcesSize = sources->sourcesSize;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1235 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1236
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1237 Py_BEGIN_ALLOW_THREADS
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1238 for (i = 0; i < sources->sourcesSize; i++) {
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1239 workerBytes += sources->sources[i].sourceSize;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1240
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1241 /*
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1242 * The last worker/thread needs to handle all remaining work. Don't
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1243 * trigger it prematurely. Defer to the block outside of the loop
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1244 * to run the last worker/thread. But do still process this loop
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1245 * so workerBytes is correct.
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1246 */
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1247 if (currentThread == threadCount - 1) {
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1248 continue;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1249 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1250
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1251 if (workerBytes >= bytesPerWorker) {
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1252 assert(currentThread < threadCount);
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1253 workerStates[currentThread].totalSourceSize = workerBytes;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1254 workerStates[currentThread].startOffset = workerStartOffset;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1255 workerStates[currentThread].endOffset = i;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1256
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1257 if (threadCount > 1) {
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1258 POOL_add(pool, (POOL_function)compress_worker, &workerStates[currentThread]);
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1259 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1260 else {
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1261 compress_worker(&workerStates[currentThread]);
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1262 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1263
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1264 currentThread++;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1265 workerStartOffset = i + 1;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1266 workerBytes = 0;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1267 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1268 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1269
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1270 if (workerBytes) {
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1271 assert(currentThread < threadCount);
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1272 workerStates[currentThread].totalSourceSize = workerBytes;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1273 workerStates[currentThread].startOffset = workerStartOffset;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1274 workerStates[currentThread].endOffset = sources->sourcesSize - 1;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1275
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1276 if (threadCount > 1) {
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1277 POOL_add(pool, (POOL_function)compress_worker, &workerStates[currentThread]);
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1278 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1279 else {
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1280 compress_worker(&workerStates[currentThread]);
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1281 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1282 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1283
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1284 if (threadCount > 1) {
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1285 POOL_free(pool);
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1286 pool = NULL;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1287 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1288
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1289 Py_END_ALLOW_THREADS
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1290
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1291 for (i = 0; i < threadCount; i++) {
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1292 switch (workerStates[i].error) {
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1293 case WorkerError_no_memory:
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1294 PyErr_NoMemory();
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1295 errored = 1;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1296 break;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1297
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1298 case WorkerError_zstd:
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1299 PyErr_Format(ZstdError, "error compressing item %zd: %s",
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1300 workerStates[i].errorOffset, ZSTD_getErrorName(workerStates[i].zresult));
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1301 errored = 1;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1302 break;
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1303
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1304 case WorkerError_nospace:
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1305 PyErr_Format(ZstdError, "error compressing item %zd: not enough space in output",
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1306 workerStates[i].errorOffset);
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1307 errored = 1;
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1308 break;
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1309
31796 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1310 default:
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1311 ;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1312 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1313
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1314 if (errored) {
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1315 break;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1316 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1317
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1318 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1319
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1320 if (errored) {
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1321 goto finally;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1322 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1323
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1324 segmentsCount = 0;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1325 for (i = 0; i < threadCount; i++) {
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1326 WorkerState* state = &workerStates[i];
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1327 segmentsCount += state->destCount;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1328 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1329
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1330 segmentsArg = PyTuple_New(segmentsCount);
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1331 if (NULL == segmentsArg) {
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1332 goto finally;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1333 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1334
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1335 segmentIndex = 0;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1336
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1337 for (i = 0; i < threadCount; i++) {
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1338 Py_ssize_t j;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1339 WorkerState* state = &workerStates[i];
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1340
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1341 for (j = 0; j < state->destCount; j++) {
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1342 DestBuffer* destBuffer = &state->destBuffers[j];
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1343 buffer = BufferWithSegments_FromMemory(destBuffer->dest, destBuffer->destSize,
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1344 destBuffer->segments, destBuffer->segmentsSize);
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1345
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1346 if (NULL == buffer) {
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1347 goto finally;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1348 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1349
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1350 /* Tell instance to use free() instsead of PyMem_Free(). */
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1351 buffer->useFree = 1;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1352
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1353 /*
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1354 * BufferWithSegments_FromMemory takes ownership of the backing memory.
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1355 * Unset it here so it doesn't get freed below.
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1356 */
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1357 destBuffer->dest = NULL;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1358 destBuffer->segments = NULL;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1359
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1360 PyTuple_SET_ITEM(segmentsArg, segmentIndex++, (PyObject*)buffer);
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1361 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1362 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1363
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1364 result = (ZstdBufferWithSegmentsCollection*)PyObject_CallObject(
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1365 (PyObject*)&ZstdBufferWithSegmentsCollectionType, segmentsArg);
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1366
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1367 finally:
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1368 Py_CLEAR(segmentsArg);
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1369
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1370 if (pool) {
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1371 POOL_free(pool);
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1372 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1373
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1374 if (workerStates) {
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1375 Py_ssize_t j;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1376
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1377 for (i = 0; i < threadCount; i++) {
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1378 WorkerState state = workerStates[i];
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1379
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1380 if (state.cctx) {
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1381 ZSTD_freeCCtx(state.cctx);
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1382 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1383
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1384 /* malloc() is used in worker thread. */
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1385
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1386 for (j = 0; j < state.destCount; j++) {
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1387 if (state.destBuffers) {
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1388 free(state.destBuffers[j].dest);
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1389 free(state.destBuffers[j].segments);
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1390 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1391 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1392
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1393
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1394 free(state.destBuffers);
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1395 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1396
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1397 PyMem_Free(workerStates);
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1398 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1399
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1400 return result;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1401 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1402
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1403 PyDoc_STRVAR(ZstdCompressor_multi_compress_to_buffer__doc__,
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1404 "Compress multiple pieces of data as a single operation\n"
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1405 "\n"
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1406 "Receives a ``BufferWithSegmentsCollection``, a ``BufferWithSegments``, or\n"
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1407 "a list of bytes like objects holding data to compress.\n"
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1408 "\n"
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1409 "Returns a ``BufferWithSegmentsCollection`` holding compressed data.\n"
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1410 "\n"
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1411 "This function is optimized to perform multiple compression operations as\n"
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1412 "as possible with as little overhead as possbile.\n"
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1413 );
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1414
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1415 static ZstdBufferWithSegmentsCollection* ZstdCompressor_multi_compress_to_buffer(ZstdCompressor* self, PyObject* args, PyObject* kwargs) {
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1416 static char* kwlist[] = {
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1417 "data",
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1418 "threads",
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1419 NULL
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1420 };
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1421
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1422 PyObject* data;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1423 int threads = 0;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1424 Py_buffer* dataBuffers = NULL;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1425 DataSources sources;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1426 Py_ssize_t i;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1427 Py_ssize_t sourceCount = 0;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1428 ZstdBufferWithSegmentsCollection* result = NULL;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1429
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1430 memset(&sources, 0, sizeof(sources));
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1431
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1432 if (!PyArg_ParseTupleAndKeywords(args, kwargs, "O\|i:multi_compress_to_buffer", kwlist,
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1433 &data, &threads)) {
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1434 return NULL;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1435 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1436
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1437 if (threads < 0) {
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1438 threads = cpu_count();
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1439 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1440
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1441 if (threads < 2) {
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1442 threads = 1;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1443 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1444
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1445 if (PyObject_TypeCheck(data, &ZstdBufferWithSegmentsType)) {
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1446 ZstdBufferWithSegments* buffer = (ZstdBufferWithSegments*)data;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1447
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1448 sources.sources = PyMem_Malloc(buffer->segmentCount * sizeof(DataSource));
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1449 if (NULL == sources.sources) {
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1450 PyErr_NoMemory();
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1451 goto finally;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1452 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1453
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1454 for (i = 0; i < buffer->segmentCount; i++) {
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1455 if (buffer->segments[i].length > SIZE_MAX) {
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1456 PyErr_Format(PyExc_ValueError,
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1457 "buffer segment %zd is too large for this platform", i);
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1458 goto finally;
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1459 }
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1460
31796 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1461 sources.sources[i].sourceData = (char*)buffer->data + buffer->segments[i].offset;
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1462 sources.sources[i].sourceSize = (size_t)buffer->segments[i].length;
31796 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1463 sources.totalSourceSize += buffer->segments[i].length;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1464 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1465
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1466 sources.sourcesSize = buffer->segmentCount;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1467 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1468 else if (PyObject_TypeCheck(data, &ZstdBufferWithSegmentsCollectionType)) {
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1469 Py_ssize_t j;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1470 Py_ssize_t offset = 0;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1471 ZstdBufferWithSegments* buffer;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1472 ZstdBufferWithSegmentsCollection* collection = (ZstdBufferWithSegmentsCollection*)data;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1473
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1474 sourceCount = BufferWithSegmentsCollection_length(collection);
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1475
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1476 sources.sources = PyMem_Malloc(sourceCount * sizeof(DataSource));
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1477 if (NULL == sources.sources) {
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1478 PyErr_NoMemory();
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1479 goto finally;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1480 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1481
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1482 for (i = 0; i < collection->bufferCount; i++) {
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1483 buffer = collection->buffers[i];
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1484
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1485 for (j = 0; j < buffer->segmentCount; j++) {
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1486 if (buffer->segments[j].length > SIZE_MAX) {
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1487 PyErr_Format(PyExc_ValueError,
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1488 "buffer segment %zd in buffer %zd is too large for this platform",
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1489 j, i);
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1490 goto finally;
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1491 }
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1492
31796 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1493 sources.sources[offset].sourceData = (char*)buffer->data + buffer->segments[j].offset;
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1494 sources.sources[offset].sourceSize = (size_t)buffer->segments[j].length;
31796 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1495 sources.totalSourceSize += buffer->segments[j].length;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1496
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1497 offset++;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1498 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1499 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1500
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1501 sources.sourcesSize = sourceCount;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1502 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1503 else if (PyList_Check(data)) {
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1504 sourceCount = PyList_GET_SIZE(data);
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1505
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1506 sources.sources = PyMem_Malloc(sourceCount * sizeof(DataSource));
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1507 if (NULL == sources.sources) {
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1508 PyErr_NoMemory();
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1509 goto finally;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1510 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1511
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1512 dataBuffers = PyMem_Malloc(sourceCount * sizeof(Py_buffer));
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1513 if (NULL == dataBuffers) {
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1514 PyErr_NoMemory();
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1515 goto finally;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1516 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1517
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1518 memset(dataBuffers, 0, sourceCount * sizeof(Py_buffer));
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1519
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1520 for (i = 0; i < sourceCount; i++) {
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1521 if (0 != PyObject_GetBuffer(PyList_GET_ITEM(data, i),
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1522 &dataBuffers[i], PyBUF_CONTIG_RO)) {
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1523 PyErr_Clear();
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1524 PyErr_Format(PyExc_TypeError, "item %zd not a bytes like object", i);
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1525 goto finally;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1526 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1527
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1528 sources.sources[i].sourceData = dataBuffers[i].buf;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1529 sources.sources[i].sourceSize = dataBuffers[i].len;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1530 sources.totalSourceSize += dataBuffers[i].len;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1531 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1532
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1533 sources.sourcesSize = sourceCount;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1534 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1535 else {
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1536 PyErr_SetString(PyExc_TypeError, "argument must be list of BufferWithSegments");
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1537 goto finally;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1538 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1539
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1540 if (0 == sources.sourcesSize) {
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1541 PyErr_SetString(PyExc_ValueError, "no source elements found");
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1542 goto finally;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1543 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1544
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1545 if (0 == sources.totalSourceSize) {
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1546 PyErr_SetString(PyExc_ValueError, "source elements are empty");
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1547 goto finally;
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1548 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1549
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1550 if (sources.totalSourceSize > SIZE_MAX) {
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1551 PyErr_SetString(PyExc_ValueError, "sources are too large for this platform");
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1552 goto finally;
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1553 }
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1554
31796 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1555 result = compress_from_datasources(self, &sources, threads);
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1556
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1557 finally:
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1558 PyMem_Free(sources.sources);
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1559
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1560 if (dataBuffers) {
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1561 for (i = 0; i < sourceCount; i++) {
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1562 PyBuffer_Release(&dataBuffers[i]);
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1563 }
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1564
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1565 PyMem_Free(dataBuffers);
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1566 }
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1567
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1568 return result;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1569 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1570
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1571 static PyMethodDef ZstdCompressor_methods[] = {
40121 73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	1572 { "chunker", (PyCFunction)ZstdCompressor_chunker,
73fef626dae3 zstandard: vendor python-zstandard 0.10.1 Gregory Szorc <gregory.szorc@gmail.com> parents: 37495 diff changeset	1573 METH_VARARGS \| METH_KEYWORDS, ZstdCompressor_chunker__doc__ },
30822 b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	1574 { "compress", (PyCFunction)ZstdCompressor_compress,
b54a2984cdd4 zstd: vendor python-zstandard 0.6.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30435 diff changeset	1575 METH_VARARGS \| METH_KEYWORDS, ZstdCompressor_compress__doc__ },
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1576 { "compressobj", (PyCFunction)ZstdCompressor_compressobj,
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1577 METH_VARARGS \| METH_KEYWORDS, ZstdCompressionObj__doc__ },
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1578 { "copy_stream", (PyCFunction)ZstdCompressor_copy_stream,
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1579 METH_VARARGS \| METH_KEYWORDS, ZstdCompressor_copy_stream__doc__ },
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1580 { "stream_reader", (PyCFunction)ZstdCompressor_stream_reader,
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1581 METH_VARARGS \| METH_KEYWORDS, ZstdCompressor_stream_reader__doc__ },
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1582 { "stream_writer", (PyCFunction)ZstdCompressor_stream_writer,
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1583 METH_VARARGS \| METH_KEYWORDS, ZstdCompressor_stream_writer___doc__ },
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1584 { "read_to_iter", (PyCFunction)ZstdCompressor_read_to_iter,
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1585 METH_VARARGS \| METH_KEYWORDS, ZstdCompressor_read_to_iter__doc__ },
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1586 /* TODO Remove deprecated API */
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1587 { "read_from", (PyCFunction)ZstdCompressor_read_to_iter,
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1588 METH_VARARGS \| METH_KEYWORDS, ZstdCompressor_read_to_iter__doc__ },
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1589 /* TODO remove deprecated API */
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1590 { "write_to", (PyCFunction)ZstdCompressor_stream_writer,
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1591 METH_VARARGS \| METH_KEYWORDS, ZstdCompressor_stream_writer___doc__ },
31796 e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1592 { "multi_compress_to_buffer", (PyCFunction)ZstdCompressor_multi_compress_to_buffer,
e0dc40530c5a zstd: vendor python-zstandard 0.8.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 30895 diff changeset	1593 METH_VARARGS \| METH_KEYWORDS, ZstdCompressor_multi_compress_to_buffer__doc__ },
37495 b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1594 { "memory_size", (PyCFunction)ZstdCompressor_memory_size,
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1595 METH_NOARGS, ZstdCompressor_memory_size__doc__ },
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1596 { "frame_progression", (PyCFunction)ZstdCompressor_frame_progression,
b1fb341d8a61 zstandard: vendor python-zstandard 0.9.0 Gregory Szorc <gregory.szorc@gmail.com> parents: 31796 diff changeset	1597 METH_NOARGS, ZstdCompressor_frame_progression__doc__ },
30435 b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1598 { NULL, NULL }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1599 };
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1600
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1601 PyTypeObject ZstdCompressorType = {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1602 PyVarObject_HEAD_INIT(NULL, 0)
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1603 "zstd.ZstdCompressor", /* tp_name */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1604 sizeof(ZstdCompressor), /* tp_basicsize */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1605 0, /* tp_itemsize */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1606 (destructor)ZstdCompressor_dealloc, /* tp_dealloc */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1607 0, /* tp_print */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1608 0, /* tp_getattr */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1609 0, /* tp_setattr */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1610 0, /* tp_compare */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1611 0, /* tp_repr */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1612 0, /* tp_as_number */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1613 0, /* tp_as_sequence */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1614 0, /* tp_as_mapping */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1615 0, /* tp_hash */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1616 0, /* tp_call */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1617 0, /* tp_str */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1618 0, /* tp_getattro */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1619 0, /* tp_setattro */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1620 0, /* tp_as_buffer */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1621 Py_TPFLAGS_DEFAULT \| Py_TPFLAGS_BASETYPE, /* tp_flags */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1622 ZstdCompressor__doc__, /* tp_doc */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1623 0, /* tp_traverse */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1624 0, /* tp_clear */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1625 0, /* tp_richcompare */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1626 0, /* tp_weaklistoffset */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1627 0, /* tp_iter */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1628 0, /* tp_iternext */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1629 ZstdCompressor_methods, /* tp_methods */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1630 0, /* tp_members */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1631 0, /* tp_getset */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1632 0, /* tp_base */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1633 0, /* tp_dict */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1634 0, /* tp_descr_get */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1635 0, /* tp_descr_set */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1636 0, /* tp_dictoffset */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1637 (initproc)ZstdCompressor_init, /* tp_init */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1638 0, /* tp_alloc */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1639 PyType_GenericNew, /* tp_new */
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1640 };
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1641
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1642 void compressor_module_init(PyObject* mod) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1643 Py_TYPE(&ZstdCompressorType) = &PyType_Type;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1644 if (PyType_Ready(&ZstdCompressorType) < 0) {
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1645 return;
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1646 }
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1647
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1648 Py_INCREF((PyObject*)&ZstdCompressorType);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1649 PyModule_AddObject(mod, "ZstdCompressor",
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1650 (PyObject*)&ZstdCompressorType);
b86a448a2965 zstd: vendor python-zstandard 0.5.0 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1651 }

Mercurial > public > mercurial-scm > hg

annotate contrib/python-zstandard/c-ext/compressor.c @ 40121:73fef626dae3