Skip to content

gh-132983: Introduce _zstd bindings module #133027

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 37 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
9814e3b
Add _zstd module
emmatyping Apr 26, 2025
fda87c8
Add _zstd to modules
emmatyping Apr 26, 2025
887e564
Fix path for compression.zstd module
emmatyping Apr 26, 2025
cdba656
Ignore _zstd module like _io
emmatyping Apr 26, 2025
6b67e9b
Expand module state macros to improve code quality
emmatyping Apr 26, 2025
a99a5d2
Remove backticks suggested in review
emmatyping Apr 27, 2025
02cd17a
Use critical sections to lock object state
emmatyping Apr 27, 2025
54eca74
Remove compress/decompress and mark module as not reliant on the GIL
emmatyping Apr 27, 2025
f605956
Lift critical section to avoid clang warning
emmatyping Apr 27, 2025
2eadc65
Respond to comments by picnixz
emmatyping Apr 27, 2025
8eac354
Call out pyzstd explicitly in license description
emmatyping Apr 27, 2025
26775be
Use a much more robust implementation...
emmatyping Apr 27, 2025
eae460f
Use PyList_GetItemRef for thread safety purposes
emmatyping Apr 27, 2025
2ab5e4a
Use a macro for the minimum supported version
emmatyping Apr 27, 2025
d5bf1c1
remove const from primivite types
emmatyping Apr 27, 2025
9e92b9f
Use PyMem_New in another spot
emmatyping Apr 27, 2025
47f815a
Simplify error handling in _get_frame_size
emmatyping Apr 27, 2025
6a4f7b8
Another simplification of error handling in get_frame_info
emmatyping Apr 27, 2025
d7b3805
Rename _module_state to mod_state
emmatyping Apr 27, 2025
c225ea6
Rewrite comment explaining the context of the code
emmatyping Apr 28, 2025
6e8c61c
Add link to pyzstd
emmatyping Apr 28, 2025
e52ad06
Add TODO about refactoring dict training code
emmatyping Apr 28, 2025
2a1ad8b
Use PyModule_AddObjectRef over PyModule_AddObject
emmatyping Apr 28, 2025
94473b9
Check result of OutputBufferGrow
emmatyping Apr 28, 2025
e2b2515
Simplify return logic in `add_constant_to_type`
emmatyping Apr 29, 2025
cd2f085
Ignore return value of _zstd_clear()
emmatyping Apr 29, 2025
79e174f
Remove redundant comments
emmatyping Apr 29, 2025
ce6f79c
Remove __reduce__ from ZstdDict
emmatyping Apr 29, 2025
e15dd85
Use PyUnicode_FromFormat instead of a buffer
emmatyping Apr 29, 2025
685a3d1
Don't use C constants/types in error messages
emmatyping Apr 29, 2025
1b9f786
Make error messages easier to understand for Python users
emmatyping Apr 29, 2025
40c653c
Lower minimum required version 1.4.0
emmatyping Apr 30, 2025
428677d
Use casts and make slot function signatures correct
emmatyping Apr 30, 2025
0962bbb
Be consistent with CPython on const usage
emmatyping Apr 30, 2025
85efc18
Make else clauses in line with PEP 7
emmatyping Apr 30, 2025
cadf6e4
Fix over-indented blocks in argument clinic
emmatyping Apr 30, 2025
e45c22a
Merge branch 'main' into 3.14-zstd-c-code
emmatyping Apr 30, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 37 additions & 0 deletions Doc/license.rst
Original file line number Diff line number Diff line change
Expand Up @@ -1132,3 +1132,40 @@ The file is distributed under the 2-Clause BSD License::
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.


Zstandard bindings
------------------

Zstandard bindings in :file:`Modules/_zstd` and :file:`Lib/compression/zstd`
are based on code from the
`pyzstd library <https://github.com/Rogdham/pyzstd/>`_, copyright Ma Lin and
contributors. The pyzstd code is distributed under the 3-Clause BSD License::

Copyright (c) 2020-present, Ma Lin and contributors.
All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:

1. Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.

2. Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.

3. Neither the name of the copyright holder nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
8 changes: 8 additions & 0 deletions Include/internal/pycore_global_objects_fini_generated.h

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

8 changes: 8 additions & 0 deletions Include/internal/pycore_global_strings.h
Original file line number Diff line number Diff line change
Expand Up @@ -325,6 +325,7 @@ struct _Py_global_strings {
STRUCT_FOR_ID(bytes_per_sep)
STRUCT_FOR_ID(c_call)
STRUCT_FOR_ID(c_exception)
STRUCT_FOR_ID(c_parameter_type)
STRUCT_FOR_ID(c_return)
STRUCT_FOR_ID(cached_datetime_module)
STRUCT_FOR_ID(cached_statements)
Expand Down Expand Up @@ -378,6 +379,7 @@ struct _Py_global_strings {
STRUCT_FOR_ID(count)
STRUCT_FOR_ID(covariant)
STRUCT_FOR_ID(cwd)
STRUCT_FOR_ID(d_parameter_type)
STRUCT_FOR_ID(data)
STRUCT_FOR_ID(database)
STRUCT_FOR_ID(day)
Expand All @@ -392,6 +394,7 @@ struct _Py_global_strings {
STRUCT_FOR_ID(deterministic)
STRUCT_FOR_ID(device)
STRUCT_FOR_ID(dict)
STRUCT_FOR_ID(dict_content)
STRUCT_FOR_ID(dictcomp)
STRUCT_FOR_ID(difference_update)
STRUCT_FOR_ID(digest)
Expand Down Expand Up @@ -457,6 +460,7 @@ struct _Py_global_strings {
STRUCT_FOR_ID(follow_symlinks)
STRUCT_FOR_ID(format)
STRUCT_FOR_ID(format_spec)
STRUCT_FOR_ID(frame_buffer)
STRUCT_FOR_ID(from_param)
STRUCT_FOR_ID(fromlist)
STRUCT_FOR_ID(fromtimestamp)
Expand Down Expand Up @@ -515,6 +519,8 @@ struct _Py_global_strings {
STRUCT_FOR_ID(intersection)
STRUCT_FOR_ID(interval)
STRUCT_FOR_ID(io)
STRUCT_FOR_ID(is_compress)
STRUCT_FOR_ID(is_raw)
STRUCT_FOR_ID(is_running)
STRUCT_FOR_ID(is_struct)
STRUCT_FOR_ID(isatty)
Expand Down Expand Up @@ -638,6 +644,7 @@ struct _Py_global_strings {
STRUCT_FOR_ID(overlapped)
STRUCT_FOR_ID(owner)
STRUCT_FOR_ID(pages)
STRUCT_FOR_ID(parameter)
STRUCT_FOR_ID(parent)
STRUCT_FOR_ID(password)
STRUCT_FOR_ID(path)
Expand Down Expand Up @@ -799,6 +806,7 @@ struct _Py_global_strings {
STRUCT_FOR_ID(write_through)
STRUCT_FOR_ID(year)
STRUCT_FOR_ID(zdict)
STRUCT_FOR_ID(zstd_dict)
} identifiers;
struct {
PyASCIIObject _ascii;
Expand Down
8 changes: 8 additions & 0 deletions Include/internal/pycore_runtime_init_generated.h

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

32 changes: 32 additions & 0 deletions Include/internal/pycore_unicodeobject_generated.h

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions Makefile.pre.in
Original file line number Diff line number Diff line change
Expand Up @@ -3335,6 +3335,7 @@ MODULE__TESTCAPI_DEPS=$(srcdir)/Modules/_testcapi/parts.h $(srcdir)/Modules/_tes
MODULE__TESTLIMITEDCAPI_DEPS=$(srcdir)/Modules/_testlimitedcapi/testcapi_long.h $(srcdir)/Modules/_testlimitedcapi/parts.h $(srcdir)/Modules/_testlimitedcapi/util.h
MODULE__TESTINTERNALCAPI_DEPS=$(srcdir)/Modules/_testinternalcapi/parts.h
MODULE__SQLITE3_DEPS=$(srcdir)/Modules/_sqlite/connection.h $(srcdir)/Modules/_sqlite/cursor.h $(srcdir)/Modules/_sqlite/microprotocols.h $(srcdir)/Modules/_sqlite/module.h $(srcdir)/Modules/_sqlite/prepare_protocol.h $(srcdir)/Modules/_sqlite/row.h $(srcdir)/Modules/_sqlite/util.h
MODULE__ZSTD_DEPS=$(srcdir)/Modules/_zstd/_zstdmodule.h $(srcdir)/Modules/_zstd/buffer.h

CODECS_COMMON_HEADERS=$(srcdir)/Modules/cjkcodecs/multibytecodec.h $(srcdir)/Modules/cjkcodecs/cjkcodecs.h
MODULE__CODECS_CN_DEPS=$(srcdir)/Modules/cjkcodecs/mappings_cn.h $(CODECS_COMMON_HEADERS)
Expand Down
1 change: 1 addition & 0 deletions Modules/Setup
Original file line number Diff line number Diff line change
Expand Up @@ -200,6 +200,7 @@ PYTHONPATH=$(COREPYTHONPATH)
#_dbm _dbmmodule.c -lgdbm_compat -DUSE_GDBM_COMPAT
#_gdbm _gdbmmodule.c -lgdbm
#_lzma _lzmamodule.c -llzma
#_zstd _zstd/_zstdmodule.c -lzstd -I$(srcdir)/Modules/_zstd
#_uuid _uuidmodule.c -luuid
#zlib zlibmodule.c -lz

Expand Down
3 changes: 2 additions & 1 deletion Modules/Setup.stdlib.in
Original file line number Diff line number Diff line change
Expand Up @@ -64,10 +64,11 @@
@MODULE__DECIMAL_TRUE@_decimal _decimal/_decimal.c

# compression libs and binascii (optional CRC32 from zlib)
# bindings need -lbz2, -lz, or -llzma, respectively
# bindings need -lbz2, -llzma, -lzstd, or -lz, respectively
@MODULE_BINASCII_TRUE@binascii binascii.c
@MODULE__BZ2_TRUE@_bz2 _bz2module.c
@MODULE__LZMA_TRUE@_lzma _lzmamodule.c
@MODULE__ZSTD_TRUE@_zstd _zstd/_zstdmodule.c _zstd/zdict.c _zstd/compressor.c _zstd/decompressor.c
@MODULE_ZLIB_TRUE@zlib zlibmodule.c

# dbm/gdbm
Expand Down
Loading
Loading