pip install --upgrade cramjam # Requires no Python or system dependencies!
By default zstd, lz4, bzip2, xz and deflate/gzip/zlib use their C libraries.
To use libcramjam's pure-Rust
implementations instead (same API, no C), install the add-on and select it
before cramjam is imported:
pip install --upgrade "cramjam[pure-rust]"
export CRAMJAM_BACKEND=pure-rust # `cramjam.backend` reports which is active
Output is interchangeable between backends, but the compressed bytes may differ.
The one-shot decompress functions for the standard codecs, plus
snappy.decompress_raw and lz4.decompress_block, accept an optional keyword-only
max_output_size in bytes:
try:
decoded = cramjam.zstd.decompress(compressed, max_output_size=16 * 1024 * 1024)
except cramjam.OutputLimitExceededError: # a subclass of cramjam.DecompressionError
...Exceeding the limit raises an error; it never returns truncated output. The limit
is cumulative across all frames decoded by a call. Zero permits only empty
output. Omitting the option (or passing None) preserves the existing behavior.
The default backend supports all standard codecs. The pure-Rust backend currently
supports bounded Snappy, Brotli and Bzip2; other codecs reject a requested limit
with NotImplementedError. This limits output storage, not decoder workspace or
total process memory. In particular, pure-Rust Bzip2 may temporarily retain about
44.5 MiB of decoded block data before writing to the bounded sink.
See bounded decompression for backend
limitations, allocation errors and the intentional differences in output_len
semantics between bounded and unbounded calls.
Install the locked development dependencies, build the extension, and run the test suite:
make dev-setup
make testRust checks can be run separately with cargo test.
npm install cramjam
A CLI interface is available as cramjam-cli
A Rust crate and C friendly library available at libcramjam
Extremely thin and easy-to-install Python bindings to de/compression algorithms in Rust. Allows for using algorithms such as Snappy, without any system or other python dependencies.
Some basic benchmarks are available in the benchmarks directory
Available algorithms:
- Snappy
cramjam.snappy - Brotli
cramjam.brotli - Bzip2
cramjam.bzip2 - Lz4
cramjam.lz4 - Gzip
cramjam.gzip - Zlib
cramjam.zlib - Deflate
cramjam.deflate - ZSTD
cramjam.zstd - XZ / LZMA
cramjam.xz
Experimental (Requires build from source enabling each feature):
- Blosc2
cramjam.experimental.blosc2 - ISA-L backend (only on 64-bit targets)
- igzip
cramjam.experimental.igzip - ideflate
cramjam.experimental.ideflate - izlib
cramjam.experimental.izlib
- igzip
All available for use as:
>>> import cramjam
>>> import numpy as np
>>> compressed = cramjam.snappy.compress(b"bytes here")
>>> decompressed = cramjam.snappy.decompress(compressed)
>>> decompressed
cramjam.Buffer(len=10) # an object which implements the buffer protocol
>>> bytes(decompressed)
b"bytes here"
>>> np.frombuffer(decompressed, dtype=np.uint8)
array([ 98, 121, 116, 101, 115, 32, 104, 101, 114, 101], dtype=uint8)Where the API is cramjam.<compression-variant>.compress/decompress and accepts
bytes/bytearray/numpy.array/cramjam.File/cramjam.Buffer / memoryview objects.
de/compress_into
Additionally, all variants support decompress_into and compress_into.
Ex.
>>> import numpy as np
>>> from cramjam import snappy, Buffer
>>>
>>> data = np.frombuffer(b'some bytes here', dtype=np.uint8)
>>> data
array([115, 111, 109, 101, 32, 98, 121, 116, 101, 115, 32, 104, 101,
114, 101], dtype=uint8)
>>>
>>> compressed = Buffer()
>>> snappy.compress_into(data, compressed)
33 # 33 bytes written to compressed buffer
>>>
>>> compressed.tell() # Where is the buffer position?
33 # goodie!
>>>
>>> compressed.seek(0) # Go back to the start of the buffer so we can prepare to decompress
>>> decompressed = b'0' * len(data) # let's write to `bytes` as output
>>> decompressed
b'000000000000000'
>>>
>>> snappy.decompress_into(compressed, decompressed)
15 # 15 bytes written to decompressed
>>> decompressed
b'some bytes here'import {Compress, Decompress} from 'cramjam';
const decoder = new TextDecoder();
const encoder = new TextEncoder();
const str = 'hello, world';
const encoded = encoder.encode(str);
const compressed = Compress.brotli(encoded);
const decompressed = Decompress.brotli(compressed);
const decoded = decoder.decode(decompressed);