Buffer Types

The zstandard module exposes a handful of custom types for interfacing with memory buffers. The primary goal of these types is to facilitate efficient multi-object operations.

The essential idea is to have a single memory allocation provide backing storage for multiple logical objects. This has 2 main advantages: fewer allocations and optimal memory access patterns. This avoids having to allocate a Python object for each logical object and furthermore ensures that access of data for objects can be sequential (read: fast) in memory.

BufferSegment

class zstandard.BufferSegment

Represents a segment within a BufferWithSegments.

This type is essentially a reference to N bytes within a BufferWithSegments.

The object conforms to the buffer protocol.

offset

The byte offset of this segment within its parent buffer.

tobytes()

Obtain bytes copy of this segment.

BufferSegments

class zstandard.BufferSegments

Represents an array of (offset, length) integers.

This type is effectively an index used by BufferWithSegments.

The array members are 64-bit unsigned integers using host/native bit order.

Instances conform to the buffer protocol.

BufferWithSegments

class zstandard.BufferWithSegments

A memory buffer containing N discrete items of known lengths.

This type is essentially a fixed size memory address and an array of 2-tuples of (offset, length) 64-bit unsigned native-endian integers defining the byte offset and length of each segment within the buffer.

Instances behave like containers.

Instances also conform to the buffer protocol. So a reference to the backing bytes can be obtained via memoryview(o). A copy of the backing bytes can be obtained via .tobytes().

This type exists to facilitate operations against N>1 items without the overhead of Python object creation and management. Used with APIs like ZstdDecompressor.multi_decompress_to_buffer(), it is possible to decompress many objects in parallel without the GIL held, leading to even better performance.

segments()

Obtain the array of (offset, length) segments in the buffer.

Returns:BufferSegments

size

Total sizein bytes of the backing buffer.

tobytes()

Obtain bytes copy of this instance.

BufferWithSegmentsCollection

class zstandard.BufferWithSegmentsCollection

A virtual spanning view over multiple BufferWithSegments.

Instances are constructed from 1 or more BufferWithSegments instances. The resulting object behaves like an ordered sequence whose members are the segments within each BufferWithSegments.

If the object is composed of 2 BufferWithSegments instances with the first having 2 segments and the second have 3 segments, then b[0] and b[1] access segments in the first object and b[2], b[3], and b[4] access segments from the second.