Buffer Types¶
The zstandard
module exposes a handful of custom types for interfacing with
memory buffers. The primary goal of these types is to facilitate efficient
multi-object operations.
The essential idea is to have a single memory allocation provide backing storage for multiple logical objects. This has 2 main advantages: fewer allocations and optimal memory access patterns. This avoids having to allocate a Python object for each logical object and furthermore ensures that access of data for objects can be sequential (read: fast) in memory.
BufferSegment
¶
-
class
zstandard.
BufferSegment
¶ Represents a segment within a
BufferWithSegments
.This type is essentially a reference to N bytes within a
BufferWithSegments
.The object conforms to the buffer protocol.
-
offset
¶ The byte offset of this segment within its parent buffer.
-
tobytes
()¶ Obtain bytes copy of this segment.
-
BufferSegments
¶
-
class
zstandard.
BufferSegments
¶ Represents an array of
(offset, length)
integers.This type is effectively an index used by
BufferWithSegments
.The array members are 64-bit unsigned integers using host/native bit order.
Instances conform to the buffer protocol.
BufferWithSegments
¶
-
class
zstandard.
BufferWithSegments
¶ A memory buffer containing N discrete items of known lengths.
This type is essentially a fixed size memory address and an array of 2-tuples of
(offset, length)
64-bit unsigned native-endian integers defining the byte offset and length of each segment within the buffer.Instances behave like containers.
Instances also conform to the buffer protocol. So a reference to the backing bytes can be obtained via
memoryview(o)
. A copy of the backing bytes can be obtained via.tobytes()
.This type exists to facilitate operations against N>1 items without the overhead of Python object creation and management. Used with APIs like
ZstdDecompressor.multi_decompress_to_buffer()
, it is possible to decompress many objects in parallel without the GIL held, leading to even better performance.-
segments
()¶ Obtain the array of
(offset, length)
segments in the buffer.Returns: BufferSegments
-
size
¶ Total sizein bytes of the backing buffer.
-
tobytes
()¶ Obtain bytes copy of this instance.
-
BufferWithSegmentsCollection
¶
-
class
zstandard.
BufferWithSegmentsCollection
¶ A virtual spanning view over multiple BufferWithSegments.
Instances are constructed from 1 or more
BufferWithSegments
instances. The resulting object behaves like an ordered sequence whose members are the segments within eachBufferWithSegments
.If the object is composed of 2
BufferWithSegments
instances with the first having 2 segments and the second have 3 segments, thenb[0]
andb[1]
access segments in the first object andb[2]
,b[3]
, andb[4]
access segments from the second.