crumpets.shm module

class crumpets.shm.DummyBufferManager

Bases: object

Dummy replacement for SharedBufferManager. Supports pack and unpack, but not next methods.

next()

pack(obj)

Pack an object using msgpack. :param obj: object to pack :return: msgpack message bytes

unpack(data)

Unpack an msgpack message. :param data: msgpack message bytes :return: packed objects

class crumpets.shm.SharedBufferManager(num_buffers, batch_size, buffer_specs, _queueclass=<bound method BaseContext.Queue of <multiprocessing.context.DefaultContext object>>)

Bases: object

SharedBufferManager allows transparent sharing of memory between processes. On creation the specified number of shared memory buffers are created according to batch size and buffer specs.

next returns dict of numpy arrays that point to a set of shared memory buffers. next blocks until as set of buffers becomes available. If more than one buffer spec is given, next will always return one buffer for each spec and will only reuse a set of buffers when none of them are in use.

pack serializes an arbitrary python object to msgpack format. It detects shared buffers and replaces them with a “pointer” as extension type EXT_SHARED. This makes packing fast and independent of array size.

unpack detects “pointer” and replaces them with the shared buffer.

Usage:

• Sender calls next to get a set ob available buffers.

• Sender modifies buffers, calls pack and sends message to receiver.

• Receiver receives the message and calls unpack.

• Receiver uses the unpacked arrays and ensures that they are deleted at some point, either by going out of scope or explicitly deleting them. Storing shared buffers permanently may cause deadlocks.

close()

Close the queue and unblock any processes waiting on next.

next()

pack(obj)

Pack an object using msgpack. Any shared object are replaced by references. :param obj: object to pack :return: msgpack message bytes

unpack(data)

Unpack an msgpack message. Any shared object references are replaced with the object. :param data: msgpack message bytes :return: packed objects

crumpets.shm.shared_array(shape, dtype=<class 'numpy.float32'>)

Create a numpy array that resides in shared memory. Memory is aligned to 8 bytes.

Parameters

• shape – array shape

• dtype – numpy dtype

Returns

np.ndarray