POSIX IPC for Python - Semaphores, Shared Memory and Message Queues


RSS

The Python extension module posix_ipc gives Python access to POSIX inter-process semaphores, shared memory and message queues on systems that support the POSIX Realtime Extensions a.k.a. POSIX 1003.1b-1993. That includes most (all?) Linuxes with kernel ≥ 2.6, FreeBSD ≥ 7.2, and OpenSolaris ≥ 2008.05.

OS X and other Unix-y platforms (including Windows + Cygwin 1.7) provide partial (or partially broken) support. See the platform notes below for more details.

This module works under Python 2.4 – 3.4. It is released under a BSD license.

You can download posix_ipc version 0.9.9 [MD5 sum] [SHA1 sum] which contains the source code, setup.py, installation instructions, tests, and sample code. The exact same posix_ipc tarball is also available on PyPI. You can also find the posix_ipc source code on BitBucket.

You might want to read all of the changes in this version and about some known bugs.

Note that this module doesn't support unnamed (anonymous) POSIX semaphores.

You might be interested in the very similar module sysv_ipc which provides Python access to IPC using System V semaphores, shared memory and message queues. System V IPC has broader OS support but is not as easy to use.

Module posix_ipc Documentation

Jump to semaphores, shared memory, or message queues.

Module Functions

unlink_semaphore(name)
unlink_shared_memory(name)
unlink_message_queue(name)
Convenience functions that unlink the IPC object described by name.

Module Constants

O_CREX, O_CREAT, O_EXCL and O_TRUNC
These flags are used when creating IPC objects. All except O_CREX are bitwise unique and can be ORed together. O_CREX is shorthand for O_CREAT | O_EXCL.

O_TRUNC is only useful when creating SharedMemory objects.

PAGE_SIZE
The operating system's memory page size, in bytes. It's probably a good idea to make shared memory segments some multiple of this size.
SEMAPHORE_TIMEOUT_SUPPORTED
True if the underlying OS supports sem_timedwait(). If False, all timeouts > 0 passed to a semaphore's acquire() method are treated as infinity.

As far as I know, this is only False under OS X.

SEMAPHORE_VALUE_SUPPORTED
True if the underlying OS supports sem_getvalue(). If False, accessing the value attribute on a Semaphore instance will raise an AttributeError.

As far as I know, this is only False under OS X.

SEMAPHORE_VALUE_MAX
The maximum value that can be assigned to a semaphore.
MESSAGE_QUEUES_SUPPORTED
True if the underlying OS supports message queues, False otherwise.
QUEUE_MESSAGES_MAX_DEFAULT
The default value for a message queue's max_messages attribute. This can be quite small under Linux (e.g. 10) but is usually LONG_MAX everywhere else.
QUEUE_MESSAGE_SIZE_MAX_DEFAULT
The default value for a message queue's max_message_size attribute. This is 8k (or possibly smaller under Linux).
QUEUE_PRIORITY_MAX
The maximum message queue message priority.
USER_SIGNAL_MIN, USER_SIGNAL_MAX
The constants define a range of signal values reserved for use by user applications (like yours). They're available only on systems that support the POSIX Realtime Signals Extension. Most systems do; NetBSD versions prior to 6.0 are a notable exception.
VERSION
The module version string, e.g. '0.9.8'. This is also available as __version__.

Module Errors

In addition to standard Python errors (e.g. ValueError), this module raises custom errors. These errors cover situations specific to IPC.

Error
The base error class for all the custom errors in this module.
SignalError
Raised when a waiting call (e.g. sem.acquire()) is interrupted by a signal other than KeyboardInterrupt.
PermissionsError
Indicates that you've attempted something that the permissions on the IPC object don't allow.
ExistentialError
Indicates an error related to the existence or non-existence of an IPC object.
BusyError
Raised when a call times out.

The Semaphore Class

This is a handle to a semaphore.

Methods

Semaphore(name, [flags = 0, [mode = 0600, [initial_value = 0]]])
Creates a new semaphore or opens an existing one.

name must be None or a string. If it is None, the module chooses a random unused name. If it is a string, it should begin with a slash and be valid according to pathname rules on your system, e.g. /wuthering_heights_by_semaphore

The flags specify whether you want to create a new semaphore or open an existing one.

acquire([timeout=None])
Waits (conditionally) until the semaphore's value is > 0 and then returns, decrementing the semaphore.

The timeout (which can be a float) specifies how many seconds this call should wait, if at all.

release()
Releases (increments) the semaphore.
close()
Closes the semaphore, indicating that the current process is done with the semaphore. The effect of subsequent use of the semaphore by the current process is undefined. Assuming it still exists, (see unlink(), below) the semaphore can be re-opened.

You must call close() explicitly; it is not called automatically when a Semaphore object is garbage collected.

Destroys the semaphore, with a caveat. If any processes have the semaphore open when unlink is called, the call to unlink returns immediately but destruction of the semaphore is postponed until all processes have closed the semaphore.

Note, however, that once a semaphore has been unlinked, calls to open() with the same name should refer to a new semaphore. Sound confusing? It is, and you'd probably be wise structure your code so as to avoid this situation.

Attributes

name (read-only)
The name provided in the constructor.
value (read-only)
The integer value of the semaphore. Not available on OS X. (See Platforms)

Context Manager Support

These semaphores provide __enter__() and __exit__() methods so they can be used in context managers. For instance --

with posix_ipc.Semaphore(name) as sem:
    # Do something...

Entering the context acquires the semaphore, exiting the context releases the semaphore. See demo4/child.py for a complete example.

The SharedMemory Class

This is a handle to a shared memory segment. POSIX shared memory segments masquerade as files, and so the handle to a shared memory segment is just a file descriptor that can be mmapped.

Methods

SharedMemory(name, [flags = 0, [mode = 0600, [size = 0, [read_only = false]]]])
Creates a new shared memory segment or opens an existing one.

name must be None or a string. If it is None, the module chooses a random unused name. If it is a string, it should begin with a slash and be valid according to pathname rules on your system, e.g. /four_yorkshiremen_sharing_memories

On some systems you need to have write access to the path.

The flags specify whether you want to create a new shared memory segment or open an existing one.

When opening an existing shared memory segment, one can also specify the flag O_TRUNC to truncate the shared memory to zero bytes. OS X does not appear to support O_TRUNC.

close_fd()
Closes the file descriptor associated with this SharedMemory object. Calling close_fd() is the same as calling os.close() on a SharedMemory object's fd attribute.

You must call close_fd() or os.close() explicitly; the file descriptor is not closed automatically when a SharedMemory object is garbage collected.

Closing the file descriptor has no effect on any mmap objects that were created from it. See the demo for an example.

unlink()
Marks the shared memory for destruction once all processes have unmapped it.

The POSIX specification for shm_unlink() says, "Even if the object continues to exist after the last shm_unlink(), reuse of the name shall subsequently cause shm_open() to behave as if no shared memory object of this name exists (that is, shm_open() will fail if O_CREAT is not set, or will create a new shared memory object if O_CREAT is set)."

I'll bet a virtual cup of coffee that this tricky part of the standard is not well or consistently implemented in every OS. Caveat emptor.

Attributes

name (read-only)
The name provided in the constructor.
fd (read-only)
The file descriptor that represents the memory segment.
size (read-only)
The size (in bytes) of the shared memory segment.

The MessageQueue Class

This is a handle to a message queue.

Methods

MessageQueue(name, [flags = 0, [mode = 0600, [max_messages = QUEUE_MESSAGES_MAX_DEFAULT, [max_message_size = QUEUE_MESSAGE_SIZE_MAX_DEFAULT, [read = True, [write = True]]]]]])
Creates a new message queue or opens an existing one.

name must be None or a string. If it is None, the module chooses a random unused name. If it is a string, it should begin with a slash and be valid according to pathname rules on your system, e.g. /my_message_queue

On some systems you need to have write access to the path.

The flags specify whether you want to create a new queue or open an existing one.

Max_messages defines how many messages can be in the queue at one time. When the queue is full, calls to .send() will wait.

Max_message_size defines the maximum size (in bytes) of a message.

Read and write default to True. If read/write is False, calling .receive()/.send() on this object is not permitted. This doesn't affect other handles to the same queue.

send(message, [timeout = None, [priority = 0]])
Sends a message via the queue.

The message string can contain embedded NULLs (ASCII 0x00). Under Python 3, the message can also be a bytes object.

The timeout (which can be a float) specifies how many seconds this call should wait if the queue is full. Timeouts are irrelevant when the block flag is False.

The priority allows you to order messages in the queue. The highest priority message is received first. By default, messages are sent at the lowest priority (0).

receive([timeout])
Receives a message from the queue, returning a tuple of (message, priority). Messages are received in the order of highest priority to lowest, and in FIFO order for messages of equal priority. Under Python 3, the returned message is a bytes object.

If the queue is empty, the call will not return immediately. The optional timeout parameter controls the wait just as for the function send(). It defaults to None.

request_notification([notification])
Depending on the parameter, requests or cancels notification from the operating system when the queue changes from empty to non-empty.

Message queues accept only one notification request at a time. If another process has already requested notifications from this queue, this call will fail with a BusyError.

The operating system delivers (at most) one notification per request. If you want subsequent notifications, you must request them by calling request_notification() again.

close()
Closes this reference to the queue.

You must call close() explicitly; it is not called automatically when a MessageQueue object is garbage collected.

unlink()
Requests destruction of the queue. Although the call returns immediately, actual destruction of the queue is postponed until all references to it are closed.

Attributes

name (read-only)
The name provided in the constructor.
mqd (read-only)
The message queue descriptor that represents the queue.
block
When True (the default), calls to .send() and .receive() may wait (block) if they cannot immediately satisfy the send/receive request. When block is False, the module will raise BusyError instead of waiting.
max_messages (read-only)
The maximum number of messages the queue can hold.
max_message_size (read-only)
The maximum message size (in bytes).
current_messages (read-only)
The number of messages currently in the queue.

Usage Tips

Tests

This module comes with fairly complete unit tests in the tests directory. To run them, install posix_ipc and then run this command from the same directory as setup.py:
python -m unittest discover

Sample Code

This module comes with three demonstrations. The first (in the directory demo) shows how to use shared memory and semaphores. The second (in the directory demo2) shows how to use message queues. The third (demo3) shows how to use message queue notifications.

Nobody Likes a Mr. Messy

IPC objects are a little different from most Python objects and therefore require a little more care on the part of the programmer. When a program creates a IPC object, it creates something that resides outside of its own process, just like a file on a hard drive. It won't go away when your process ends unless you explicitly remove it. And since many operating systems don't even give you a way to enumerate existing POSIX IPC entities, it might be hard to figure out what you're leaving behind.

In short, remember to clean up after yourself.

Semaphores and References

I know it's verboten to talk about pointers in Python, but I'm going to do it anyway.

Each Semaphore object created by this module contains a C pointer to the IPC object created by the system. When you call sem.close(), the object's internal pointer is set to NULL. This leaves the object in a not-quite-useless state. You can still call sem.unlink() or print sem.name, but calls to sem.aquire() or sem.release() will raise an ExistentialError.

If you know you're not going to use a Semaphore object after calling sem.close() or sem.unlink(), you could you set your semaphore variable to the return from the function (which is always None) like so:

    my_sem = my_sem.close()

That will ensure you don't have any nearly useless objects laying around that you might use by accident.

This doesn't apply to shared memory and message queues because they're referenced at the C level by a file descriptor rather than a pointer.

Permissions

It appears that the read and write mode bits on IPC objects are ignored by the operating system. For instance, on OS X, OpenSolaris and Linux one can write to semaphores and message queues with a mode of 0400.

Message Queues

When creating a new message queue, you specify a maximum message size which defaults to QUEUE_MESSAGE_SIZE_MAX_DEFAULT (currently 8192 bytes). You can create a queue with a larger value, but be aware that posix_ipc allocates a buffer the size of the maximum message size every time receive() is called.

Resizing Shared Memory Segments

Under OS X/Darwin, ftruncate() can be used to set the memory size once after the initial call to shm_open(). This module does that in the SharedMemory constructor, so subsequent attempts to resize the shared memory will fail.

I don't know if this holds true on all platforms. If your platform supports multiple calls to ftruncate(), you can call that via Python's os module, passing the file descriptor exposed in the SharedMemory object.

Consult Your Local man Pages

The posix_ipc module is just a wrapper around your system's API. If your system's implementation has quirks, the man pages for sem_open, sem_post, sem_wait, sem_close, sem_unlink, shm_open, shm_unlink, mq_open, mq_send mq_receive, mq_getattr, mq_close, mq_unlink and mq_notify will probably cover them.

Last But Not Least

For Pythonistas –

Known Bugs

I don't know of any bugs in this code. However, under Python 3 the standard library modules accept bytes and bytearray objects for filenames in addition to strings. One could argue that this module should behave the same way.

Also, this module doesn't support Python 3 memory views, which it probably should (for shared memory objects). Support for that might come in a later version.

Platform Notes

This module is just a wrapper around the operating system's functions, so if the operating system doesn't provide a function, this module can't either. The POSIX Realtime Extensions (POSIX 1003.1b-1993) are, as the name implies, an extension to POSIX and so a platform can claim "POSIX conformance" and still not support any or all of the IPC functions.

Linux with kernel ≥ 2.6
All features supported.
OpenSolaris ≥ 2008.05
All features supported.
FreeBSD ≥ 7.2
All features supported.

Under 7.2, posix_ipc's demos fail unless they're run as root. It's a simple permissions problem. Prefix the IPC object names with /tmp in params.txt and the problem goes away. I didn't see this behavior under FreeBSD 8.0, so it must have been fixed at some point.

If you don't have the sem and mqueuefs kernel modules loaded, you'll get a message like this (or something similarly discouraging) when you try to create a semaphore or message queue:
Bad system call: 12 (core dumped)

Type kldstat to list loaded modules, and kldload sem or kldload mqueuefs if you need to load either of these. BTW, mqueuefs has some cool features.

Prior to 7.2, FreeBSD POSIX semaphore support was broken.

OS X (up to and including 10.9)
Message queues are not supported by OS X. Also, sem_getvalue() and sem_timedwait() are not supported.

From what I can tell, OS X does not support sem_init() or sem_destroy(), so even if this module adds support for unnamed semaphores, they won't be available under OS X.

Windows + Cygwin 1.7
Cygwin is a Linux-like environment for Windows.

Versions of Cygwin prior to 1.7 didn't support POSIX IPC. Under Cygwin 1.7 beta 62 (released in early October 2009), posix_ipc compiles and runs both demos.