This describes the sysv_ipc
module which gives Python access
to System V inter-process semaphores, shared memory and message queues
on most (all?) *nix flavors. Examples include macOS/OS X, Linux, FreeBSD,
OpenSolaris 2008.11, HPUX, and AIX.
It might also work under Windows with a library like
Cygwin.
It works with Python ≥ 3.6. It's released under a BSD license.
You can download sysv_ipc version 1.1.0 ([md5 sum], [sha1 sum], [sha256 sum]) which contains the source code, setup.py, installation instructions and sample code. You can read about all of the changes in this version.
You might also want to read about some known bugs.
You might be interested in the very similar module
posix_ipc
which provides Python access to POSIX IPC primitives. POSIX IPC is a little
easier to use than SysV IPC, but not all operating systems support it
completely.
sysv_ipc
Jump to semaphores, shared memory, or message queues.
id
and
returns a new SharedMemory object. See
SharedMemory.attach() for details on the
address
and flags
parameters.
This method is useful only under fairly unusual circumstances. You probably don't need it.
ftok(path, id)
. Note that
ftok()
has limitations, and this
function will issue a warning to that effect unless
silence_warning
is True.
id
.id
.id
.IPC_CREAT
and IPC_EXCL
are flags used when
creating IPC objects. They're
bitwise unique and can be ORed together. IPC_CREX
is
shorthand for IPC_CREAT | IPC_EXCL
.
When passed to an IPC object's constructor, IPC_CREAT
indicates
that you want to create a new object or open an existing one. If you want
the call to fail if an object with that key already exists, specify
the IPC_EXCL
flag, too.
fork()
).
key_t
.
Keys randomly generated by this module are in the range
1 ≤ key ≤ SHRT_MAX
.
That's type-safe unless your OS has a very bizarre
definition of key_t
.
SharedMemory.attach()
to attach the segment
read-only.shmat()
for more information.
.attach()
function in the case of
SHM_REMAP. See your system's man page for shmget()
and shmat()
for more information.
In addition to standard Python errors (e.g. ValueError
),
this module raises custom errors. These errors cover
situations specific to IPC.
.P()
or .Z()
either times out
or would be forced to wait when its block
attribute is False.
This is a handle to a semaphore.
key must be None
,
IPC_PRIVATE
or
an integer > KEY_MIN
and ≤ KEY_MAX
. If the key
is None
, the module chooses a random unused key.
The flags specify whether you want to create a new semaphore or open an existing one.
0
, the module attempts
to open an existing semaphore identified by key and raises
a ExistentialError
if that semaphore doesn't exist.
IPC_CREAT
, the module
opens the semaphore identified by
key or creates a new
one if no such semaphore exists. Using IPC_CREAT
by itself
is not recommended. (See Semaphore Initialization.)
IPC_CREX
(IPC_CREAT | IPC_EXCL
),
the module
creates a new semaphore identified by key. If a
semaphore with that key already exists, the call raises an
ExistentialError
.
The initial_value is ignored unless
both of these flags are specified or
if the semaphore is read-only.
When opening an existing semaphore, mode is ignored.
The timeout (which can be a float) specifies how many seconds this call should wait, if at all.
The semantics of the timeout changed a little in version 0.3.
BusyError
if it can't immediately
acquire the semaphore. Since it will
return immediately if not asked to wait, this can be
thought of as "non-blocking" mode.
BusyError
.
When the call returns, the semaphore's value decreases by
delta
(or more precisely, abs(delta)
)
which defaults to 1.
On platforms that don't support the semtimedop()
API call,
all timeouts (including zero) are treated as infinite. The call
will not return until its wait condition is satisfied.
Most platforms provide semtimedop()
. macOS is a
notable exception. The module's Boolean constant
SEMAPHORE_TIMEOUT_SUPPORTED
is True on platforms that support semtimedop()
.
The semaphore's value increases by delta
(or more precisely, abs(delta)
)
which defaults to 1.
.acquire()
that takes the same parameters.
"P" stands for prolaag or probeer te verlagen (try to decrease), the original name given by Edsger Dijkstra.
.release()
that takes the same parameters.
"V" stands for verhoog (increase), the original name given by Edsger Dijkstra.
Timeout has
the same meaning as described in .acquire()
.
As far as I can tell, the effect of deleting a semaphore that
other processes are still using is OS-dependent. Check your system's
man pages for semctl(IPC_RMID)
.
When True, operations that change the semaphore's value will be undone (reversed) when the process exits. Note that when a process exits, an undo operation may imply that a semaphore's value should become negative or exceed its maximum. Behavior in this case is system-dependent, which means that using this flag can make your code non-portable.
acquire()
and
release()
will not return
until their wait conditions are satisfied.
When False, these calls will not block but will instead raise an error if they are unable to return immediately.
Tip: the following Python code will display
the mode in octal:
print int(str(my_sem.mode), 8)
semop()
(.P()
,
.V()
or .Z()
) on this semaphore.
Linux and macOS also set this when the semaphore's value is changed, although doing so disagrees with the POSIX specification. See Linux kernel bug 112271.
.P()
).
.Z()
).
semop()
(i.e. .P()
, .V()
or
.Z()
) was called on this semaphore.
These semaphores provide __enter__()
and __exit__()
methods so they can be used in context managers. For instance --
with sysv_ipc.Semaphore(name) as sem: # Do something...
Entering the context acquires the semaphore, exiting the context releases
the semaphore. See demo4/child.py
for a complete example.
This is a handle to a shared memory segment.
In addition to the
methods and attributes described below, SharedMemory
supports
Python's buffer protocol, which means you can create bytearray
and memoryview
objects based on a SharedMemory
segment.
See demos/buffer_protocol
for an example.
key must be None
,
IPC_PRIVATE
or
an integer > 0
and ≤ KEY_MAX
. If the key
is None
, the module chooses a random unused key.
The flags specify whether you want to create a new shared memory segment or open an existing one.
0
, the module attempts
to open an existing shared memory segment identified by
key and raises
a ExistentialError
if it doesn't exist.
IPC_CREAT
, the module
opens the shared memory segment identified
by key or
creates a new one if no such segment exists.
Using IPC_CREAT
by itself
is not recommended. (See Memory Initialization.)
IPC_CREX
(IPC_CREAT | IPC_EXCL
),
the module
creates a new shared memory segment identified by
key. If
a segment with that key already exists, the call raises
a ExistentialError
.
When both IPC_CREX
is specified
and the caller has write permission, each byte in the new memory segment will be
initialized to the value of init_character.
The value of size depends on whether one is opening an existing segment or creating a new one.
0
.
In addition, some round the size
up to the next multiple of PAGE_SIZE.
This module supplies a default
size of PAGE_SIZE
when
IPC_CREX
is specified and 0
otherwise.
.read()
or .write()
. Note that the
constructor automatically attaches the memory
so you won't need to call this method unless you explicitly detach it
and then want to use it again.
The address parameter allows one to specify (as a Python long) a memory
address at which to attach the segment. Passing None (the default)
is equivalent to passing NULL to shmat()
. See that
function's man page for details.
The flags are mostly only relevant if one specifies a specific address.
One exception is the flag SHM_RDONLY
which, surprisingly,
attaches the segment read-only.
Note that on some (and perhaps all) platforms, each call to .attach()
increments the system's "attached" count. Thus, if each call to
.attach()
isn't paired with a call to .detach()
,
the system's "attached" count for the shared memory segment will not
go to zero when the process exits. As a result, the shared memory
segment may not disappear even when its creator calls .remove()
and exits.
str
under Python 2).
If byte_count is zero (the default) the entire buffer is returned.
This method will never attempt to read past the end of the shared memory segment, even when offset + byte_count exceeds the memory segment's size. In that case, the bytes from offset to the end of the segment are returned.
str
in Python 2) to the shared memory,
starting at offset. Passing a Unicode object
may work, but doing so is unsupported and may be explicitly deprecated
in a future version.
If the offset + data would write outside of the segment,
this function raises ValueError
.
The bytes may contain embedded NULL bytes ('\0'
).
Tip: the following Python code will display
the mode in octal:
print int(str(my_mem.mode), 8)
This is a handle to a FIFO message queue.
key must be None
,
IPC_PRIVATE
or
an integer > 0
and ≤ KEY_MAX
. If the key
is None
, the module chooses a random unused key.
The flags specify whether you want to create a new queue or open an existing one.
0
, the module attempts
to open an existing message queue identified by
key and raises
a ExistentialError
if it doesn't exist.
IPC_CREAT
, the module
opens the message queue identified
by key or
creates a new one if no such queue exists.
IPC_CREX
(IPC_CREAT | IPC_EXCL
),
the module
creates a new message queue identified by
key. If
a queue with that key already exists, the call raises
a ExistentialError
.
The max_message_size can be increased from the default, but be aware of the issues discussed in Message Queue Limits.
The message should be a bytes object
(a.k.a. str
in Python 2) and can contain embedded
NULLs (ASCII 0x00
). Passing a Unicode object
may work, but doing so is unsupported and may be explicitly deprecated
in a future version.
The block flag specifies whether or
not the call should wait if the message can't be sent (if, for
example, the queue is full). When block
is False
, the call will raise a BusyError
if
the message can't be sent immediately.
The type is
associated with the message and is relevant when calling
receive()
. It must be > 0.
(message, type)
. The message is a bytes object
(a.k.a. str
in Python 2).
The block flag specifies whether or
not the call should wait if there's no messages of the
specified type to retrieve. When block
is False
, the call will raise a BusyError
if
a message can't be received immediately.
The type permits some control over which messages are retrieved.
== 0
, the call
returns the first message on the queue regardless of its
type.
> 0
, the call
returns the first message of that type.
< 0
, the call
returns the first message of the lowest type that is ≤ the
absolute value of type.
Tip: the following Python code will display
the mode in octal:
print int(str(my_mem.mode), 8)
This module comes with four sets of demonstration code in the
directory demos
.
sem_and_shm
demonstrates using semaphores and
shared memory to share data between processes.
message_queues
demonstrates using message queues
to share data between processes.
buffer_protocol
demonstrates creating bytearray
and
memoryview
objects atop shared memory.
semaphore_context_manager
demonstrates using
semaphore instances in a context manager (with
statement).
ftok()
Most System V IPC sample code recommends ftok()
for generating an
integer key that's more-or-less random.
It does not, however, guarantee that the key it generates is unused. If
ftok()
gives your application a key that some other application is
already using,
your app is in trouble unless it has a reliable second mechanism for generating
a key. And if that's the case, why not just abandon ftok()
and use the
second mechanism exclusively?
This is the weakness of ftok()
-- it isn't guaranteed to give you
what you want. The BSD
man page for ftok
says it is "quite possible for the routine to
return duplicate keys". The term "quite possible" isn't quantified, but suppose
it means one-tenth of one percent. Who wants to have 1-in-1000 odds of a
catastrophic failure in their program, or even 1-in-10000?
This module obviates the need for ftok()
by generating random
keys for you. If your application can't use sysv_ipc
's automatically
generated keys because it needs to know the key in advance, hardcoding a
random number like 123456 in your app might be no worse than using
ftok()
and has the advantage of not hiding its limitations.
This module provides ftok()
in case you want to experiment with it.
However, to emphasize its weakness, this version of ftok()
raises a
warning with every call unless you explicitly pass a flag to silence it.
This package also provides ftok_experiment.py
so that you can observe
how often ftok()
generates duplicate keys on your system.
When a System V sempahore is created at the C API level, the OS is not required
to initialize the semaphore's value. (This per
the
SUSv3 standard for semget()
.)
Some (most? all?) operating systems initialize it to zero, but this behavior
is non-standard and therefore can't be relied upon.
If sempahore creation happens in an predictable, orderly fashion, this isn't a
problem. But a
race condition arises when multiple processes vie to create/open the same semaphore. The
problem lies in the fact that when an application calls semget()
with only
the IPC_CREAT
flag, the caller can't tell whether or not he has
created a new semaphore or opened an existing one.
This makes it
difficult to create reliable code without using IPC_EXCL
.
W. Richard Stevens' Unix Network Programming Volume 2
calls this "a fatal flaw in the design of System V semaphores" (p 284).
For instance, imagine processes P1 and P2. They're executing the same code, and that code intends to share a binary semaphore. Consider the following sequence of events at the startup of P1 and P2 –
semget(IPC_CREAT)
to create the semaphore S.semget(IPC_CREAT)
to open S.acquire()
, decrementing the value to 0.acquire()
, decrementing the value to 0. Both processes
now think they own the lock.W. Richard Stevens' solution for this race condition is to check the value of
sem_otime
(an element in the semid_ds
struct that's
populated on the call to semctl(IPC_STAT)
and which is exposed to
Python by this module) which
is initialized to zero when the semaphore is created and otherwise holds
the time of the last
call to semop()
(which is called by P()
/acquire()
,
V()
/release()
, and Z()
).
In Python, each process would run something like this:
try: sem = sysv_ipc.Semaphore(42, sysv_ipc.IPC_CREX) except sysv_ipc.ExistentialError: # One of my peers created the semaphore already sem = sysv_ipc.Semaphore(42) # Waiting for that peer to do the first acquire or release while not sem.o_time: time.sleep(.1) else: # Initializing sem.o_time to nonzero value sem.release() # Now the semaphore is safe to use.
With shared memory,
using the IPC_CREAT
flag without IPC_EXCL
is problematic unless you know the size of the segment
you're potentially opening.
Why? Because when creating a new segment,
many (most? all?) operating systems demand a non-zero size. However,
when opening an existing segment, zero is the only guaranteed safe value
(again, assuming one doesn't know the size of the segment in advance).
Since IPC_CREAT
can open or create a segment, there's no safe value for the size under
this circumstance.
As a (sort of) side note, the
SUSv3
specification for shmget()
says only that the size of a new
segment must not be less than "the system-imposed minimum". I
gather that at one time, some systems set the minimum at zero despite the
fact that it doesn't make much sense to create a zero-length shared memory
segment. I think most modern systems do the sensible thing and insist on
a minimum length of 1.
Python programmers can usually remain blissfully ignorant of memory allocation issues. Unfortunately, a combination of factors makes them relevant when dealing with System V message queues.
Some implementations impose extremely stingy limits. For instance, many BSDish systems (macOS, FreeBSD, NetBSD, and OpenBSD) limit queues to 2048 bytes. Note that that's the total queue size, not the message size. Two 1k messages would fill the queue.
Those limits can be very difficult to change. At best, only privileged processes can increase the limit. At worst, the limit is a kernel parameter and requires a kernel change via a tunable or a recompile.
This module can't figure out what the limits are, so
it can't cushion them or even report them to you.
On some systems the limits are expressed in header files, on others
they're available through kernel interfaces (like FreeBSD's sysctl
).
Under macOS and to some extent OpenSolaris I can't figure out where they're
defined and what I report here is the result of experimentation and educated
guesses formed by Googling.
The good news is that this module will still behave as advertised no
matter what these limits are. Nevertheless you might be surprised when a
call to .send()
get stuck because a queue is full even though you've
only put 2048 bytes of messages in it.
Here are the limits I've been able to find under my test operating systems, ordered from best (most generous) to worst (most stingy). This information was current as of 2009 when I wrote the message queue code. It's getting pretty stale now. I hope the situation has improved over the 2009 numbers I describe below.
Under OpenSolaris 2008.05 each queue's maximum size defaults
to 64k. A privileged process (e.g. root) can change this through the
max_size
attribute of a sysv_ipc.MessageQueue
object.
I was able to increase it to 16M and successfully sent sixteen 1M messages to
the queue.
Under Ubuntu 8.04 (and perhaps other Linuxes) each queue's maximum size defaults to 16k. As with OpenSolaris, I was able to increase this to 16M, but only for a privileged process.
Under FreeBSD 7 and I think NetBSD and OpenBSD, each
queue's maximum size defaults to 2048 bytes. Furthermore, one can (as root)
set max_size
to something larger and FreeBSD doesn't complain, but
it also ignores the change.
macOS is the worst of the lot. Each queue is limited to 2048 bytes and macOS silently ignores attempts to increase this (just like FreeBSD). To add insult to injury, there appears to be no way to increase this limit short of recompiling the kernel. I'm guessing at this based on the Darwin message queue limits.
If you want
to search for these limits on your operating system, the key constants are
MSGSEG
, MSGSSZ
, MSGTQL
, MSGMNB
,
MSGMNI
and MSGMAX
. Under BSD, sysctl kern.ipc
should tell you what you need to know and may allow you to change these
parameters.
Semaphores and especially shared memory are a little different from most Python objects and therefore require a little more care on the part of the programmer. When a program creates a semaphore or shared memory object, it creates something that resides outside of its own process, just like a file on a hard drive. It won't go away when your process ends unless you explicitly remove it.
In short, remember to clean up after yourself.
man
PagesThe sysv_ipc module is just a wrapper around your system's API. If your
system's implementation has quirks, the man
pages for semget, semctl, semop
shmget, shmat, shmdt
and shmctl
will probably cover them.
Many systems (although not some older versions of OS X) come
with ipcs
and ipcrm
.
The former shows existing shared memory, semaphores and message queues on your system and
the latter allows you to remove them.
For Pythonistas –
Bugs? My code never has bugs! There are, however, some suboptimal anomalies...
These are features that may or may not be added depending on technical difficulty, user interest and so forth.
SEMAPHORE_VALUE_MAX
more accurate.I don't plan on adding support for semaphore sets.