Apache/IBM HTTP Server: No space left on device: Couldn’t create accept lock

As part of restart of IBM HTTP Server (IHS) to effect a configuration I recently came across this error. We couldn’t find any syntax errors, and even reversing the change the error message was still written to the Apache error logs:

[emerg] (28)No space left on device: Couldn't create accept lock

My initial impression was the message is related to a lack of disk space, but there was plenty free on the box. Further research revealed the problem is related to semaphores which are used for communicating between the parent and child processes.

To resolve the problem I performed the following steps:

Run this command as root:

# ipcs -s

If you see a list of semaphores, Apache has not cleaned up after itself, and some semaphores are stuck. Clear them out with this command:

# for i in `ipcs -s | awk '/httpd/ {print $2}'`; do (ipcrm -s $i); done

Now, in almost all cases, Apache should start properly. If it doesn’t, you may just be completely out of available semaphores. You may want to increase your available semaphores. Add this to /etc/sysctl.conf:

kernel.msgmni = 1024
kernel.sem = 250 256000 32 1024

And then run sysctl -p to pick up the new changes.