Apache/IBM HTTP Server: No space left on device: Couldn’t create accept lock

As part of restart of IBM HTTP Server (IHS) to effect a configuration I recently came across this error. We couldn’t find any syntax errors, and even reversing the change the error message was still written to the Apache error logs:

[emerg] (28)No space left on device: Couldn't create accept lock

My initial impression was the message is related to a lack of disk space, but there was plenty free on the box. Further research revealed the problem is related to semaphores which are used for communicating between the parent and child processes.

To resolve the problem I performed the following steps:

Run this command as root:

# ipcs -s

If you see a list of semaphores, Apache has not cleaned up after itself, and some semaphores are stuck. Clear them out with this command:

# for i in `ipcs -s | awk '/httpd/ {print $2}'`; do (ipcrm -s $i); done

Now, in almost all cases, Apache should start properly. If it doesn’t, you may just be completely out of available semaphores. You may want to increase your available semaphores. Add this to /etc/sysctl.conf:

kernel.msgmni = 1024
kernel.sem = 250 256000 32 1024

And then run sysctl -p to pick up the new changes.

Eaga/Carillion Warmsure Shambolic Service

So now I have now been without working hot water on my new boiler since 21st December. This is unexpected as our boiler was only fitted in February, but ‘luckily’ it came with 2 years Warmsure cover – or so I thought.

After an engineer visit on the 23rd December, just to diagnose the issue and not do anything further and numerous further phone calls to chase the repair we were told it’s a van stocked part and would be fixed on the 3rd January.

So the engineer turned up, but didn’t phone in advance like he was suppose to and as only he was a contractor so didn’t carry the part and he wasn’t allowed to aquire one from a plumbers merchants. So more waiting till a part is ordered and another engineer is dispached..

Worse part is the engineer manually moved the diverter valve so we could get hot water, however central heating which did work is no longer working..genius..

Screwfix – cheap towel rail radiator and bathroom ventilation kit

Since starting DIY work on my home I have become a huge fan of Screwfix Direct, especially since they are considerably cheaper than most of the DIY “Sheds” here in the UK (Focus, Homebase, Wickes, B & Q).

I have bought a designer towel rail radiator including valves for only £60.00 and a bathroom extractor fan kit which I be attempting to fit this weekend.

I will post my progress on this with some helpful hints and tips later.

HTC Wildfire – Excellent Budget Smartphone

After leaving my girlfriend left out when I’m out shopping/eating and surfing the web on my IPhone, since she can’t beat me I have persuaded her to join me by picked up the superb HTC Wildfire from excellent value mobiles.co.uk.

After doing much research I decided to recommend the HTC Wildfire as it’s received excellent reviews and finds itself on the PC Pro A-List, my usual first port of call when buying any kind of tech.  Another deciding factor was that Android 2.2 (aka Froyo) is now being rolled out to the Wildfire which brings enhancements such as wi-fi hotspot and USB tethering, improved Exchange support and SD card app installation.

It’s going to be a steep learning curve for her coming from a relatively simple Nokia handset, so I will post her experiences on here.


Apple iPhone with SBS Exchange 2003

I recently encountered an issue getting an IPhone 3gs to receive push email from a SBS 2003 Exchange server.

Having read many article, let me say that this IS POSSIBLE. There are a lot of posts out there that say:

  • You can’t do it with Exchange 2003 you need Exchange 2007 or 2010
  • You need to use IMAP
  • You need ISA server in the middle

These are incorrect.

The customer has a fairly standard Single SBS 2003 box with exchange all on the one server, running SP2 and Outlook over RPC/Outlook anywhere fully working – a typical setup for a small company.

The key problem I discovered was that OMA (Outlook Mobile Access) is required for by the iPhone to synchronize with Exchange. To enable this:

  • Open the Default Website in IIS 6
  • Disable the IP address restrictions on the exchange-oma directory
  • Restart IIS (iisreset)

Once done the iPhone email synchronization started working fine.

Direct Debit Errors and Santander’s customer service

So today as usual I had my monthly telephone line rental taken out by direct debit. Unusually I’m not with BT or Virgin Media (for those in cables areas) but I use The Phone-Coop, the original reason being they offered line rental contracts for a minimum contract of 6 months against the usual 12 months from BT.

I don’t actually use my land-line, but sadly it’s a pre-requesite for ADSL in the UK and naked DSL isn’t available due to lack of demand, says BT.

So today I notice two sums being deducted, one for myself and another nearly double. A closer inspection of my statement shows this has been going on for the past 4 months* resulting in a total overpayment of ~£130.00.
*Yes, I know I should have checked my statements more closely before…

The Phone Co-Op were my first port of call, I reported the issue to them and they said they would investigate and get back to me. Still annoyed and quite impatient I did some research into Direct Debits and found that under the terms of the direct debit guarantee I was entitled to a full and immediate refund, so decided to call Santander…

Initially the call centre advisor was quite helpful, I ran off the four direct debits and she advised they have been reclaimed – unfortunately when she read them back to me it was clear two of the four were incorrect. When I asked if the two incorrect claims could be cancelled she advised they couldn’t and I would have to repay the company concerned manually.

Great, I thought, Santander have made an error on my account, I am going to have to mess about with a manual payment and explain to another company why their direct debits have been reversed. Knowing how call centres are structured I asked to be escalated to a manager who thankfully was based in Britain and not East Asia; he apologised and made a £15.00 credit to my an account for the inconvenience.

They still weren’t able to reverse the incorrect direct debit reclaims, and apparently they can only go back two months on their system whereas I needed to go back four; His answer was that I needed to “contact the originating company for a refund.”, which as shown here is not true. This shouldn’t be a suprise to me at all given that Santander has been voted the worst bank for customer service , they also recently managed to send out 35,000 bank staments to the wrong addresses.

Moral of the story, don’t settle for poor customer service.

Installing a Godaddy SSL Certificate into Apache Ubuntu Linux

Godaddy is probably the cheapest SSL certificate provide on the web with their root CA present in all the major browsers. I was able to pick up a 12 month SSL certificate for only $12.99 (~£10.00 inc VAT) by following the link HERE

Now on to the good stuff.

  • Webserver: Ubuntu 8.04 LTS
  • Apache2 with http.conf: /etc/apache2/http.conf

First we need to generate a ‘key’ file that tells our server apart from other servers. You should have openssl installed on your machine, if not, simply type in

sudo apt-get install openssl

Then change to the directory to the place you would like to store everything. In my case, I chose:

mkdir /etc/apache2/ssl
kdir /etc/apache2/ssl/certs
mkdir /etc/apache2/ssl/private
cd /etc/apache2/ssl/private
openssl genrsa -out websitename.key 2048

This will spit out a key for our server to create the CSR which is what we need to send to send to Godaddy in order to get the required files to finish the setup. Next we generate a CSR (so in the same directory):

openssl req -new -key websitename.key -out websitename.csr

When filling out the CSR via the terminal it will ask you for several things(Country Name, State or Province, Locality Name, Organization Name, Organizational Unit Name, Common Name, Email Address, and Password) Some of these are optional, but make sure that under Common Name you put your website url (website.com).

Now, we need the contents of this CSR to give to godaddy.

cat websitename.csr

Copy the text and paste it into Godaddy’s CSR pane. Make sure you keep the —Begin— and —End— stuff or else godaddy will reject it. I also chose the Starfield Technologies certificate just because it sounded cooler than Godaddy. Once accepted, you would then download the certificate.

Click on the common name [yourwebsite] then on download. Select Apache server type. This will give you a zip file with two files in it. You need to copy the website.crt to /etc/apache2/ssl/certs and the sf_bundle.crt to /etc/apache2/ssl I used a program called WinSCP to drag and drop these files in there via ssh.


Folder Layout >
        |__ website.crt
        |__ website.key
        |__ website.csr (Can be deleted)

We need to edit our http.conf file to make sure that we have a valid path for the SSL. Here is my Vhost listing for the ssl. As you can see I have added a few extra things to make it play nice. I turned on SSLEngine and included the files for it to work properly. Feel free to copy and paste with minor changes.

<VirtualHost *:443>
  ServerName example.com
  ServerAlias example.com
  SSLEngine on
  SSLOptions +FakeBasicAuth +ExportCertData +StrictRequire
  SSLCertificateFile /etc/apache2/ssl/certs/website.crt
  SSLCertificateKeyFile /etc/apache2/ssl/private/website.key
  SSLCertificateChainFile /etc/apache2/ssl/sf_bundle.crt
  DocumentRoot /var/www/www.website.com/htdocs

Now when you restart apache:

/etc/init.d/apache2 restart

Apache Secured Shared Hosting with mod_proxy and mod_chroot

I have been investigating how I can isolate the various sites I host on my VPS box. Ideally I would have my own dedicated server with each site in it’s own VPS, this may be an option if I roll my own dedicated server in the near future but for now the next best option appears to be a chroot jail for each site proxied behind the primary Apache instance.

First step ensure Apache2 and mod_chroot are installed on your box, for Ubuntu this is:

apt-get install apache2 libapache2-mod-chroot

Then for each domain, create a virtual host config like /etc/apache2/sites-available/example.com:

<VirtualHost *:80>
  ServerAdmin test@example.com
  ServerName example.com
  ServerAlias *.example.com

  ProxyRequests Off
  ProxyPreserveHost On
  ProxyPass / http://localhost:8000/
  ProxyPassReverse / http://localhost:8000/
  <Proxy *>
    Allow from all

  ErrorLog /home/example/www/example.com/log/error.log
  LogLevel warn
  CustomLog /home/example/www/example.com/log/access.log combined

This will be used by the Apache proxy, which is the normal system apache2 running as the default “www-data” user. This proxy handles name-based virtual hosts, and proxies the requests to a second process, running at localhost on port 8080.

Note that the logs are configured here and not in the user’s Apache process, for two reasons:

  1. Keep logs pristine in the event of a break-in on a user site (for example via a buggy or malicious PHP script)
  2. Single system-wide log file analysis process instead of per-user

Next, create a user account for each domain:

mkdir -p /home/example/www/example.com
cd /home/example/www/example.com
mkdir htdocs logs conf
chown www-data:www-data logs
chown example:example htdocs/

Create the following in /home/example/www/example.com/conf/apache2.conf:

ServerRoot "/home/example/www/example.com/"
LockFile /home/example/www/example.com/conf/accept.lock
PidFile /home/example/www/example.com/conf/apache2.pid
Timeout 300
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 15
User example
Group example
AccessFileName .htaccess
<Files ~ "^\.ht">
  Order allow,deny
  Deny from all

Options -Indexes

DefaultType text/plain
HostnameLookups Off
ErrorLog /home/example/www/example.com/conf/error.log
LogLevel warn
LoadFile /lib/libnss_dns.so.2
LoadModule chroot_module /usr/lib/apache2/modules/mod_chroot.so
Include /etc/apache2/mods-enabled/*.load
Include /etc/apache2/mods-enabled/*.conf
Listen 8000
DocumentRoot /home/example/www/example.com/htdocs
ChrootDir /home/example/www/example.com

Once the chroot has been applied the root directory as far as the Apache process is concerned is /home/example/www/example.com – this creates a problem whereby the DocumentRoot is actually now /htdocs. The way round this is to use the –bind option of mount shown here:

mkdir -p /home/example/www/example.com/home/example/www/example.com
mount --bind /home/example/www/example.com /home/example/www/example.com/home/example/www/example.com

Now launch the new Apache process:

apache2 -f /home/example/www/example.com/conf/apache2.conf -k start

Everything should now work, but you may notice some problems, particularly with scripts like PHP, CGI etc. Generally the way round this is to make parts of the system available in the chroot and by ammending paths using the apache2.conf file in the jail.

This is as far as I have got, I have been working on getting startup right with an init script and automating the mount –bind by editing the /etc/fstab entries. I will cover my success (or otherwise) in another post.