We recently deployed two new Linux web servers, a couple of Sun v20z’s. Ubuntu installed cleanly, we put them into production, and everything was going smoothly. Then Apache processes started hanging, and we ended up rebooting a couple of times to get the servers functioning again.

Turns out it was /dev/random blocking the processes. When /dev/random runs out of randomness, it just sits there until it gets more. I suspected this earlier in the week, and removed some calls to rand() from some frequently called PHP scripts. I also suspected jpcache, which makes a lot of mt_rand() calls. I moved the cache files off of nfs and made them local to each web server, in case NFS was having trouble with flock(). Since then we only saw a few short slowdowns, lasting maybe 5 minutes. The whole time we did see that the available randomness (/proc/sys/kernel/random/entropy_avail) was pretty low.

I didn’t know this, but apparently the 2.4 kernel used to use the hard drive, keyboard, mouse, and network activity to fill the entropy pool. With 2.6 they stopped using the network, because it was theoretically possible to exploit it. With these web servers we don’t have a mouse or keyboard, and the hard drive doesn’t do much because the web files are all on NFS, so we don’t get a lot of entropy. When I moved jpcache to the local disk, it gave it just enough entropy to stop crashing hard.

I did some research and found that the servers had their own hardware randomness generator built in. Ubuntu even came with the appropriate kernel module and loaded it. But there’s a daemon that needs to run to collect entropy from the hardware and fill /dev/random, and that daemon was not installed. So I installed rng-tools, and now we’ve got entropy for days.

Here’s a nice web page that explains it:
http://www.linuxcertified.com/hw_random.html