July 18th, 2007

  • k001

one kernel bug story among 305

A few days ago one of OpenVZ kernel team members, Pavel Emelyanov, posted a one-line patch to fix a bug in Linux kernel. He received the following reply from Andrew Morton, one of the upstream kernel maintainers:

I'm curious. For the past few months, people@openvz.org have discovered (and fixed) an ongoing stream of obscure but serious and quite long-standing bugs.

How are you discovering these bugs?

Andrew added later:

hm, OK, I was visualising some mysterious Russian bugfinding machine or something.

Don't stop ;)

So, here is the story behind that bug.

A few months ago, in the course of OpenVZ kernel testing, our QA (Quality Assurance) team found a strange issue. The thing is, every container (VE) in OpenVZ has a set of resource usage counters (and limits) called beancounters. All the usage counters should be zero when a VE is stopped, since naturally then all the resources are released. The issue was that a resource called kmemsize (a kernel memory used on behalf of given VE) had a usage counter of 78 bytes after the VE was stopped -- which effectively means 78 bytes of kernel memory were lost (or leaked, as programmers say).

Who cares about 78 bytes, especially on a server with 16 gigabytes (17,179,869,184 bytes) of RAM? We do. Pavel checked the beancounters debug information which showed that one struct user object has leaked. He then tried to reproduce that but with no luck.

Bugs that can not be reproduced are tough. The only option left was to audit the kernel source code. That involved finding all the places where struct user object is referenced, and checking the code correctness (the term "correctness" in this context means that every object that is allocated must later be released). It took him 4 hours to do the audit, and he found one place where the reference to an object might be lost (which means it could not later be released). It's the same as if you lend a book to your friend and later forgot whom you gave it to -- you lost the reference and you can't get the book back.

In this case, after the problem was found, fixing it was pretty straightforward. So Pavel wrote a fix and a demo code to trigger the bug, tested the fix and sent it to Linux kernel mailing list.

Why is this particular incident so important?
* It's OpenVZ code (beancounters) which helped to detect the leak in the first place -- as the bug is very hard to trigger (unless you know how) and the leak is small enough that it might not be discovered at all.
* It demonstrates OpenVZ developers dedicated attitude. They never dismiss real bugs as "works for me" or "invalid", and work to find the root cause and fix the problem.
* This bug is in fact a security issue. An ordinary user (actually two users are needed in this case) could exploit the bug and eat all the kernel memory, thus bringing the whole system down. Worse scenarious are possible as well.
* Incidentally, OpenVZ is protected from this security issue -- because kmemsize beancounter (which helped to found it) limits kernel memory usage per Virtual Environment.

Most important of all, this is just one out of 305 kernel patches by our team which were accepted into the mainstream Linux kernel during a one-year period. Almost one patch a day, excluding weekends and holidays. And we are not going to stop! :-)