OpenVZ

SOPA protest

Wed, 18 Jan 2012 15:17:01 GMT

wiki.openvz.org is down today, joining the SOPA and PIPA protest. Read more:
* http://americancensorship.org/
* https://www.eff.org/deeplinks/2012/01/how-pipa-and-sopa-violate-white-house-principles-supporting-free-speech
* http://google.com/takeaction

The site will come alive tomorrow. Now, if you need access to the site, just reload the page.

Recent improvements in vzctl

Tue, 27 Dec 2011 16:03:39 GMT

VSwap is an excellent feature, simplifying container resource management a lot. Now it's time to also simplify the command line interface, i.e. vzctl. Here is what we did recently (take a look at vzctl git repo if you want to review the actual changes):

1. We no longer require to set kmemsize, dcachesize and lockedpages parameters to non-unlimited values (this is one of the enhancements in the recent kernels we have talked about recently). Therefore, setting these parameters to fractions of CT RAM (physpages) are now removed from configuration files and vzsplit output.

2. There is no longer a need to specify all the UBC parameters in per-container configuration file. If you leave some parameters unset, the kernel will use default values (usually unlimited). So, VSwap example configs are now much smaller, with as much as 19 parameters removed from those.

3. vzctl set now supports two new options: --ram and --swap. These are just convenient short aliases for --physpages and --swappages, the differences being that you only need to specify one value (the limit) and the argument is in bytes rather than pages.

So, to configure a container named MyCT to have 1.5GB of RAM and 3GB of swap space available, all you need to do is just run this command:
vzctl set MyCT --ram 1.5G --swap 3G --save

4. This is not VSwap-related, but nevertheless worth sharing. Let's illustrate it by a copy-paste from a terminal:

# vzctl create 123 --ostemplate centos-4-x86_64
Creating container private area (centos-4-x86_64)
Found centos-4-x86_64.tar.gz at http://download.openvz.org/template/precreated//centos-4-x86_64.tar.gz
Downloading...
-- 2011-11-29 18:54:08 -- http://download.openvz.org/template/precreated//centos-4-x86_64.tar.gz
Resolving download.openvz.org... 64.131.90.11
Connecting to download.openvz.org|64.131.90.11|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 171979832 (164M) [application/x-gzip]
Saving to: `/vz/template/cache/centos-4-x86_64.tar.gz'

100%[======================================>] 171,979,832 588K/s in 4m 27s 

2011-11-29 18:58:35 (630 KB/s) - `/vz/template/cache/centos-4-x86_64.tar.gz' saved [171979832/171979832]

Success!
Performing postcreate actions
Saved parameters for CT 123
Container private area was created

All this will be available in vzctl-3.0.30, which is to be released soon (next week? who knows). If you can't wait and want to test this stuff, here's a nightly build of vzctl available (version 3.0.29.3-27.git.0535fe1) from http://wiki.openvz.org/Download/vzctl/nightly. Please give it a try and tell me what you think.

On vSwap and 042stab04x kernel improvements

Mon, 14 Nov 2011 20:03:12 GMT

vSwap

The best feature of the new (RHEL6-based) 042 series of the OpenVZ kernels is definitely vSwap. The short story is, we used to have 22 user beancounter parameters which every seasoned OpenVZ user knows by heart. Each of these parameters is there for a reason, but 22 knobs are a bit too complex to manage for a mere mortal, especially bearing in mind that

many of them are interdependent;
the sum of all limits should not exceed the resources of a given physical server.

Keeping this configuration optimal (or even consistent) is quite a challenging task even for a senior OpenVZ admin (with a probable exception of an ex airline pilot). This complexity is the main reason why there are multiple articles and blog entries complaining OpenVZ is worse than Xen, or that it is not suitable for hosting Java apps. We do have some workarounds to mitigate this complexity, such as:

precreated container configs with sane defaults for beancounters;
some special tools (vzsplit and vzcfgvalidate);
the comprehensive User Beancounters manual.

This is still not the way to go. While we think high of our users, we do not expect all of them to be ex airline pilots. To solve the complexity, the number of per-container knobs and handles should be reduced to some decent number, or at least most of these knobs should be optional.

We worked on that for a few years, and the end result is called vSwap (where V is for Vendetta, oh, pardon me, Virtual).

vSwap concept is as simple as a rectangular. For each container, there are only two required parameters: the memory size (known as physpages) and the swap size (swappages). Almost everyone (not only an admin, but even an advanced end user) knows what is RAM and what is swap. On a physical server, if there is not enough memory, the system starts to swap out memory pages to disk, then swap in some other pages, which results in severe performance degradation but it keeps the system from failing miserably.

It's about the same with vSwap, except that

RAM and swap are configured on a per container basis;
no I/O is performed until it is really necessary (this is why swap is virtual).

Some VSwap internals

Now, there are only two knobs per container on a dashboard, namely RAM and swap, and all the complexity is hidden under the hood. I am going to describe just a bit of that undercover mechanics and explain what does the "Reworked VSwap kernel memory accounting" line from the 042stab040.1 kernel changelog stands for.

The biggest problem is, RAM for containers is not just RAM. First of all, there is a need to distinguish between

the user memory,
the kernel memory,
the page cache,
and the directory entry cache.

The user memory is more or less clear, it is simply the memory that programs allocate for themselves to run. It is relatively easy to account for, and it is relatively simple to limit it (but read on).

The kernel memory is really complex thingie. Right, it is the memory that kernel allocates for itself in order for programs in a particular container to run. This includes a lot of stuff I'd rather not dive into, if I want to keep this piece as an article not a tome. Having said that, two particular kernel memory types are worth explaining.

First is the page cache, the kernel mechanism that caches disk contents in memory (that would be unused otherwise) to minimize the I/O. When a program reads some data from a disk, that data are read into the page cache first, and when a program writes to a disk, data goes to the page cache (and then eventually are written (flushed) to disk). In case of repeated disk access (which happens quite often) data is taken from a page cache, not from the real disk, which greatly improves the overall system performance, since a disk is much slower than RAM. Now, some of the page cache is used on behalf of a container, and this amount must be charged into "RAM used by this container" (i.e. physpages).

Second is the directory entry cache (dcache for short) is yet another sort of cache, and another sort of the kernel memory. Disk contents is a tree of files and directories, and such a tree is quite tall and wide. In order to read the contents of, say, /bin/sh file, kernel have to read the root (/) directory, find 'bin' entry in it, read /bin directory, find 'sh' entry in it and finally read it. Although these operations are not very complex, there is a multitude of those, they take time and are repeated often for most of the "popular" files. In order to improve performance, kernel keeps directory entries in memory — this is what dcache is for. The memory used by dcache should also be accounted and limited, since otherwise it's easily exploitable (not only by root, but also by an ordinary user, since any user is free to change into directories and read files).

Now, the physical memory of a container is the sum of its user memory, the kernel memory, the page cache and the dcache. Technically, dcache is accounted into the kernel memory, then kernel memory is accounted into the physical memory, but it's not overly important.

Improvements in the new 042stab04x kernels

Better reclamation and memory balancing

What to do if a container hit a physical memory limit? Free some pages by writing their contents to the abovementioned virtual swap. Well, not quite yet. Remember that there is also a page cache and a dcache, so the kernel can easily discard some of the pages from these caches, which is way cheaper than swapping out.

The process of finding some free memory is known as reclamation. Kernel needs to decide very carefully when to start reclamation, how many and what exact pages to reclaim in every particular situation, and when it is the right time to swap out rather than discard some of the cache contents.

Remember, we have four types of memory (kernel, user, dcache and page cache) and only one knob which limits the sum of all these. It would be easier for the kernel, but not for the user, to have separate limits for each type of memory. But, for the user convenience and simplicity, the kernel only have one knob for these four parameters, so it needs to balance between those four. One major improvement in 042stab040 kernel is that such balancing is now performed better.

Stricter memory limit

During the lifetime of a container, the kernel might face a situation when it needs more kernel memory, or user memory, or perhaps more dcache entries, and the memory for the container is tight (i.e. close to the limit), so it needs to either reclaim or swap. The problem is there are some situations when neither reclamation nor swapping is possible, so the kernel can either fail miserably (say by killing a process) or go beyond the limit and hope that everything will be fine and mommy won't notice. Another big improvement in 042stab040 kernel is it reduces the number of such situations, in other words, the new kernel obeys memory limit in a more strict way.

Polishing

Finally, the kernel is now in a pretty good shape, so we can afford some polishing, minor optimizations, and fine tuning. Such polishing was performed in a few subsystems, including checkpointing, user beancounters, netfilter, kernel NFS server and VZ disk quota.

Some numbers

Totally, there are 53 new patches in 042stab040.1, compared to previous 039 kernels. On top of that, 042stab042.1 adds another 30. We hope that the end result is improved stability and performance.

Announcing rhel6-testing kernel branch/repo

Fri, 14 Oct 2011 21:40:02 GMT

Instead of having a nice drink in a bar, I spent this Friday night splitting the RHEL6-based OpenVZ kernel branch/repository into two, so now we have 'rhel6' and 'rhel6-testing' branches/repos. Let me explain why.

When we made an initial port of OpenVZ to RHEL6 kernel and released the first kernel (in October 2010, named 042test001), I created a repository named openvz-kernel-rhel6 (or just rhel6), and this repository was marked as "development, unstable". When, after almost a year, we announced it as "testing" and then, finally, "stable" (in August 2011, named 042stab035.1).

After that, all the kernels in that repository were supposed to be stable, because they are incremental improvements of the kernel we call stable. In theory it is. In practice, of course, there can always be new bugs (both introduced by us and by Red Hat folks releasing their kernel updates which we rebase to). Thus a kernel update from a repo which is supposed to be stable can do bad things.

Better late than never, I have fixed the situation tonight by basically renaming "rhel6" repository into "rhel6-testing", and creating a new repository called just "rhel6". For now, I put 042stab037.1 (which is the latest kernel which has passed our internal QA) into rhel6 (aka stable), while all the other kernels, up to and including 042stab039.3, are in rhel6-testing repo.

Now, very similar to what we do with RHEL5 kernels, all the new fresh-from-the-build-farm kernels will appear in rhel6-testing repo, at about the same time they go to internal QA. Then, the kernels which will have QA approval will appear in rhel6 (aka -stable) repo. What it means for you as a user is you can now choose whether to stay at the bleeding edge and have the latest stuff, or to take a conservative approach and have less frequent and delayed updates, but be more confident about kernel quality and stability.

A few links:
* Stable RHEL6-based OpenVZ kernels
* Testing RHEL6-based OpenVZ kernels
* OpenVZ yum repository setup file
* Official announce of rhel6-testing

LinuxCon Europe 2011 in Prague is coming

Fri, 30 Sep 2011 11:26:07 GMT

And we are coming to Prague, too! This time, there will be as many as six people and two talks from us, plus we will held a memory cgroup controller meeting.

The following OpenVZ/Parallels people are coming:

James Bottomley, Parallels virtualization CTO
Kir Kolyshkin, OpenVZ project manager
Pavel Emelyanov, OpenVZ kernel team leader (he's also taking part in Linux Kernel Summit)
Glauber Costa, OpenVZ kernel developer
Maxim Patlasov, OpenVZ kernel developer
Andrey Vagin, OpenVZ kernel developer

Two talks will be presented. Since linuxsymposium.org site is currently down, let me quote talk descriptions here.

1. Container in a file by Maxim Patlasov.

One of the feature differences between hypervisors and containers is the ability to store a virtual machine image in a single file, since most containers exist as a chroot within the host OS rather than as fully independent entities. However, the ability to save and restore state in a machine image file is invaluable in managing virtual machine life cycles in the data centre.

This talk will début a new loopback device which gives all the advantages of virtual machine images by storing the container in a file while preserving the benefits of sharing significant portions with the host OS. We will compare and contrast the technology with the traditional loopback device, and describe some changes to the ext4 filesystem which make it more friendly to new loopback device needs.

This talk will be technical in nature but should be accessible to people interested in cloud, virtualisation and container technologies.

2. OpenVZ and Linux kernel testing by Andrey Vagin.

One of the less appealing but very important part of software development is testing. This talk tries to summarize our 10+ years of experience in Linux kernel testing (including OpenVZ and Red Hat Enterprise Linux kernels). Overall description of our test system is provided, followed by details on some of the interesting test cases developed. Finally, a few anecdotal cases of bugs found will be presented.

In a sense, the talk is an answer to Andrew Morton's question from 2007: "I'm curious. For the past few months, people@openvz.org have discovered (and fixed) an ongoing stream of obscure but serious and quite long-standing bugs. How are you discovering these bugs?"

Talk is of interest to those concerned about kernel quality, and in general to people doing development and testing.

Finally, there will be a memcg meeting. Since LinuxCon will be right after the Kernel Summit, a number of kernel guys will still be there so anyone interested in cgroups can come. This meeting is a continuation of our recent discussion at Linux Plumbers (see etherpad and presentations).

See you all in Prague in less than a month!

RHEL6 goes stable!

Tue, 30 Aug 2011 17:22:43 GMT

Guys, I am very proud to inform you that today we mark RHEL6 kernel branch as stable. Below is a copy-paste from the relevant announce@ post. I personally highly recommend RHEL6-based OpenVZ kernel to everyone -- it is a major step forward compared to RHEL5.

In the other news, Parallels has just released Virtuozzo Containers for Linux 4.7, bringing the same cool stuff (VSwap et al) to commercial customers. Despite being only a "dot" (or "minor") release, this product incorporates an impressive amount of man-hours of best Parallels engineers.

== Stable: RHEL6 ==

This is to announce that RHEL6-based kernel branch (starting from kernel 042stab035.1) is now marked as stable, and it is now the recommended branch to use.

We are not aware of any major bugs or show-stoppers in this kernel. As always, we still recommend to test any new kernels before rolling out to production.

New features of RHEL6-based kernel branch (as compared to previous stable kernel branch, RHEL5) includes better performance, better scalability (especially on high-end SMP systems), and better resource management (notably, vSwap support -- see http://wiki.openvz.org/VSwap).

RHEL6 kernels can be downloaded from http://wiki.openvz.org/Download/kernel/rhel6

== Frozen: 2.6.27, 2.6.32 ==

Also, from now we no longer maintain the following kernel branches:

* 2.6.27
* 2.6.32

No more new releases of the above kernels are expected. Existing users (if any) are recommended to switch to other (maintained) branches, such as RHEL6-2.6.32 or RHEL5-2.6.18.

This change does not affect vendor OpenVZ kernels (such as Debian or Ubuntu) -- those will still be supported for the lifetime of their distributions via the usual means (i.e. bugzilla.openvz.org).

== Development: none ==

Currently, there are no non-stable kernels in development. Eventually we will port to Linux kernel 3.x, but it might not happen this year. Instead, we are currently focused on bringing more of OpenVZ features to mainstream Linux kernels.

Regards, OpenVZ team.

Linux Plumbers: Containers and CGroups miniconf

Wed, 24 Aug 2011 15:16:32 GMT

We have finally filed a number of proposals for the up-coming Containers and CGroups miniconf to be held during Linux Plumbers Conference, 7 to 9 September 2011 in Santa Rosa, CA.

From those proposals, one can clearly see what are our plans regarding the mainline integration. In a few words: dcache management, memory and CPU cgroup controllers improvements, container enter, improved /proc virtualization, checkpoint/restart [mostly] in userspace (of which I have blogged recently), and making vzctl work with mainline kernel containers. Oh, and the interesting loopback-like block device to hold a container filesystem (a.k.a. ploop).

Quite a lot of interesting stuff, what do you think?

Checkpoint/restart (mostly) in user space

Sat, 13 Aug 2011 09:26:08 GMT

There is a good article at lwn.net telling about one of our latest development.

We have checkpoint/restart (CPT) and live migration in OpenVZ for ages (well, OK, since 2007 or so), allowing for containers to be freely moved between physical servers without any service interruption. It is a great feature which is valued by our users. The problem is we can't merge it upstream, ie to vanilla kernel.

Various people from our team worked on that, and they all gave up. Then, Oren Laadan was trying very hard to merge his CPT implementation -- unfortunately it didn't worked out very well either. The thing is, checkpointing is a complex thing, and the patch implementing it is very intrusive.

Recently, our kernel team leader Pavel Emelyanov got a new idea of moving most of the checkpointing complexity out of the kernel and into user space, thus minimizing the amount of the in-kernel changes needed. In about two weeks of time he wrote a working prototype. So far the reaction is mostly positive, and he's going to submit a second RFC version for review to lkml.

For more details, read the lwn.net article. After all, while I am sitting next to Pavel, Mr. Corbet ability to explain complex stuff in simple terms is way better than mine.

ioping

Thu, 02 Jun 2011 18:01:18 GMT

My colleague koct9i, whose daily job is developing and fixing OpenVZ kernel, was feeling bored last weekend, and to entertain himself he wrote a small utility called ioping. The idea is simple and straightforward: to wrote a utility similar to ping, which will show I/O latency in the same way ping shows network latency. The idea is very simple but I haven't see something like this. Actually, the tool was written to help investigating OpenVZ bug #1880).

I liked ioping and worked on it a bit, too, just for run. Among some other minor stuff I have added a man page and spec file, so it is now available as an RPM package.

Official project site: http://code.google.com/p/ioping/

My RPM packages and stuff: http://kir.sacred.ru/ioping/

Ubuntu 11.04 template

Wed, 04 May 2011 07:15:36 GMT

In addition to a bunch of template updates released last week, yesterday night we released Ubuntu 11.04 templates for OpenVZ, for both x86 and x86_64 architectures. They are currently in beta and therefore available from http://wiki.openvz.org/Download/template/precreated/beta (which is actually just a wiki page with pretty links to http://download.openvz.org/template/precreated/beta/).

Make sure you use latest vzctl (3.0.26.2), otherwise vzctl enter won't work with Ubuntu 11.04 containers. As usual, report all bugs to http://bugzilla.openvz.org/

template update

Fri, 01 Apr 2011 22:22:04 GMT

Good news everyone, I am in the process of updating all the precreated templates. From now on I will probably do full updates quarterly.

New templates:
* Debian 6.0
* SUSE 11.4

Moved from beta:
* SUSE 11.2, 11.3
* Fedora 14

Moved to unsupported (because they are no longer supported by upstream vendors):
* Fedora 12 (EOL Dec 02 2010)
* SUSE 11.1 (EOL Jan 14 2011)

The update will appear in a few days, probably as soon as next Tuesday or so.

And yes, we are still waiting for CentOS 6 to be ready...

Update: it took me much more time than expected; updates are finally released 28 Apr 2011
http://wiki.openvz.org/Download/template/precreated

Solar Designer on Owl, OpenVZ and stuff: things you ached to know even if you didn't realize you did

Fri, 25 Feb 2011 17:35:47 GMT

Openwall GNU/*/Linux recently has released version 3.0. Owl project (yeah, you can call it Owl,too, which stands for Openwall Linux) is afloat around ten years, and it's an active and evolving. One of the neat features of Owl 3.0 is OpenVZ support. That was exactly the item which we got excited about, so we asked Solar (and he agreed kindly!) for a short interview.

— When someone is trying to find out what do you do, they mostly come across that you developed John the Ripper, currently lead the Owl project and participate in different open source security projects. Perhaps there are some new fancy things you're busy with, could you tell us a bit about it?

— This is nothing fancy, but besides the things you mentioned, we (some folks on the Openwall team) spend plenty of time on related client-facing work, including setting up and maintaining Owl/OpenVZ systems remotely, with lots of additional software running on top of them. This is one reason why our Live CDs include SSH remote access, after trivial initial configuration by someone with physical access or over IP-KVM. Then we can proceed with installation starting with partitioning of the hard disks while working over SSH. Besides providing income, which is much needed to fund Owl development and our other activities, this also ensures that we're testing our stuff under real-world usage conditions, and it provides stability for the project (we won't unexpectedly “lose interest” in it one day; we're going to continue to need updates for those systems that we're responsible for ourselves).

As to fancier things, I have plenty of potential project ideas, but too little time. So I mostly have to limit my activities to work related to projects already started. You mentioned that I “developed” John the Ripper, which I did, but it shouldn't just stay “developed” for many years more, it needs to evolve further, and there's demand for that (such as from penetration testing firms). This is about coming up with algorithms to produce more optimal candidate password streams, as well as about having them reflect new kinds of weak passwords that people start to use. This is also about efficient use of new CPU types, parallel processing, distributed processing, and use of GPUs. I did only a little bit of work on some of these things lately, being busy with lots of other stuff, but this is something I need to get back to and have specific ideas on.

A luckier project in 2010 was passwdqc, our proactive password/passphrase strength checking tool set (PAM module, library, command-line programs). Unlike John the Ripper, which audits password hash files, passwdqc does not permit weak passwords to be set in the first place. Dmitry V. Levin (of ALT Linux) and I made numerous enhancements to passwdqc: implemented new features, more effective checks (and tested those checks on thousands of real-world passwords), and at the same time made passwdqc even more portable. Proactive password/passphrase strength checking with passwdqc (applied when a user changes their Unix password) is now supported on Linux, FreeBSD, DragonFly BSD, OpenBSD, Solaris, and HP-UX.

— I've read through Owl release notes, slides and even that frightening "Weee SUID ftw!/O, no, no SUID" discussion on Slashdot, so what's so unique in your distro except no SUID (thanks to Openwall guys) and OpenVZ on board (feeling proud of ourselves)?

— It's a combination of things that make our distro unique, although a few things are also unique on their own: the lack of SUID programs in a usable default install, and syslogd credential passing (anti-spoofing). Some other things that make Owl special are:
* Proactive source code reviews of security-critical components of third-party software that we package.
* Security hardening changes in many packages, including some relatively uncommon changes (e.g., environment variable sanitization in glibc, Debian-generated weak key blacklisting in OpenSSH, privilege separation added to telnetd).
* Self-contained yet small: although the distro is small, it contains everything needed to easily rebuild itself from source.
* The included build environment is additionally capable generating ISOs and OpenVZ container templates ("make iso" and "make vztemplate").
* A single CD contains a live system, installable packages, the installer program, as well as full source code and the build environment.
* Old-fashioned text-only installer, simple scripts and configuration files, and reduced bloat overall. The system does not try to outsmart its administrator.
* RPM'ed default kernel, yet manual custom kernel builds are conveniently supported as well (only a few cumulative patch files are applied in the package).
* A good selection of small yet handy sysadmin and diagnostic tools, such as dmidecode, hdparm, ltrace, mdadm, mtree and nc (our ports from OpenBSD), Nmap (and Ncat), pciutils, smartmontools, strace. Of course, most of these are also available for/on mainstream distros, yet given Owl's purpose as a good "base system" for servers it is worth mentioning that we include all of these and more in our otherwise small set of packages, as well as in a default install.

— How big approximately is the user base?

— There's no "Owl user" registration mechanism, so I have no numbers on the size of our user base. It "feels" small, yet sufficient for us to continue with the project. Besides, much of the software that we develop as part of Owl we're also making available separately, and many of our patches, etc. are reused by other distros. For example, our "tcb" suite, which allows for the "passwd" command to work without being SUID, is also reused by ALT Linux's distributions and by Mandriva. This increases its user base a lot.

— What are the most common use cases for the Owl?

— Most common use cases? For us, that's OpenVZ "host systems", as well as OpenVZ container templates built with Owl as the "base system" and with extra software added (sort of "virtual appliances" that we make and install ourselves). We (or rather our client companies) mostly use this to provide "advanced" web hosting.

Here's a project built on top of OpenVZ and Owl, WebEnabled, which is a website development platform, involving multiple backend servers internally, with features such as "one-click" installs of popular web apps and cloning of existing "virtual hosts" (such as for the development/QA/production lifecycle).

We've also experimented with building Owl-based virtual machine hard disk images for a specific project, where the VMs would be run on end-users' desktop/laptop computers. Owl worked well for this purpose.

Since Owl's OpenVZ support is still relatively new (introduced into Owl-current a year ago, but only included in a stable release with Owl 3.0), we're hoping that more people and companies will make use of it in various ways, including in "virtual appliances" that they'd release. These could be VM images that also run OpenVZ containers, or they could be OpenVZ container templates or a mix of both.

Finally, we're aware of a McAfee (now Intel) hardware appliance product that is built upon an Owl-derived system (without OpenVZ, though). We'd be happy to see more hardware appliances make use of Owl as well, including of its OpenVZ integration.

We'd be happy to help more companies make and maintain Owl-based and Owl-derived systems, appliances, etc. Much of this work can be outsourced to us if desired. I think that hosting providers offering Linux-KVM VPS hosting may be interested in making Owl one of their standard distro options. They may specifically advertise the ability for a customer to create multiple OpenVZ containers inside such a VPS, which may provide an extra selling point for their offering vs. the competitors'. I am not aware of hosting providers doing this yet, although I am aware of a user of Owl having tried this out on his own and he was pleased with the result. Thus, I am mentioning this not so much as an answer to your question, but rather to encourage such use by hosting providers.

— I usually pay attention to mascots :) Yours looks like an owl near the boundary post/booth, it's really cool! Was it the first idea? Or there was a contest for a best picture?

— I'm glad that you like it. There was no formal contest. The Owl project logo that we currently use was based on one of several sketches contributed by Gleb Todosov, who also created our CD jewel case artwork. We used the latter up through our 2.0 release, and you can still see a portion of it in use on the first slide in our presentation.

— We're excited that you support OpenVZ in your distro and would like to dig into details. Why did you decide to do that? I guess you tried OpenVZ before, what was your previous experience? What did you use it for?

— Personally, I got interested in container-based virtualization in 1999 or thereabout (although I didn't know the concept would be known under this name). My first exposure to Virtuozzo was during SWsoft's public beta testing, which I think was in 2000. It was possible to register for a free trial account, which I did, and I got onto an SWsoft-hosted free trial system running a 2.4.0-test kernel with early and unreleased Virtuozzo patches. I ended up finding a vulnerability and reporting it to SWsoft.

Fast forward to 2005, the OpenVZ project was about to be announced to the public, and Openwall was contracted by SWsoft to perform a security audit of OpenVZ, which I personally worked on in close coordination with the OpenVZ developers. This was a pleasant experience, and the code quality was better than what I was used to seeing. Most of the issues that were discovered during the audit got fixed promptly. During the audit, I made my own test install of OpenVZ (for fuzzing, etc.) on top of an Owl system on a machine dedicated for the purpose.

OpenVZ was released in 2006. As far as I can tell by looking at file timestamps now, it was August 2006 when we slowly started to deploy it on top of some of our clients' Owl systems - replacing the kernels and adding vz* tools, but keeping the Owl userland on what became OpenVZ host systems. After a long while, we had plenty of systems running Owl and OpenVZ together, and we also had Owl-based userlands running inside "VEs" (the word "container" was not in use in OpenVZ context yet). At some point, we needed to introduce more sysadmins to the team and we also needed more systems of this kind installed to satisfy our clients' needs. It proved painful to explain all the little details (hacks) of making things work right.

We had the idea of integrating OpenVZ into Owl before - I even briefly mentioned it during a 2006 interview - but the difficulty mentioned above might have been the straw that finally made us do it, despite that it meant temporarily giving up some of our kernel hardening patches (not reimplemented for 2.6/OpenVZ kernels yet).

— What do you like in latest releases (already tried VSwap?)? What do you not like (I myself can hardly imagine that after there is no need to set all the UBC manually and get the blind headaches there is something left to improve, but I'm sure I need a fresh look on those things), so is there anything you find obscure or inconvenient?

— Of your "latest releases", we're only making (a lot of) use of the RHEL5 branch kernels ("testing" and/or "stable" on different occasions), with some additional changes (we make our own builds). I guess that you were referring to your newer branches. Unfortunately, we have not tried those out yet. This means that we haven't tried VSwap yet, even though this is a very welcome and anticipated feature (and change from the older UBC model). We're likely to start moving to your RHEL6 branch kernels in Owl-current this year, and we're likely to use them in our next major release (Owl 4.0). We'll be sure to provide feedback as we do this.

As to further improvements that we'd like to see in OpenVZ, it's security hardening, primarily against Linux kernel vulnerabilities that might turn into "container escape" ones in OpenVZ-patched kernels. One fairly obvious enhancement would be to be able to set a container to 32-bit only, 64-bit only, or mixed. The first two settings would disallow the other type of syscalls, thereby significantly reducing the exposure of kernel interfaces to possible attacks from the container's system. Another Owl developer tells me that this specific enhancement was also suggested upstream (LXC, I guess).

— Is there anything you would like to add or tell to millions :) of OVZ blog readers?

— Feedback, please! We need more feedback on Owl; join the owl-users mailing list and post there, please. Successes, failures, comments, opinions, requests, all kinds of feedback is welcome. Also, please contribute to our wiki. So far, it appears that most people who use Owl successfully don't bother contacting us ("no need"), those who run into minor issues and solve/workaround the issues on their own also don't bother notifying us ("no need" or so they think, wrongly), and most of those who run into trouble or are unhappy with Owl for whatever reason either notify individual Owl developers privately or (possibly) give up without telling us of the problem. If you use Owl (or have tried or considered to), please do your part to help correct this "feedback problem".

devel@ mailing list mess is no more

Thu, 24 Feb 2011 14:13:02 GMT

OK, I must admit is was a very bad idea of me to subscribe our devel at openvz dot org mailing list to containers at linux-foundation dot com mailing list.

This is to announce that from now on devel@ is a separate list, not mirroring containers@ or anything. From now on, if the topic is openvz-specific, like a patch to OpenVZ, please use devel@. If the topic is about containers (as appearing in mainline), use containers@.

Let me explain. Initially, when we started moving OpenVZ project forward, we wanted to discuss all the things about containers on a mailing list, and therefore I created devel@. Later, then other parties joined, it was decided to create containers at osdl.org mailing list (remember OSDL later became the Linux Foundation). At that time I was worried that the discussions will split, and decided to just subscribe our devel@ to containers@, so devel@ becomes a super-set of containers@ (i.e. every message posted to containers@ will appear on devel@, but not vice versa).

Of course it ended up being a big mess. Better late than never, mess is no more!

back to 2006, or openvz bug #60

Sat, 19 Feb 2011 23:44:11 GMT

Some software bugs, while being simple and stupid, have an interesting and long lasting life. Here is the story of such a very simple bug with a lifespan of about 5 years (or more? I don't know when it was introduced). The bug doesn't worth looking at otherwise, so I'll try to be short, and more info is available from the links. OK,

back in 2006 I whined about a bug in sysvinit we found. Until today I thought is was never fixed upstream.

This night I found out that it's actually fixed in sysvinit (2.87dsf), released in Jul 2009, according to its changelog:

 * Adjust init to terminate argv0 with one 0 rather than two so that
    process name can be one character longer.  Patch by Kir Kolyshkin.

Unfortunately it wrongly contributes me as a patch author. The actual author is Dmitry Mishin, as seen in OpenVZ bug #60, I just submitted it.

on static function declaration

Mon, 07 Feb 2011 22:02:25 GMT

If you are a seasoned C programmer, skip this post entirely (or try to find bugs in it). If you know C but don't consider yourself an expert, please read on -- it might be helpful.

I was working a bit on vzctl today (my target was bug #1757, which is still a work in progress) and ... I am not sure how, but I ended up declaring most functions in src/vzlist.c as static. I thought it doesn't have any practical value -- I was wrong!

In C, if you declare the function as static, it means its visibility is limited to the translation unit (i.e. a file) in which it is defined. In other words, you can only call/use a static function from another function in the same file.

Now, in vzctl sources vzlist.c is only linked to one binary -- vzlist, and therefore I thought it doesn't make much sense to declare functions as static. Nevertheless I did it (see git commit).

Next thing I got is a set of compiler warnings! OK, all right, let's take a look...

First set of warnings is self-explanatory. See:vzlist.c:825:14: warning: ‘parse_var’ defined but not used vzlist.c:1075:14: warning: ‘remove_sp’ defined but not used vzlist.c:1357:12: warning: ‘get_stop_quota_stats’ defined but not used

Easy! In some ancient time, these functions were used, now the code has changed and no one needs these three, but they were not removed for some reason (probably just forgotten). Solution: remove the dead code (see git commit).

Second set of warnings looks similar:vzlist.c:400:1: warning: ‘dcachesize_m_sort_fn’ defined but not used vzlist.c:400:1: warning: ‘dcachesize_l_sort_fn’ defined but not used vzlist.c:400:1: warning: ‘dcachesize_b_sort_fn’ defined but not used vzlist.c:400:1: warning: ‘dcachesize_f_sort_fn’ defined but not used vzlist.c:411:1: warning: ‘diskinodes_s_sort_fn’ defined but not used vzlist.c:411:1: warning: ‘diskinodes_h_sort_fn’ defined but not used

Hmm... all these *_sort_fn are sort functions generated by means of a few #define statements, and they are used when vzlist needs to sort its output by some column or parameter (vzlist -s). It is very strange that these are not used, because they should be. Let's take a closer look... zOMG! it's a bug!

Apparently, someone was using copy-paste technique* and forgot to change the names of the functions. The bug is, when you ask vzlist to sort its output to, say, dcachesize failcounter values, it sorts it by dcachesize held values instead, because of the wrong sort function used. Such bugs are hard to notice manually, and there are no autotests for vzlist.

* Yes some parts of vzlist is a copy-pasted mess, I am slowly working on untangling it. For example, see my previous cleanup patches (committed back in June 2010):
src/vzlist.c: streamline a few macros
vzlist: put similar print_ functions in a macro
vzlist.c: simplify last_field logic

Morale: sometimes declaring functions as static actually helps!

PS if you see mistakes in this blog post, patches to it are welcome. It's 1am here and I am a bit sleepy.

Kernel 2.6.27 repin aka "Unexpected return"

Wed, 26 Jan 2011 15:13:42 GMT

You probably thought we have abandoned 2.6.27 kernel branch. Well, we ourselves thought we did (although it was not yet officially announced). Then, out of a sudden, kernel 2.6.27-repin.1 is released, rebasing to latest upstream kernel (2.6.27.57), and fixing OpenVZ bug #1593.

The thing is, this kernel is called after Ilya Repin, a leading Russian painter and sculptor of the Peredvizhniki artistic school. One of his best paintings is called "Unexpected Return", and I happen to enjoy the original in Tretyakov Gallery here in Moscow a couple of weeks ago. So here it is: the unexpected return of 2.6.27 kernel. It took Ilya 4 years to finish the painting, it took Pavel 6 months to release the fix. Better late than never, that is.

Please enjoy: Ilya Repin. Unexpected return. 1884-1888.

news from the VSwap front

Tue, 25 Jan 2011 18:44:57 GMT

I have added vswap confguration samples to vzctl git. Basically, you set physpages and swappages and leave every other beancounter at unlimited. For example, this is how ve-vswap-256m-conf.sample looks like:

# UBC parameters (in form of barrier:limit)
PHYSPAGES="0:256M"
SWAPPAGES="0:512M"
KMEMSIZE="unlimited"
LOCKEDPAGES="unlimited"
PRIVVMPAGES="unlimited"
SHMPAGES="unlimited"
NUMPROC="unlimited"
VMGUARPAGES="unlimited"
OOMGUARPAGES="unlimited"
NUMTCPSOCK="unlimited"
NUMFLOCK="unlimited"
NUMPTY="unlimited"
NUMSIGINFO="unlimited"
TCPSNDBUF="unlimited"
TCPRCVBUF="unlimited"
OTHERSOCKBUF="unlimited"
DGRAMRCVBUF="unlimited"
NUMOTHERSOCK="unlimited"
DCACHESIZE="unlimited"
NUMFILE="unlimited"
NUMIPTENT="unlimited"

# Disk quota parameters (in form of softlimit:hardlimit)
DISKSPACE="1G"
DISKINODES="200000"
QUOTATIME="0"

# CPU fair scheduler parameter
CPUUNITS="1000"

As you can see, physpages (ie RAM size) is set to 256 megabytes, while swappages (ie swap size) is set to 512 megabytes, all the other beancounters are unlimited. Wow, it's never been easier to configure your containers!

Now, we can utilize this stuff using RHEL6 based kernel. This is what we see from inside the container:

[root@localhost ~]# vzctl enter 103
entered into CT 103
[root@localhost /]# free
             total       used       free     shared    buffers     cached
Mem:        262144      23936     238208          0          0      10968
-/+ buffers/cache:      12968     249176
Swap:       524288          0     524288

 [root@localhost Version: 2.5 uid  resource 103:  kmemsize lockedpages privvmpages shmpages dummy numproc physpages vmguarpages oomguarpages numtcpsock numflock numpty numsiginfo tcpsndbuf tcprcvbuf othersockbuf dgramrcvbuf numothersock dcachesize numfile dummy dummy dummy numiptent

[root@localhost /]# cat /proc/meminfo MemTotal:         262144 kB MemFree:          238208 kB Cached:            10968 kB Active:            16956 kB Inactive:           1384 kB Active(anon):       6352 kB Inactive(anon):     1020 kB Active(file):      10604 kB Inactive(file):      364 kB Unevictable:           0 kB Mlocked:               0 kB SwapTotal:        524288 kB SwapFree:         524288 kB Dirty:                 0 kB AnonPages:          7364 kB Mapped:             3416 kB Shmem:               124 kB Slab:               4012 kB SReclaimable:       1088 kB SUnreclaim:         2924 kB

/]# cat /proc/user_beancounters held maxheld barrier limit failcnt 4722976 4853726 9223372036854775807 9223372036854775807 0 0 0 9223372036854775807 9223372036854775807 0 4296 13875 9223372036854775807 9223372036854775807 0 31 31 9223372036854775807 9223372036854775807 0 0 0 0 0 0 33 33 9223372036854775807 9223372036854775807 0 5984 5985 0 65536 0 0 0 9223372036854775807 9223372036854775807 0 2696 2696 9223372036854775807 9223372036854775807 0 4 4 9223372036854775807 9223372036854775807 0 5 6 9223372036854775807 9223372036854775807 0 1 1 9223372036854775807 9223372036854775807 0 12 18 9223372036854775807 9223372036854775807 0 69760 0 9223372036854775807 9223372036854775807 0 65536 0 9223372036854775807 9223372036854775807 0 2312 10768 9223372036854775807 9223372036854775807 0 0 0 9223372036854775807 9223372036854775807 0 51 53 9223372036854775807 9223372036854775807 0 1172451 1172451 9223372036854775807 9223372036854775807 0 370 390 9223372036854775807 9223372036854775807 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 14 14 9223372036854775807 9223372036854775807 0

cpulimit is back in RHEL6 based kernel

Thu, 20 Jan 2011 16:53:15 GMT

Hard CPU limit (ability to specify that you don't want this container to use more than X per cent of CPU no matter what) is back in latest RHEL6-based kernel, 042test006.1, which has just been released.

The feature was only available for the stable (i.e RHEL4 and RHEL5-based) kernels, and was missing from all of our development kernels from 2.6.20 to 2.6.32. So while it was always there in stable branches, the feeling is like it's back.

In order to use CPU limit feature, set the limit using vzctl set $CTID --cpulimit X, where X is in per cent of one single CPU. For example, if you have single 2 GHz CPU and want container 123 to use no more than 1 GHz, use vzctl set 123 --cpulimit 50. If you have 2 GHz quad-core system and want to use no more than 4 GHz, use vzctl set 123 --cpulimit 200. Well, in the second case it might be better to just use --cpus 2. Anyways, see vzctl man page.

RHEL6-based kernel: try today not next year!

Wed, 29 Dec 2010 15:21:05 GMT

We have just released a new RHEL6-based kernel, 042test005. It is shaping up pretty good — as you can see from the changelog, it's not just bug fixes but also performance improvements. If you haven't tried it yet, I suggest to do it today! Do not postpone this until 2011 — after all, this is what will become the next stable OpenVZ kernel.

RHEL6 kernel needs an appropriate (i.e. recent) Linux distribution. If you don't want latest Fedora releases, can't afford RHEL6, and tired of waiting for CentOS 6, I suggest you go with Scientific Linux 6 (SL6). This is yet another RHEL6 clone developed and used by CERN, Fermilabs and other similar institutions.

While SL6 is still at its infancy (they have recently released alpha 3 and plan to release beta 1 at Jan 7 2011), it it worth trying since it's based on a very stable set of sources from RHEL6. Repositories and stuff are available from http://ftp.scientificlinux.org/linux/scientific/6rolling/

vzctl 3.0.25 released today, woo-hoo!

Mon, 20 Dec 2010 15:39:10 GMT

I thought that Monday is a good day to release a new version of vzctl which was worked on for the last six months. There are a few important changes in this release, let me go through it.

First, we have finally removed all the cronscripts trickery required for CT reboot. The thing is, if the container owner issues 'reboot' command from inside, the container just stops. Now, something needs to be done on the host system to start it again. Until this release, this was achieved by a hackish combination of vzctl (which adds an initscript inside container to add a reboot mark) and a cron script (which checks for the stopped containers having that reboot mark and starts those). Yet another cron script takes care about a situation when a CT is stopped from the inside -- in this case some cleanup needs to be done from the host system, namely we need to unmount the CT private area, and remove the routing and ARP records for the CT IP.

There are a few problems with this cron-based approach. First, initscript handling can be different in different distributions, and it's really hard to support all of the distros. Second, cron script is run every 5 minutes, which means a mean time to reboot (or clean up network rules) is 2.5 minutes. To say it simple, it's all hackish and unreliable.

Now, this hairy trickery is removed and replaced by a simple and clean daemon called vzeventd, which listens to CT stop and reboot events, and runs clean and simple scripts. No more trickery, no more waiting for reboot. The only catch is this requires support from the kernel (which comes in a form of vzevent kernel module).

Second, new vzctl is able to start Fedora 14 containers on our stable (i.e. RHEL5-2.6.18) kernels. The thing is, Fedora 14 have glibc patched to check for specific kernel version (>=2.6.32 in this case) and refuse to work otherwise. This is done to prevent glibc from using the old kernels with some required features missing. We patch our kernels to have those features, but glibc just checks the version. So, our recent kernels is able to set osrelease field of uname structure to any given value for a given container. Now, vzctl 3.0.25 comes with a file (/etc/vz/osrelease.conf) which lists different distros and their required kernel version, which it sets during start and exec.

I want to briefly mention yet another feature of recent vzctl (which, again, needs kernel support) -- an ability to delegate a PCI device into a container. It is only supported on RHEL6 kernel at the moment, and the only devices that we have tried are NVidia GPUs.

Besides these three big things, there are a lot of improvements, fixes, and documentation updates all over the tree. I don't know of any known regressions in this release but I guess it's not entirely Bug Free. Fortunately there's a way to handle it -- if anything really bad appears in this version, it will be fixed by a quick 3.0.25.1 update. This worked pretty well for vzctl-3.0.24, should work fine this time, too.

As always, please report all the bugs found to http://bugzilla.openvz.org/

On kernel exploits and OpenVZ user beancounters

Fri, 26 Nov 2010 15:37:24 GMT

Yesterday a guy with his name written in Cyrillic letters ("Марк Коренберг") and a @gmail.com email address posted a kernel exploit to the Linux kernel mailing list (aka LKML). This morning one brave guy from our team tried to run it on his desktop -- and had to reboot it after a few minutes of total system unresponsiveness.

The bad news are the exploit is pretty serious and causes Denial of Service. It looks like most kernels are indeed vulnerable.

The good news is OpenVZ is not vulnerable. Why? Because of user beancounters.

The nature of exploit is to create an unlimited number of sockets, thus rendering the whole system unusable so you need to power-cycle it to bring it back to life. Now, if you run this exploit in an OpenVZ container, you will hit the numothersock beancounter limit pretty soon and the script will exit. This is an except from /proc/user_beancounters after the run:

 cat /proc/user_beancounters 
       uid  resource                     held              maxheld              barrier                limit              failcnt
.....
            numothersock                    9                  360                  360                  360                    1
.....

As you can see, current usage is 9, while peak usage is 360, same as limit. Finally, failcnt is 1 meaning there was once a situation when the limit was hit.

I went further and set numothersock limit to 'unlimited', and re-run the exploit. The situation is much worse in that case (don't try it at home, you've been warned), system slows down considerably, but I was still able to login to the physical server using ssh and kill the offending task from the host system using SIGTERM.

Now, this is how /proc/user_beancounters look after the second run:

       uid  resource                     held              maxheld              barrier                limit              failcnt
      111:  kmemsize                  1237973             14372344             14372700             14790164                   80
.....
            numothersock                    9                 2509  9223372036854775807  9223372036854775807                    1

As you can see, even with numothersock set to unlimited, another beancounter, kmemsize, is working to save the system.

Of course, if you set all beancounters to unlimited, exploit will work. So don't do that, unless your CT is completely trusted. Those limits are there for a reason, you know.

042test002.1 kernel is based on final RHEL6 and contains VSwap.

Mon, 22 Nov 2010 16:13:03 GMT

Guys and girls (hugs to LinuxChix at this point!), you'll choke in with delight with the news incoming, I guarantee it! So, let me climb a stool and announce with the full pathos: we released a new kernel based on final RHEL6: 042test002.1. Yes, I knew it would leave you speechless. Besides fixed modules signing and many fixes and reworks in network, VFS, UBC, CPT and sysctl it contains brand new memory management model, we call it VSwap (trumpets here!). Previous version of this kernel, 042test002.1 was based on RHEL6 beta. I guess no one here needs an explanation why final releases are better than betas.

So, back to VSwap. We used to set User Beancounters to manage a container's resources and memory, but it appeared to become a bit complicated and required reading lots of manuals if you wanted things to work the way you planned and not just set some parameters randomly. (When I installed Gentoo for the first time, I was totally unaware about USE flags details, for most of them I was even unable to guess what they were for, so I simply enabled those, which names I liked the most). It seems, some users tried the same method at least once while dealing with UBC. In other words, VSwap is a killer feature, which simplifies a ton CT management, you just think about it: instead of setting 20 different beancounters you need to set only two, RAM and swap!

At the moment there are two primary parameters to set: physpages and swappages. The rest of beancounters are secondary parameters and do not need your obligatory attention. In short, physpages limits the physical memory (RAM) available to processes inside a container. The barrier is ignored, and the limit sets the limit. Currently the user memory and the page cache are accounted into physpages. swappages limits the amount of swap space which can be used for processes inside a container. The barrier is ignored, and the limit sets the limit.

The sum of physpages.limit and swappages.limit limits the maximum amount of allocated memory which can be used by a container. When physpages limit is reached, memory pages belonging to the container are pushed out to so called virtual swap (vswap). The difference between normal swap and vswap is that with vswap no actual disk I/O usually occurs. Instead, a container is artificially slowed down, to emulate the effect of the real swapping. Actual swap out occurs only if there is a global memory shortage on the system.

Our beloved developers put a lot of effort into this feature (it's actually several years of hard working days and nights), so please do try this kernel! To do that you need either RHEL6 (or CentOS 6, which is not yet available) or recent Fedora releases (12, 13, 14). As usual, it's available in Download section.

028stab070.12 is dead-born: three good reasons to keep your system up-to-date.

Fri, 19 Nov 2010 16:59:46 GMT

You just think about what people do when your stable kernel has a bug. Do they pity you? Do they file the reports? Do they try to help and contribute to the community? No way. They just flood you with sarcastic emails telling something like 'I thought you test your kernels before being released'. Of course we do test them (not mentioning that we do love them as much as our own kids and care for them!). We even find bugs in RHEL from time to time while testing those kernels of ours. OK, let's skip the sentimental part and pass to the very core of the subject on.

Q: Aaaa! My containers are invisible for the rest of people! What happened?
A: ARP entries are not set for host-routed (i.e. venet) devices, leading to CTs unreachable from the network.

Q: Why on Earth had this happened?
A: The problem actually was in the old version of patch utility (2.5.4) used by one of developers on a machine different from his usual and beloved one. This buggy utility (2.5.4) misapplied one hunk of a patch, which led to a buggy code in net/core/neighbour.c. Take a look what it did (the following piece is the interdiff between the good code and the bad one):

diff -U2 linux-2.6.18.ovz/net/core/neighbour.c linux-2.6.18.ovz/net/core/neighbour.c
--- linux-2.6.18.ovz/net/core/neighbour.c       2010-10-04 14:27:32.000000000 +0400
+++ linux-2.6.18.ovz/net/core/neighbour.c       2010-11-12 17:35:34.000000000 +0300
@@ -1549,4 +1549,6 @@
                if (!ve_accessible_strict(tbl->owner_env, get_exec_env()))
                        continue;
+               if (!ve_accessible_strict(tbl->owner_env, get_exec_env()))
+                       continue;
                read_unlock(&neigh_tbl_lock);

@@ -1602,6 +1604,4 @@
                if (tbl->family != ndm->ndm_family)
                        continue;
-               if (!ve_accessible_strict(tbl->owner_env, get_exec_env()))
-                       continue;
                read_unlock(&neigh_tbl_lock);

So, instead of applying the same code to two identical functions, neigh_add() and neigh_delete(), it applied the same hunk twice into neigh_delete(), left neigh_add() untouched. Thus ARP record was not added, because vzctl calls it in form "ip neigh add proxy x.x.x.x dev eth0", which is supposed to add an ARP record (by the way, you can see an ARP table using 'arp -a'). Because of the mispatched kernel (those scary lines you've seen above), the record was not added and containers became unreachable.

Q: Oh, pain! And what should I do now?
A: Yesterday when things happened it was better to roll back to the previous kernel, 028stab070.7. But today we have already released the new kernel with the fix for this problem: 028stab070.14.

Oh, I was going to narrate about three good reasons to keep your system up-to-date (what a fairy-tale without an obligatory epimyth?). The first one is obvious: when you update the system, you've got all the utilities up-to-date and they won't spoil your patches. The second one is pretty clear, too: when your system is up-to-date, you've got new fancy stuff. Also, when you update your system and use brand new kernels, a flower grows in a soul of a developer or a QA person :)

Speaking of flowers, Kir himself put all of his heart in rhel5-testing branch. It was supposed to improve the quality of our stable kernels. This particular bug (really easy to catch, actually) and the whole story tells us one sad thing: almost nobody runs rhel5-testing. When you run those kernels too (along with our QA, which does the testing), you add different hardware, different setups and different usage scenarios. If you want to make sure things work for you and contribute to the community, please invest some of your time and hardware and test it on your box(es).

On RHEL5 kernel releases

Sat, 13 Nov 2010 17:43:28 GMT

You might have noticed that we have announced a new kernel branch named rhel5-testing a while ago (back in July, to be more specific). The idea is pretty simple: at the same time as giving the new kernel to our internal QA we are releasing it to rhel5-testing.

Although this change imposes some more work on me (more kernels to release, scripts to run, changelogs to prepare), I'm pleased to say that this model works very well. First, vendors who use our kernels as a base for theirs (for example, OWL) now enjoy earlier access to the sources. Second, new kernels get more testing coverage due to OpenVZ users who choose to use this branch. Finally, it works as a “technology preview”.

Now, let me explain why we have so strange version numbers in the recent rhel5-testing kernels — kernels 028stab07x are intermixed with 028stab070.y. The thing is, we still keep updating 028stab070.y with new fixes and upstream (RHEL) updates, while 028stab07x is a newer “sub-branch” which adds a few new features:

live migration of containers with NFS and AutoFS mounts
iotop working in containers and the host system

Because of these new features, these kernels haven't reached the stability yet so we keep releasing those in rhel5-testing. Hopefully soon it will end up being stable enough and we will abandon 028stab070.y in favor of 028stab078 (or so).

Update: this post was mostly written yesterday. Today we have just released 028stab078.1 kernel.

OpenVZ spiced things up for e-reader on-line store.

Thu, 11 Nov 2010 08:39:57 GMT

When it comes to e-reader advertising, what do you expect to see on its screen? That's right, some trendy titles, I bet they simply browse popular e-reading resources (I won't mention them here to avoid shilling and luring) and choose some titles from best-selling top ten. There still are open minded and I'd rather dare to call them alternative folks, who put there OpenVZ users guide for the demo on the main page!

Just before someone starts blaming it's shopped, here's the proof link
Also it's good bearing O.Henry's company on that screen! I tried to understand why the users guide. Is it "buy-this-one-because-you're-geeky"? Or this is because OpenVZ manuals are as popular as Stieg Larsson endless series? Or perhaps someone wanted to cheer OpenVZ team up? Whatever the purpose was, they did cheer us up. Hurray!