OpenVZ

Announcing the OpenVZ Maintenance Partnership

Wed, 08 May 2013 00:06:13 GMT

Big news for every serious OpenVZ user. Finally we have it!

Parallels, a sponsor behind OpenVZ project, is now offering an OpenVZ Maintenance Partnership program. The program provides bug resolution support and feature development to the OpenVZ community. The OpenVZ Maintenance Partnership has a small annual fee and provides two benefits to partnership members.

Partnership members will receive a support ID that will allow them to submit up to 10 high priority bugs per year. These bugs will be placed at the highest priority level in the development stack.

Partnership members will also be able to submit a feature request(s) which will be reviewed by the Parallels engineering team. They will work with you to clarify the requirements and implementation options and provide an implementation estimate and a schedule.

Learn more and join the OpenVZ Maintenance Partnership here

vzstats in beta

Mon, 29 Apr 2013 18:52:48 GMT

For the last two weeks or so we've been working on vzstats -- a way to get some statistics about OpenVZ usage. The system consists of a server, deployed to http://stats.openvz.org/, and clients installed onto OpenVZ machines (hardware nodes). This is currently in beta testing, with 71 servers participating at the moment. If you want to participate, read http://openvz.org/vzstats and run yum install vzstats on your OpenVZ boxes.

So far we have some interesting results. We are not sure how representative they are -- probably they aren't, much more servers are needed to participate-- but nevertheless they are interested. Let's share a few preliminary findings.

First, it looks like almost no one is using 32-bits on the host system anymore. This is reasonable and expected. Indeed, who needs system limited to 4GB of RAM nowdays?

Second, many hosts stay on latest stable RHEL6-based OpenVZ kernel. This is pretty good and above our expectations.

Third, very few run ploop-based containers. We don't understand why. Maybe we should write more about features you get from ploop, such as instant snapshots and improved live migration.

SCALE 11x Day 1 Report

Sat, 23 Feb 2013 13:22:00 GMT

The OpenVZ Project has a booth at SCALE 11x. Kir and I will be staffing it. I wrote a lengthy blog post with the day 1 report where you can read more if you want:

http://www.montanalinux.org/scale-11x-day-1.html

I talked with Kir for about a hour last night and asked him a lot of questions. I got a lot of inside information that I will be sharing in the coming days... I promise... with pictures even.

If you are in the Southern California area and can make it to SCALE, please stop by our booth. We are #93. Assuming there is a reasonable Internet connection at the booth, I also try to be "Live from SCALE" in the #openvz IRC channel. The times are Saturday 10:00 - 18:00 PST and Sunday 10:00-16:00 PST.

yet another strange performance comparison

Mon, 22 Oct 2012 15:15:19 GMT

You know I love to write about performance comparisons, right?

I just came across the one I saw some time ago and forgot about it, but now it's linked from the Wikipedia article about OpenVZ. The paper (presented at OLS'08) is comparing performance of OpenVZ, Linux-VServer, Xen, KVM and QEMU with the performance of non-virtualized Linux. The results are shocking! OpenVZ is twice slower than reference system when doing bzip -9 of an iso image! Yet better, on a dbench test OpenVZ performance was about 13% of a reference system (which is almost eight times slower), while Linux-VServer gave about 98%.

Well guys, I want to tell you just one thing. If someone offers to sell you a car for 13% of the usual price — don't buy it, it's a scam and a seller is not to be trusted. If someone tells you OpenVZ is 2 (or 8) times slower than non-virtualized system — don't buy it either!

I mean, I do know how complicated it is to have a sane test results. I spent about a year leading SWsoft QA team, mostly testing Linux kernels, and trying to make sure test results are sane and reproducible (not dependent on the moon phase etc). There are lots and lots of factors involved. But the main idea is simple: if results doesn't look plausible, if you can't explain them, dig deeper and find out why. Once you will, you will know how to exclude all the bad factors and conduct a proper test.

Update: comments disabled due to spam

vzctl / ploop update

Tue, 16 Oct 2012 08:15:31 GMT

We have updated vzctl, ploop and vzquota recently (I wrote about vzctl here). Some changes in packaging are tricky, so let me explain why and give some hints.

For RHEL5-based kernel users (i.e. ovzkernel-2.6.18-028stabXXX) and earlier kernels

Since ploop is only supported in RHEL6 kernel, we have removed ploop dependency from vzctl-4.0 (ploop library is loaded dynamically when needed and if available). Since you have earlier vzctl version installed, you also have ploop installed. Now you can remove it, at the same time upgrading to vzctl-4.0. That "at the same time" part is done via yum shell:

# yum shell
> remove ploop ploop-lib
> update vzctl
> run

That should fix it. In the meantime, think about upgrading your systems to RHEL6-based kernel which is better in terms of performance, features, and speed of development.

For RHEL6-based users (i.e. vzkernel-2.6.32-042stabXXX)

New ploop library (1.5) requires very recent RHEL6-based kernel kernel (version 2.6.32-042stab061.1 or later) and is not supposed to work with earlier kernels. To protect ploop from earlier kernels, its packaging says "Conflicts: vzkernel < 2.6.32-042stab061.1" which usually prevents ploop 1.5 installation on systems having those kernels.

To fix this conflict, make sure you run the latest kernel, and then remove the old ones:

# yum remove "vzkernel < 2.6.32-042stab061.1"

Then you can run update as usual:

# yum update

Hope that helps

Update: comments disabled due to spam.

OpenVZ turns 7, gifts are available!

Sat, 06 Oct 2012 09:31:53 GMT

OpenVZ project is 7 years old as of last month. It's hard to believe the number, but looking back, we've done a lot of things together with you, our users.

One of the main project goals was (and still is) to include the containers support upstream, i.e. to vanilla Linux kernel. In practice, OpenVZ kernel is a fork of the Linux kernel, and we don't like it that way, for a number of reasons. The main ones are:

We want everyone to benefit from containers, not just ones using OpenVZ kernel. Yes to world domination!
We'd like to concentrate on new features, improvements and bug fixes, rather than forward porting our changes to the next kernel.

So, we were (and still are) working hard to bring in-kernel containers support upstream, and many key pieces are already there in the kernel -- for example, PID and network namespaces, cgroups and memory controller. This is the functionality that lxc tool and libvirt library are using. We also use the features we merged into upstream, so with every new kernel branch we have to port less, and the size of our patch set decreases.

CRIU

One of such features for upstream is checkpoint/restore, an ability to save running container state and then restore it. The main use of this feature is live migration, but there are other usage scenarios as well. While the feature is present in OpenVZ kernel since April 2006, it was never accepted to upstream Linux kernel (nor was the other implementation proposed by Oren Laadan).

For the last year we are working on CRIU project, which aims to reimplement most of the checkpoint/restore functionality in userspace, with bits of kernel support where required. As of now, most of the additional kernel patches needed for CRIU are already there in kernel 3.6, and a few more patches are on its way to 3.7 or 3.8. Speaking of CRIU tools, they are currently at version 0.2, released 20th of September, which already have limited support for checkpointing and restoring an upstream container. Check criu.org for more details, and give it a try. Note that this project is not only for containers -- you can checkpoint any process trees -- it's just the container is better because it is clearly separated from the rest of the system.

One of the most important things about CRIU is we are NOT developing it behind the closed doors. As usual, we have wiki and git, but most important thing is every patch is going through the public mailing list, so everyone can join the fun.

vzctl for upstream kernel

We have also released vzctl 4.0 recently (25th of September). As you can see by the number, it is a major release, and the main feature is support for non-OpenVZ kernels. Yes it's true -- now you can have a feeling of OpenVZ without installing OpenVZ kernel. Any recent 3.x kernel should work.

As with OpenVZ kernel, you can use ready container images we have for OpenVZ (so called "OS templates") or employ your own. You can create, start, stop, and delete containers, set various resource parameters such as RAM and CPU limits. Networking (aside from routed-based) is also supported -- you can either move a network interface from host system to inside container (--netdev_add), or use bridged setup (--netif_add). I personally run this stuff on my Fedora 17 desktop using stock F17 kernel -- it just works!

Having said all that, surely OpenVZ kernel is in much better shape now as it comes for containers support -- it has more features (such as live container shapshots and live migration), better resource management capabilities, and overall is more stable and secure. But the fact that the kernel is now optional makes the whole thing more appealing (or so I hope).

You can find information on how to setup and start using vzctl at vzctl for upstream kernel wiki page. The page also lists known limitations are pointers to other resources. I definitely recommend you to give it a try and share your experience! As usual, any bugs found are to be reported to OpenVZ bugzilla.

Update: comments disabled due to spam

OpenVZ Container Migration Screencast

Wed, 29 Aug 2012 21:02:16 GMT

In the last week or so I've had several questions from new OpenVZ users about container migration. They want to know how it works, what is the difference between live / online migration vs. offline... and how long it will take. I made a silent screencast a few years ago but it is rather dated... and since I can easily add sound now, I thought I'd make an updated screencast. I didn't really put a lot of production value into it but overall it meets its objective. I posted it to archive.org as a webm but they automatically convert it to flash and embed it... so enjoy whichever format you prefer.

http://archive.org/details/OpenvzContainerMigration (archive.org page)
http://archive.org/download/OpenvzContainerMigration/openvz-vzmigrate.webm (direct link to webm)

[CRIU] CRtools 0.1 released!

Tue, 24 Jul 2012 13:48:59 GMT

Checkpoint/restore, or CPT for short, is a nice feature of OpenVZ, and probably the most amazing one. In a nutshell, it's a way to freeze a container and dump its complete state (processes, memory, network connections etc) to a file on disk, and then restore from that dump, resuming processes execution as if nothing happened. This opens a way to do nifty things such as live container migration, fast reboots, high availability setups etc.

It is our ultimate goal to merge all bits and pieces of OpenVZ to the mainstream Linux kernel. It's not a big secret that we failed miserably trying to merge the checkpoint/restore functionality (and yes, we have tried more than once). The fact that everyone else failed as well soothes the pain a bit, but is not really helpful. The reason is simple: CPT code is big, complex, and touches way too many places in the kernel.

So we^* came up with an idea to implement most of CPT stuff in user space, i.e. as a separate program not as a part of the Linux kernel. In practice this is impossible because some kernel trickery is still required here and there, but the whole point was to limit kernel intervention to the bare minimum.

Guess what? It worked even better that we expected. As of today, after about a year of development, up to 90% of the stuff that is needed to be in the kernel is already there, and the rest is ready and seems to be relatively easy to merge (see this table to get an idea what's in and what's not).

As for the user space stuff, we are happy to announce the release of CRtools version 0.1. Now, let me step aside and quote Pavel's announcement:

The tool can already be used for checkpointing and restoring various individual
applications. And the greatest thing about this so far is that most of the below
functionality has the required kernel support in the recently released v3.5!

So, we support now

* x86_64 architecture
* process' linkage
* process groups and sessions (without ttys though :\ )
* memory mappings of any kind (shared, file, etc.)
* threads
* open files (shared between tasks and partially opened-and-unlinked)
* pipes and fifos with data
* unix sockets with packet queues contents
* TCP and UDP sockets (TCP connections support exists, but needs polishing)
* inotifies, eventpoll and eventfd
* tasks' sigactions setup, credentials and itimers
* IPC, mount and PID namespaces

Though namespaces support is in there, we do not yet support an LXC container c/r,
but we're close to it :)

I'd like to thank everyone who took part in new kernel APIs discussions, the
feedback was great! Special thanks goes to Linus for letting the kernel parts
in early, instead of making them sit out of tree till becoming stable enough.

Tarball with the tool sources is at
http://download.openvz.org/criu/crtools-0.1.tar.bz2

The git repo is at
http://git.criu.org/

And some sort of docs growing at
http://criu.org/

There are still things for which we don't have the kernel support merged (SysVIPC
and various anon file descriptors, i.e. inotify, eventpoll, eventfd) yet. We have
the kernel branch with the stuff applied available at

https://github.com/cyrillos/linux-2.6.git

What's next? We will be rebasing OpenVZ to Linux Kernel 3.5 (most probably) and will try to re-use CRIU for checkpoint and restore of OpenVZ containers, effectively killing a huge chunk of out-of-tree kernel code that we have in OpenVZ kernel.

* - In fact it was Pavel Emelyanov, our chief kernel architect, but it feels oh so nice to say we that we can't refrain).

Update: comments disabled due to spam.

vzubc: better cat /proc/user_beancounters

Fri, 29 Jun 2012 12:25:25 GMT

Managing user beancounters is not for the faint of heart, I must say. One has to read through a lot of documentation and understand all this stuff pretty well. Despite the fact we have made a great improvement recently, a feature called VSwap, many people still rely on traditional beancounters.

This post is about a utility I initially wrote for myself in May 2011. Surely I have added it to vzctl, it is available since vzctl 3.0.27. Simply speaking, it is just a replacement for cat /proc/user_beancounters, and of course it can do much more than cat.

Here's a brief list of vzubc features:

Shows human-readable held, maxheld, barrier, limit, and fail counter for every beancounter, fitting into standard 80-columns terminal (unlike /proc/user_beancounters on an x86_64 system)
Values that are in pages (such as physpages) are converted to bytes
Long values are then converted to kilo-, mega-, gigabytes etc, similar to -h flag for ls or df
For held and maxheld, it shows how close the value to the barrier and the limit, in per cent
Can be used both inside CT and on HN
User can specify CTIDs or CT names to output info about specific containers only
Optional top-like autoupdate mode (internally using "watch" utility)
Optional "relative failcnt" mode (show increase in UBC fail counters since the last run
Optional quiet mode (only shows "worth to look at" UBCs, ie ones close to limits and/or with failcnt)

Now, this is how default vzubc output for a particular VE (with non-vswap configuration) looks like:

# vzubc 1009
----------------------------------------------------------------
CT 1009      | HELD Bar% Lim%| MAXH Bar% Lim%| BAR | LIM | FAIL
-------------+---------------+---------------+-----+-----+------
     kmemsize|1.52M  11%  10%|1.81M  13%  12%|13.7M|14.1M|    - 
  lockedpages|   -    -    - |   -    -    - |   8M|   8M|    - 
  privvmpages| 2.6M   1%   1%|4.32M   2%   2%| 256M| 272M|    - 
     shmpages|   -    -    - |   -    -    - |  84M|  84M|    - 
      numproc|  10    4%   4%|  15    6%   6%| 240 | 240 |    - 
    physpages|17.3M   -    - |18.6M   -    - |   - |   - |    - 
  vmguarpages|   -    -    - |   -    -    - | 132M|   - |    - 
 oomguarpages|1.52M   2%   - |1.52M   2%   - | 102M|   - |    - 
   numtcpsock|   2  0.6% 0.6%|   3  0.8% 0.8%| 360 | 360 |    - 
     numflock|   1  0.5% 0.5%|   2    1%   1%| 188 | 206 |    - 
       numpty|   -    -    - |   -    -    - |  16 |  16 |    - 
   numsiginfo|   -    -    - |   6    2%   2%| 256 | 256 |    - 
    tcpsndbuf|34.1K   2%   1%|51.1K   3%   2%|1.64M|2.58M|    - 
    tcprcvbuf|  32K   2%   1%|  48K   2%   2%|1.64M|2.58M|    - 
 othersockbuf|2.26K 0.2% 0.1%|14.6K   1% 0.7%|1.07M|   2M|    - 
  dgramrcvbuf|   -    -    - |   -    -    - | 256K| 256K|    - 
 numothersock|  44   12%  12%|  47   13%  13%| 360 | 360 |    - 
   dcachesize| 618K  18%  17%| 627K  18%  17%|3.25M|3.46M|    - 
      numfile| 114    1%   1%| 125    1%   1%|9.09K|9.09K|    - 
    numiptent|  20   15%  15%|  20   15%  15%| 128 | 128 |    - 
    swappages|   -    -    - |   -    -    - |   - |   - |    - 
----------------------------------------------------------------

As you can see, it shows all beancounters in human-readable form. Zero or unlimited values are shown by a dash. It also shows the ratio of held and maxheld to barrier and limit, in per cent.

Now, let's try to explore functionality available via command-line switches.

First of all, -q or --quiet enables quiet mode to only show beancounters with fails and those with held/maxheld values close to limits. If vzubc --q produces empty output, that probably means you don't have to worry about anything related to UBC. There are two built-in thresholds for quiet mode, one for barrier and one for limit. They can be changed to your liking using -qh and -qm options.

Next, -c or --color enables using colors to highlight interesting fields. In this mode "warnings" are shown in yellow, and "errors" are in red. Here a warning means a parameter close to limit (same thresholds are used as for the quiet mode), and an error means non-zero fail counter.

The following screenshot demonstrates the effect of -c and -q options. I have run a forkbomb inside CT to create a resource shortage:

Now, -r or --relative is the ultimate answer to the frequently asked "how do I reset failcnt?" question. Basically it saves the current failcnt value during the first run, and shows its delta (rather than the absolute value) during the next runs. This is how it works:

There is also -i or --incremental flag to activate a mode in which an additional column shows a difference in held value from the previous run, so you can see the change in usage. This option also affects quiet mode: all lines with changed held values are shown.

Here's a screenshot demonstrating the "color, relative, incremental, quiet" mode for vzubc:

Finally, you can use -w or --watch to enable a la top mode, to monitor beancounters. It's not as powerful as top, in fact it just uses watch(1) tool to run vzubc every 2 seconds and that's it. Please note that this mode is not compatible with --color, and you have to press Ctrl-C to quit. Oh, and since I am not a big fan of animated gifs, there will be no screenshot.

Man page vzubc(8) man page gives you more formal description of vzubc, including some minor options I haven't described here. Enjoy.

Update: comments disabled due to spam

Effective live migration with ploop write tracker

Sun, 03 Jun 2012 10:53:25 GMT

During my last holiday on the of hospitable Turkey sunny seaside at night, instead of quenching my thirst or taking rest after a long and tedious day at a beach, I was sitting in a hotel lobby, where they have free Wi-Fi, trying to make live migration of a container on a ploop device working. I succeeded, with about 20 commits to ploop and another 15 to vzctl, so now I'd like to share my findings and tell the story about it.

Let's start from the basics and see how migration (i.e. moving a container from one OpenVZ server to another) is implemented. It's vzmigrate, a shell script which does the following (simplified for clarity):

1. Checks that a destination server is available via ssh w/o entering a password, that there is no container with the same ID on it, and so on.
2. Runs rsync of /vz/private/$CTID to the destination server
3. Stops the container
4. Runs a second rsync of /vz/private/$CTID to the destination
5. Starts the container on the destination
6. Removes it locally

Obviously, two rsync runs are needed, so the first one moves most of the data, while the container is still up and running, and the second one moves the changes made during the time period between the first rsync run and the container stop.

Now, if we need live migration (option --online to vzmigrate), then instead of CT stop we do vzctl checkpoint, and instead of start we do vzctl restore. As a result, a container will move to another system without your users noticing (process are not stopping, just freezing for a few seconds, TCP connections are migrating, IP addresses are not changed etc. -- no cheating, just a little bit of magic).

So this is the way it was working for years, making users happy and singing in the rain. One fine day, though, ploop was introduced and it was soon discovered that live migration is not working for ploop-based containers. I found a few reasons why (for example, one can't use rsync --sparse for copying ploop images, because in-kernel ploop driver can't work with files having holes). But the main thing I found is the proper way of migrating a ploop image: not with rsync, but with ploop copy.

ploop copy is a mechanism of effective copy of a ploop image with the help of build-in ploop kernel driver feature called write tracker. One ploop copy process is reading blocks of data from ploop image and sends them to stdout (prepending each block with a short header consisting of a magic label, a block position and its size). The other ploop copy process receives this data from stdin and writes them down to disk. If you connect these two processes via a pipe, and add ssh $DEST in between, you are all set.

You can say, cat utility can do almost the same thing. Right. The difference is, before starting to read and send data, ploop copy asks the kernel to turn on the write tracker, and the kernel starts to memorize a list of data blocks that are modified (written into). Then, after all the blocks are sent, ploop copy is politely expressing the interest in this list, and send the blocks from the list again, while the kernel is creating another list. The process repeats a few times, and the list becomes shorter and shorter. After a few iterations (either the list is empty, or it is not getting shorter, or we just decide that we already did enough iterations) ploop copy executes an external command, which should stop any disk activity for this ploop device. This command is either vzctl stop for offline migration or vzctl checkpoint for live migration; obviously, stopped or frozen container will not write anything to disk. After that, ploop copy asks the kernel for the list of modified blocks again, transfers the blocks listed, and finally asks the kernel for this list again. If this time the list is not empty, that means something is very wrong, meaning that stopping command haven't done what it should and we fail. Otherwise all is good and ploop copy sends a marker telling the transfer is over. So this is how the sending
process works.

The receiving ploop copy process is trivial -- it just reads the blocks from stdin and writes those to file (to a specified position). If you want to see the code of both sending and receiving sides, look no further.

All right, in the previous migration description ploop copy is used in place of second rsync run (steps 3 and 4). I'd like to note that this way it's more effective, because rsync is trying to figure out which files have changed and where, while ploop copy just asks about it from the kernel. Also, because the "ask and send" process is iterative, container will be stopped or frozen as late as it can be and, even if container is writing data to disk actively, the period for which it is stopped is minimal.

Just out of pure curiosity I performed a quick non-scientific test, having "od -x /dev/urandom > file" running inside a container and live migrating it back and forth. ploop copy time after the freeze was a bit over 1 second, and the total frozen time a bit less than 3 seconds. Similar numbers can be obtained from the traditional simfs+rsync migration, but only if a container is not doing any significant I/O. Then I tried to migrate similar container on simfs running the same command running inside, and the frozen time increased to 13-16 seconds. I don't want to say these measurements are to be trusted, I just ran it without any precautions, with OpenVZ instances running inside Parallels VMs, with the physical server busy with something else...

Oh, the last thing. All this functionality is already included into latest tools releases: ploop 1.3 and vzctl 3.3.

Update: comments disabled due to spam

Ubuntu 12.04 templates available

Fri, 27 Apr 2012 00:39:50 GMT

I have just uploaded +Ubuntu 12.04 templates for OpenVZ to http://wiki.openvz.org/Download/template/precreated#Beta_templates

In case it doesn't work with vzctl-3.1 (haven't tried it, going to release 3.2 really soon), try the recent pre-3.2 vzctl nightly build, see more info at http://wiki.openvz.org/Download/vzctl/nightly

Enjoy =)

Update: comments disabled due to spam

OpenVZ containers

Tue, 03 Apr 2012 13:31:48 GMT

OpenVZ concept art by Andrew Vagin. Notice the high containers density!

[click for full size picture]

Update: comments disabled due to spam bots.

Announcing OpenVZ rebase to Windows

Sun, 01 Apr 2012 13:00:50 GMT

I am glad to officially announce that we decided to rebase OpenVZ from Linux to Windows. The main reason is of course Windows market share, but it's not the only reason. Windows is well-known for being user friendly, stable and secure operating system, very well supported by third-party vendors, and we would like to leverage on that and be a part of that huge and wonderful Windows ecosystem.

Has it ever come to your mind that while Linux guys are only blabbering about clouds, many huge Windows-based clouds (they call them botnets) are already fully operational? This is yet another indication of the platform superiority.

The project of rebasing is currently on its early stages, frankly speaking we haven't done much yet except for some preliminary research. But I have already designed the new logo, which I am proud to show to you today.

Logo design was tough because it should not be too different from the old one (after all, OpenVZ is still OpenVZ), but at the same time remind our users that this software is Windows-based. You can already enjoy the new logo and the tagline on http://wiki.openvz.org/.

As for the actual software, don't hold your breath, we will let you know than it will become available. But don't waste your time waiting! I suggest everyone to stop using Linux and start using Windows on their personal machines in order to familiarize with it. Me myself, I am planning to rm -rf / (or should I say delete *.* instead?) for both Fedora 16 on my notebook and the Gentoo Linux on my home desktop, and reinstall Windows 8!

Update: comments disabled due to excessive spam.

Introducing container in a file aka ploop

Tue, 27 Mar 2012 08:48:03 GMT

OpenVZ have just introduced kernel and tools support for container-in-a-file technology, also known as ploop. This post tries to summarize why ploop is needed, and why is it a superior technology to what we had before.

Before ploop: simfs and vzquota

First of all, a few facts about the pre-ploop era technologies and their limitations.

As you are probably aware, a container file system was just a directory on the host, which a new container was chroot()-ed into. Although it seems like a good and natural idea, there are a number of limitations.

Since containers are living on one same file system, they all share common properties of that file system (it's type, block size, and other options). That means we can not configure the above properties on a per-container basis.

One such property that deserves a special item in this list is file system journal. While journal is a good thing to have, because it helps to maintain file system integrity and improve reboot times (by eliminating fsck in many cases), it is also a bottleneck for containers. If one container will fill up in-memory journal (with lots of small operations leading to file metadata updates, e.g. file truncates), all the other containers I/O will block waiting for the journal to be written to disk. In some extreme cases we saw up to 15 seconds of such blockage.

Since many containers share the same file system with limited space, in order to limit containers disk space we had to develop per-directory disk quotas (i.e. vzquota).

Again, since many containers share the same file system, and the number of inodes on a file system is limited [for most file systems], vzquota should also be able to limit inodes on a per container (per directory) basis.

In order for in-container (aka second-level) disk quota (i.e. standard per-user and per-group UNIX dist quota) to work, we had to provide a dummy file system called simfs. Its sole purpose is to have a superblock which is needed for disk quota to work.

When doing a live migration without some sort of shared storage (like NAS or SAN), we sync the files to a destination system using rsync, which does the exact copy of all files, except that their i-node numbers on disk will change. If there are some apps that rely on files' i-node numbers being constant (which is normally the case), those apps are not surviving the migration.

Finally, a container backup or snapshot is harder to do because there is a lot of small files that need to be copied.

Introducing ploop

In order to address the above problems and ultimately make a world a better place, we decided to implement a container-in-a-file technology, not different from what various VM products are using, but working as effectively as all the other container bits and pieces in OpenVZ.

The main idea of ploop is to have an image file, use it as a block device, and create and use a file system on that device. Some readers will recognize that this is exactly what Linux loop device does! Right, the only thing is loop device is very inefficient (say, using it leads to double caching of data in memory) and its functionality is very limited.

Ploop implementation in the kernel have a modular and layered design.

The top layer is the main ploop module, which provides a virtual block device to be used for CT file system.

The middle layer is the image format module, which does translation of block device block numbers into image file block numbers. A simple format module which is called "raw" is doing trivial 1:1 translation, same as existing loop device.

More sophisticated format module is keeping the translation table and is able to dynamically grow and shrink the image file. That means, if you create a container with 2GB of disk space, the image file size will not be 2GB, but less -- the size of the actual data stored in the container. It is also possible to support other image formats by writing other ploop format modules, such as the one for QCOW2 (used by QEMU and KVM).

The bottom layer is the I/O module. Currently modules for direct I/O on an ext4 device, and for NFS are available. There are plans to also have a generic VFS module, which will be able to store images on any decent file system, but that needs an efficient direct I/O implementation in the VFS layer which is still being worked on.

Ploop benefits

In a nutshell:

File system journal is not bottleneck anymore
Large-size image files I/O instead of lots of small-size files I/O on management operations
Disk space quota can be implemented based on virtual device sizes; no need for per-directory quotas
Number of inodes doesn't have to be limited because this is not a shared resource anymore (each CT has its own file system)
Live backup is easy and consistent
Live migration is reliable and efficient
Different containers may use file systems of different types and properties

In addition:

Efficient container creation
[Potential] support for QCOW2 and other image formats
Support for different storage types

Update: comments are disabled due to spam.

CT console

Thu, 23 Feb 2012 14:28:15 GMT

Are you ready for the next cool feature? Please welcome CT console.

Available in RHEL6-based kernel since 042stab048.1, this feature is pretty simple to use. Use vzctl attach CTID to attach to this container's console, and you will be able to see all the messages CT init is writing to console, or run getty on it, or anything else.

Please note that the console is persistent, i.e. it is available even if a container is not running. That way, you can run ~~vzctl attach~~ vzctl consoleand then (in another terminal) vzctl start. That also means that if a container is stopped, vzctl attach is still there.

Press Esc . to detach from the console.

The feature (vzctl git commit) will be available in up-coming vzctl-3.0.31. I have just made a nightly build of vzctl (version 3.0.30.2-18.git.a1f523f) available so you can test this. Check http://wiki.openvz.org/Download/vzctl/nightly for information of how to get a nightly build.

Update: the feature is renamed to vzctl console.
Update: comments disabled due to spam.

Kir's OpenVZ presentation at FOSDEM 2012

Tue, 14 Feb 2012 19:21:30 GMT

Just got done watching the video recording of Kir's presentation from last Saturday at FOSDEM 2012. They recorded it and I'm writing this to share it with you too. Here's a link to the video in .webm format:

http://video.fosdem.org/2012/maintracks/janson/Linux_containers_and_OpenVZ.webm

Feel free to right click on the link and "Save link as..." to download it, or stream play it in your browser if desired since Firefox 4.x and up, as well as Chrome and Opera will play .webm directly. I prefer to download and watch with gmplayer so it plays more efficiently. You do have a video player that supports webm, right?

In the first half Kir talks generally about what containers are and what features OpenVZ has. In the second half of it he talks about recent features that are available now (vswap) and a few upcoming features (ploop and criu). Then he takes a number of questions from the audience. I really enjoyed the questions and the answers Kir gave. Some nice conversation on the "OpenVZ vs. LXC" concept.

SOPA protest

Wed, 18 Jan 2012 15:17:01 GMT

wiki.openvz.org is down today, joining the SOPA and PIPA protest. Read more:
* http://americancensorship.org/
* https://www.eff.org/deeplinks/2012/01/how-pipa-and-sopa-violate-white-house-principles-supporting-free-speech
* http://google.com/takeaction

The site will come alive tomorrow. Now, if you need access to the site, just reload the page.

Recent improvements in vzctl

Tue, 27 Dec 2011 16:03:39 GMT

Update: note you need VSwap enabled (ie currently RHEL6-based) kernel for the below stuff to work, see http://wiki.openvz.org/VSwap

VSwap is an excellent feature, simplifying container resource management a lot. Now it's time to also simplify the command line interface, i.e. vzctl. Here is what we did recently (take a look at vzctl git repo if you want to review the actual changes):

1. We no longer require to set kmemsize, dcachesize and lockedpages parameters to non-unlimited values (this is one of the enhancements in the recent kernels we have talked about recently). Therefore, setting these parameters to fractions of CT RAM (physpages) are now removed from configuration files and vzsplit output.

2. There is no longer a need to specify all the UBC parameters in per-container configuration file. If you leave some parameters unset, the kernel will use default values (usually unlimited). So, VSwap example configs are now much smaller, with as much as 19 parameters removed from those.

3. vzctl set now supports two new options: --ram and --swap. These are just convenient short aliases for --physpages and --swappages, the differences being that you only need to specify one value (the limit) and the argument is in bytes rather than pages.

So, to configure a container named MyCT to have 1.5GB of RAM and 3GB of swap space available, all you need to do is just run this command:
vzctl set MyCT --ram 1.5G --swap 3G --save

4. This is not VSwap-related, but nevertheless worth sharing. Let's illustrate it by a copy-paste from a terminal:

# vzctl create 123 --ostemplate centos-4-x86_64
Creating container private area (centos-4-x86_64)
Found centos-4-x86_64.tar.gz at http://download.openvz.org/template/precreated//centos-4-x86_64.tar.gz
Downloading...
-- 2011-11-29 18:54:08 -- http://download.openvz.org/template/precreated//centos-4-x86_64.tar.gz
Resolving download.openvz.org... 64.131.90.11
Connecting to download.openvz.org|64.131.90.11|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 171979832 (164M) [application/x-gzip]
Saving to: `/vz/template/cache/centos-4-x86_64.tar.gz'

100%[======================================>] 171,979,832 588K/s in 4m 27s 

2011-11-29 18:58:35 (630 KB/s) - `/vz/template/cache/centos-4-x86_64.tar.gz' saved [171979832/171979832]

Success!
Performing postcreate actions
Saved parameters for CT 123
Container private area was created

All this will be available in vzctl-3.0.30, which is to be released soon (next week? who knows). If you can't wait and want to test this stuff, here's a nightly build of vzctl available (version 3.0.29.3-27.git.0535fe1) from http://wiki.openvz.org/Download/vzctl/nightly. Please give it a try and tell me what you think.

Update: comments disabled due to spam

On vSwap and 042stab04x kernel improvements

Mon, 14 Nov 2011 20:03:12 GMT

vSwap

The best feature of the new (RHEL6-based) 042 series of the OpenVZ kernels is definitely vSwap. The short story is, we used to have 22 user beancounter parameters which every seasoned OpenVZ user knows by heart. Each of these parameters is there for a reason, but 22 knobs are a bit too complex to manage for a mere mortal, especially bearing in mind that

many of them are interdependent;
the sum of all limits should not exceed the resources of a given physical server.

Keeping this configuration optimal (or even consistent) is quite a challenging task even for a senior OpenVZ admin (with a probable exception of an ex airline pilot). This complexity is the main reason why there are multiple articles and blog entries complaining OpenVZ is worse than Xen, or that it is not suitable for hosting Java apps. We do have some workarounds to mitigate this complexity, such as:

precreated container configs with sane defaults for beancounters;
some special tools (vzsplit and vzcfgvalidate);
the comprehensive User Beancounters manual.

This is still not the way to go. While we think high of our users, we do not expect all of them to be ex airline pilots. To solve the complexity, the number of per-container knobs and handles should be reduced to some decent number, or at least most of these knobs should be optional.

We worked on that for a few years, and the end result is called vSwap (where V is for Vendetta, oh, pardon me, Virtual).

vSwap concept is as simple as a rectangle. For each container, there are only two required parameters: the memory size (known as physpages) and the swap size (swappages). Almost everyone (not only an admin, but even an advanced end user) knows what is RAM and what is swap. On a physical server, if there is not enough memory, the system starts to swap out memory pages to disk, then swap in some other pages, which results in severe performance degradation but it keeps the system from failing miserably.

It's about the same with vSwap, except that

RAM and swap are configured on a per container basis;
no I/O is performed until it is really necessary (this is why swap is virtual).

Some VSwap internals

Now, there are only two knobs per container on a dashboard, namely RAM and swap, and all the complexity is hidden under the hood. I am going to describe just a bit of that undercover mechanics and explain what does the "Reworked VSwap kernel memory accounting" line from the 042stab040.1 kernel changelog stands for.

The biggest problem is, RAM for containers is not just RAM. First of all, there is a need to distinguish between

the user memory,
the kernel memory,
the page cache,
and the directory entry cache.

The user memory is more or less clear, it is simply the memory that programs allocate for themselves to run. It is relatively easy to account for, and it is relatively simple to limit it (but read on).

The kernel memory is really complex thingie. Right, it is the memory that kernel allocates for itself in order for programs in a particular container to run. This includes a lot of stuff I'd rather not dive into, if I want to keep this piece as an article not a tome. Having said that, two particular kernel memory types are worth explaining.

First is the page cache, the kernel mechanism that caches disk contents in memory (that would be unused otherwise) to minimize the I/O. When a program reads some data from a disk, that data are read into the page cache first, and when a program writes to a disk, data goes to the page cache (and then eventually are written (flushed) to disk). In case of repeated disk access (which happens quite often) data is taken from a page cache, not from the real disk, which greatly improves the overall system performance, since a disk is much slower than RAM. Now, some of the page cache is used on behalf of a container, and this amount must be charged into "RAM used by this container" (i.e. physpages).

Second is the directory entry cache (dcache for short) is yet another sort of cache, and another sort of the kernel memory. Disk contents is a tree of files and directories, and such a tree is quite tall and wide. In order to read the contents of, say, /bin/sh file, kernel have to read the root (/) directory, find 'bin' entry in it, read /bin directory, find 'sh' entry in it and finally read it. Although these operations are not very complex, there is a multitude of those, they take time and are repeated often for most of the "popular" files. In order to improve performance, kernel keeps directory entries in memory — this is what dcache is for. The memory used by dcache should also be accounted and limited, since otherwise it's easily exploitable (not only by root, but also by an ordinary user, since any user is free to change into directories and read files).

Now, the physical memory of a container is the sum of its user memory, the kernel memory, the page cache and the dcache. Technically, dcache is accounted into the kernel memory, then kernel memory is accounted into the physical memory, but it's not overly important.

Improvements in the new 042stab04x kernels

Better reclamation and memory balancing

What to do if a container hit a physical memory limit? Free some pages by writing their contents to the abovementioned virtual swap. Well, not quite yet. Remember that there is also a page cache and a dcache, so the kernel can easily discard some of the pages from these caches, which is way cheaper than swapping out.

The process of finding some free memory is known as reclamation. Kernel needs to decide very carefully when to start reclamation, how many and what exact pages to reclaim in every particular situation, and when it is the right time to swap out rather than discard some of the cache contents.

Remember, we have four types of memory (kernel, user, dcache and page cache) and only one knob which limits the sum of all these. It would be easier for the kernel, but not for the user, to have separate limits for each type of memory. But, for the user convenience and simplicity, the kernel only have one knob for these four parameters, so it needs to balance between those four. One major improvement in 042stab040 kernel is that such balancing is now performed better.

Stricter memory limit

During the lifetime of a container, the kernel might face a situation when it needs more kernel memory, or user memory, or perhaps more dcache entries, and the memory for the container is tight (i.e. close to the limit), so it needs to either reclaim or swap. The problem is there are some situations when neither reclamation nor swapping is possible, so the kernel can either fail miserably (say by killing a process) or go beyond the limit and hope that everything will be fine and mommy won't notice. Another big improvement in 042stab040 kernel is it reduces the number of such situations, in other words, the new kernel obeys memory limit in a more strict way.

Polishing

Finally, the kernel is now in a pretty good shape, so we can afford some polishing, minor optimizations, and fine tuning. Such polishing was performed in a few subsystems, including checkpointing, user beancounters, netfilter, kernel NFS server and VZ disk quota.

Some numbers

Totally, there are 53 new patches in 042stab040.1, compared to previous 039 kernels. On top of that, 042stab042.1 adds another 30. We hope that the end result is improved stability and performance.

Announcing rhel6-testing kernel branch/repo

Fri, 14 Oct 2011 21:40:02 GMT

Instead of having a nice drink in a bar, I spent this Friday night splitting the RHEL6-based OpenVZ kernel branch/repository into two, so now we have 'rhel6' and 'rhel6-testing' branches/repos. Let me explain why.

When we made an initial port of OpenVZ to RHEL6 kernel and released the first kernel (in October 2010, named 042test001), I created a repository named openvz-kernel-rhel6 (or just rhel6), and this repository was marked as "development, unstable". When, after almost a year, we announced it as "testing" and then, finally, "stable" (in August 2011, named 042stab035.1).

After that, all the kernels in that repository were supposed to be stable, because they are incremental improvements of the kernel we call stable. In theory it is. In practice, of course, there can always be new bugs (both introduced by us and by Red Hat folks releasing their kernel updates which we rebase to). Thus a kernel update from a repo which is supposed to be stable can do bad things.

Better late than never, I have fixed the situation tonight by basically renaming "rhel6" repository into "rhel6-testing", and creating a new repository called just "rhel6". For now, I put 042stab037.1 (which is the latest kernel which has passed our internal QA) into rhel6 (aka stable), while all the other kernels, up to and including 042stab039.3, are in rhel6-testing repo.

Now, very similar to what we do with RHEL5 kernels, all the new fresh-from-the-build-farm kernels will appear in rhel6-testing repo, at about the same time they go to internal QA. Then, the kernels which will have QA approval will appear in rhel6 (aka -stable) repo. What it means for you as a user is you can now choose whether to stay at the bleeding edge and have the latest stuff, or to take a conservative approach and have less frequent and delayed updates, but be more confident about kernel quality and stability.

A few links:
* Stable RHEL6-based OpenVZ kernels
* Testing RHEL6-based OpenVZ kernels
* OpenVZ yum repository setup file
* Official announce of rhel6-testing

Update: comments disabled due to spam.

LinuxCon Europe 2011 in Prague is coming

Fri, 30 Sep 2011 11:26:07 GMT

And we are coming to Prague, too! This time, there will be as many as six people and two talks from us, plus we will held a memory cgroup controller meeting.

The following OpenVZ/Parallels people are coming:

James Bottomley, Parallels virtualization CTO
Kir Kolyshkin, OpenVZ project manager
Pavel Emelyanov, OpenVZ kernel team leader (he's also taking part in Linux Kernel Summit)
Glauber Costa, OpenVZ kernel developer
Maxim Patlasov, OpenVZ kernel developer
Andrey Vagin, OpenVZ kernel developer

Two talks will be presented. Since linuxsymposium.org site is currently down, let me quote talk descriptions here.

1. Container in a file by Maxim Patlasov.

One of the feature differences between hypervisors and containers is the ability to store a virtual machine image in a single file, since most containers exist as a chroot within the host OS rather than as fully independent entities. However, the ability to save and restore state in a machine image file is invaluable in managing virtual machine life cycles in the data centre.

This talk will début a new loopback device which gives all the advantages of virtual machine images by storing the container in a file while preserving the benefits of sharing significant portions with the host OS. We will compare and contrast the technology with the traditional loopback device, and describe some changes to the ext4 filesystem which make it more friendly to new loopback device needs.

This talk will be technical in nature but should be accessible to people interested in cloud, virtualisation and container technologies.

2. OpenVZ and Linux kernel testing by Andrey Vagin.

One of the less appealing but very important part of software development is testing. This talk tries to summarize our 10+ years of experience in Linux kernel testing (including OpenVZ and Red Hat Enterprise Linux kernels). Overall description of our test system is provided, followed by details on some of the interesting test cases developed. Finally, a few anecdotal cases of bugs found will be presented.

In a sense, the talk is an answer to Andrew Morton's question from 2007: "I'm curious. For the past few months, people@openvz.org have discovered (and fixed) an ongoing stream of obscure but serious and quite long-standing bugs. How are you discovering these bugs?"

Talk is of interest to those concerned about kernel quality, and in general to people doing development and testing.

Finally, there will be a memcg meeting. Since LinuxCon will be right after the Kernel Summit, a number of kernel guys will still be there so anyone interested in cgroups can come. This meeting is a continuation of our recent discussion at Linux Plumbers (see etherpad and presentations).

See you all in Prague in less than a month!

RHEL6 goes stable!

Tue, 30 Aug 2011 17:22:43 GMT

Guys, I am very proud to inform you that today we mark RHEL6 kernel branch as stable. Below is a copy-paste from the relevant announce@ post. I personally highly recommend RHEL6-based OpenVZ kernel to everyone -- it is a major step forward compared to RHEL5.

In the other news, Parallels has just released Virtuozzo Containers for Linux 4.7, bringing the same cool stuff (VSwap et al) to commercial customers. Despite being only a "dot" (or "minor") release, this product incorporates an impressive amount of man-hours of best Parallels engineers.

== Stable: RHEL6 ==

This is to announce that RHEL6-based kernel branch (starting from kernel 042stab035.1) is now marked as stable, and it is now the recommended branch to use.

We are not aware of any major bugs or show-stoppers in this kernel. As always, we still recommend to test any new kernels before rolling out to production.

New features of RHEL6-based kernel branch (as compared to previous stable kernel branch, RHEL5) includes better performance, better scalability (especially on high-end SMP systems), and better resource management (notably, vSwap support -- see http://wiki.openvz.org/VSwap).

RHEL6 kernels can be downloaded from http://wiki.openvz.org/Download/kernel/rhel6

== Frozen: 2.6.27, 2.6.32 ==

Also, from now we no longer maintain the following kernel branches:

* 2.6.27
* 2.6.32

No more new releases of the above kernels are expected. Existing users (if any) are recommended to switch to other (maintained) branches, such as RHEL6-2.6.32 or RHEL5-2.6.18.

This change does not affect vendor OpenVZ kernels (such as Debian or Ubuntu) -- those will still be supported for the lifetime of their distributions via the usual means (i.e. bugzilla.openvz.org).

== Development: none ==

Currently, there are no non-stable kernels in development. Eventually we will port to Linux kernel 3.x, but it might not happen this year. Instead, we are currently focused on bringing more of OpenVZ features to mainstream Linux kernels.

Regards, OpenVZ team.

Update: comments disabled due to spam

Linux Plumbers: Containers and CGroups miniconf

Wed, 24 Aug 2011 15:16:32 GMT

We have finally filed a number of proposals for the up-coming Containers and CGroups miniconf to be held during Linux Plumbers Conference, 7 to 9 September 2011 in Santa Rosa, CA.

From those proposals, one can clearly see what are our plans regarding the mainline integration. In a few words: dcache management, memory and CPU cgroup controllers improvements, container enter, improved /proc virtualization, checkpoint/restart [mostly] in userspace (of which I have blogged recently), and making vzctl work with mainline kernel containers. Oh, and the interesting loopback-like block device to hold a container filesystem (a.k.a. ploop).

Quite a lot of interesting stuff, what do you think?

Checkpoint/restart (mostly) in user space

Sat, 13 Aug 2011 09:26:08 GMT

There is a good article at lwn.net telling about one of our latest development.

We have checkpoint/restart (CPT) and live migration in OpenVZ for ages (well, OK, since 2007 or so), allowing for containers to be freely moved between physical servers without any service interruption. It is a great feature which is valued by our users. The problem is we can't merge it upstream, ie to vanilla kernel.

Various people from our team worked on that, and they all gave up. Then, Oren Laadan was trying very hard to merge his CPT implementation -- unfortunately it didn't worked out very well either. The thing is, checkpointing is a complex thing, and the patch implementing it is very intrusive.

Recently, our kernel team leader Pavel Emelyanov got a new idea of moving most of the checkpointing complexity out of the kernel and into user space, thus minimizing the amount of the in-kernel changes needed. In about two weeks of time he wrote a working prototype. So far the reaction is mostly positive, and he's going to submit a second RFC version for review to lkml.

For more details, read the lwn.net article. After all, while I am sitting next to Pavel, Mr. Corbet ability to explain complex stuff in simple terms is way better than mine.

ioping

Thu, 02 Jun 2011 18:01:18 GMT

My colleague koct9i, whose daily job is developing and fixing OpenVZ kernel, was feeling bored last weekend, and to entertain himself he wrote a small utility called ioping. The idea is simple and straightforward: to wrote a utility similar to ping, which will show I/O latency in the same way ping shows network latency. The idea is very simple but I haven't see something like this. Actually, the tool was written to help investigating OpenVZ bug #1880).

I liked ioping and worked on it a bit, too, just for run. Among some other minor stuff I have added a man page and spec file, so it is now available as an RPM package.

Official project site: http://code.google.com/p/ioping/

My RPM packages and stuff: http://kir.sacred.ru/ioping/