Donation: a way to help or just say thanks 09 July 2014 @ 01:05 pm
Kir Kolyshkin

For many years, when people asked us how can they help the project, our typical answer was something like "just use it, file bugs, spread the word". Some people were asking specifically about how to donate money, and we had to say "currently we don't have a way to accept", so it was basically "no, we don't need your money". It was not a good answer. Even if our big sponsor is generously helping us with everything we need, that doesn't mean your money would be useless.

Today we have opened a PayPal account to accept donations. Here:

How are we going to spend your money? In general:

  • Hardware for development and testing
  • Travel budget for conferences, events etc.
  • Accolades for active contributors

In particular, right now we need:

  • About $200 to cover shipping expenses for donated hardware
  • About $300 for network equipment
  • About $2000 for hard drives

Donations page is created on our wiki to track donations and spending. We hope that it will see some good progress in the coming days, with a little help from you!

NOTE that if you feel like spending $500 or more, there is yet another way to spend it -- a support contract from Parallels, with access to their excellent support team and 10 free support tickets.

Permanent Link2 comments | Leave a comment
Yet more live migration goodness 12 June 2014 @ 04:35 pm
Kir Kolyshkin

It's been two months since we have released vzctl 4.7 and ploop 1.11 with some vital improvements to live migration that made it about 25% faster. Believe it or not, this is another "make it faster" post, making it, well, faster. That's right, we have more surprises in store for you! Read on and be delighted.

Asynchronous ploop send

This is something that should have been implemented at the very beginning, but the initial implementation looked like this (in C):

/* _XXX_ We should use AIO. ploopcopy cannot use cached reads and
 * has to use O_DIRECT, which introduces large read latencies.
 * AIO is necessary to transfer with maximal speed.

Why there are no cached reads (and, in general, why ploop library is using O_DIRECT, and the kernel also uses direct I/O, bypassing the cache)? Note that ploop images are files that are used as block devices containing a filesystem. That ploop block device and filesystem access is going through cache as usual, so if we'd do the same for ploop image itself, it would result in double caching and waste of RAM. Therefore, we do direct I/O on lower level (when working with ploop images), and allow usual Linux cache to be used on the upper level (when container is accessing files inside the ploop, which is the most common operation).

So, ploop image itself is not (usually) cached in memory, and other tricks like read-ahead are also not used by the kernel, and reading each block from the image takes time as it is read directly from disk. In our test lab (OpenVZ running inside KVM with a networked disk) it takes about 10 milliseconds to read each 1 MB block. Then, sending each block to the destination system over network also takes time, it's about 15 milliseconds in our setup. So, it takes 25 milliseconds to read and send a block of data, if we do it one after another. Oh wait, this is exactly how we do it!

Solution -- let's do reading and sending at the same time! In other words, while sending the first block of data we can already read the second one, and so on, bringing the time required to transfer one block down to MAX(Tread, Tsend) instead of Tread + Tsend.

Implementation uses POSIX threads. Main ploop send process spawns a separate sending thread and two buffers -- one for reading and one for sending, then they change place. This works surprisingly well for something that uses threads, maybe because it is as simple as it could ever be (one thread, one mutex, one conditional). If you happen to know something about pthreads programming, please review the appropriate commit 55e26e9606.

The new async send is used by default, you just need to have a newer ploop (1.12, not yet released). Clone and compile ploop from git if you are curious.

As for how much time we save using this, it happen to be 15 microseconds instead of 25 per block of data, or 40% faster! Now, the real savings depend on the number of blocks needs to be migrated after container is frozen, it can be 5 or 100, so overall savings can be from 0.05 to 1 second.

ploop copy with feedback

The previous post on live migration improvements described an optimization of doing fdatasync() on the receiving ploop side before suspending the container on the sending side. It also noted that implementation is sub-par:

The other problem is that sending side should wait for fsync to finish, in order to proceed with CT suspend. Unfortunately, there is no way to solve this one with a one-way pipe, so the sending side just waits for a few seconds. Ugly as it is, this is the best way possible (let us know if you can think of something better).

So, the problem is trivial -- there's a need for a bi-directional channel between ploop copy sender and receiver, so the receiver can say "All right, I have synced the freaking delta back to sender". In addition, we want to do it in a simple way, not much more complicated as it is now.

After some playing around with different approaches, it seemed that OpenSSH port forwarding, combined with tools like netcat, socat, or bash /dev/tcp feature, can do the trick of establishing a two-way pipe between ploop copy sides.

The netcat (or nc) is available in various varieties, which might or might not be available and/or compatible. socat is a bit better, but one problem with it is it ignores its child exit code, so there is no (simple) way to figure out an error. A problem with /dev/tcp feature is it is bash-specific, but bash itself might not be universally available (Debian and Ubunty users are well aware of that fact).

So, all this was replaced with a tiny home-grown vznnc ("nano-netcat) tool. It is indeed nano, as there is only only 200 lines of code in it. It can either listen or connect to a specified port at localhost (note that ssh is doing real networking for us), and run a program with its stdin/stdout (or a specified file descriptor) already connected to that TCP socket. Again, similar to netcat, but small, to the point, and hopefully bug-free.

Finally, with this in place, we can make sending side of ploop copy to wait for feedback from the receiving side, so it can suspend the container as soon as remote side finished syncing. This makes the whole migration a bit faster (by eliminating the previously hard-coded 5 seconds wait-for-sync delay), but it also helps to slash the frozen time as we suspend the container as soon as we should, so it won't be given any extra time to write some more data we'll be needing to copy while it's frozen.

It's hard to measure the practical impact of the feature, but in our tests it saves about 3.5 seconds of total migration time, and from 0 to a few seconds of frozen time, depending on container's disk I/O activity.

For the feature to work, you need the latest versions of vzctl (4.8) and ploop (1.12) on both nodes (if one node doesn't have newer tools, vzmigrate falls back to old non-feedback way of copying). Note that those new version are not yet released at the time of writing this, but you can get ones from git and compile yourself.

Using ssh connection multiplexing

It takes about 0.15 seconds to establish an ssh connection in our test lab. Your mileage may wary, but it can't be zero. Well, unless an OpenSSH feature of reusing an existing ssh connection is put to work! It's call connection miltiplexing, and when used, once a connection is established, you can have subsequent ones in practically no time. As vzmigrate is a shell script and runs a number of commands at remote (destination) side, using it might save us some time.

Unfortunately, it's a relatively new OpenSSH feature and is implemented quite ugly in a version available in CentOS 6 -- you need to keep one open ssh session running. They fixed it in a later version by adding a special daemon-like mode enabled with ControlPersist option, but alas not in CentOS 6. Therefore, we have to maintain a special "master" ssh connection for the duration of vzmigrate. For implementation details, see commits 06212ea3d and 00b9ce043.

This is still experimental, so you need to specify --ssh-mux flag to vzmigrate to use it. You won't believe it (so, go test it yourself!), but this alone slashes container frozen time by about 25% (which is great, as we fight for every microsecond here), and improves the total time taken by vzmigrate by up to 1 second (which is probably not that important but still nice).

What's next?

Current setup used for development and testing is two OpenVZ instances running in KVM guests on a ThinkCentre box. While it's convenient and oh so very space-saving, is is probably not the best approximation of a real world OpenVZ usage. So, we need some better hardware:

If you are able to spend $500 to $2000 on ebay, please let us know by email to donate at openvz dot org so we can arrange it. Now, quite a few people offered hosted servers with a similar configuration. While we are very thankful to all such offers, this time we are looking for physical, not hosted, hardware.

Update: Thanks to FastVPS, we got the Supermicro 4 node server. If you want to donate, see wiki: Donations.
Permanent Link3 comments | Leave a comment
ploop and live migration: 2 years later 03 April 2014 @ 07:45 pm
Kir Kolyshkin
It has been almost two years since we wrote about effective live migration with ploop write tracker. It's time to write some more about it, since we have managed to make ploop live migration yet more effective, by means of pretty simple optimizations. But let's not jump to resolution yet, it's a long and interesting story to tell.

As you know, live migration is not quite live, although it looks that way to a user. There is a short time period, usually a few seconds, during which the container being migrated is frozen. This time (shown if -t or -v option to vzmigrate --live is used) is what needs to be optimized, making it as short as possible. In order to do that, one needs to dig into details on what's happening when a container is frozen.

Typical timings obtained via vzmigrate -t --live look like this. We ran a few iterations migrating a container back and forth between two OpenVZ instances (running inside Parallels VMs on the same physical machine), so there are a few columns at the right side.

(Software: vzctl 4.6.1, ploop-1.10)

              Iteration     1     2     3     4     5     6    AVG

        Suspend + Dump:   0.58  0.71  0.64  0.64  0.91  0.74  0.703
   Pcopy after suspend:   1.06  0.71  0.50  0.68  1.07  1.29  0.885
        Copy dump file:   0.64  0.67  0.44  0.51  0.43  0.50  0.532
       Undump + Resume:   2.04  2.06  1.94  3.16  2.26  2.32  2.297
                        ------  ----  ----  ----  ----  ----  -----
  Total suspended time:   4.33  4.16  3.54  5.01  4.68  4.85  4.428

Apparently, the first suspect to look at is that "undump + resume". Basically, it shows timing of vzctl restore command. Why it is so slow? Apparently, ploop mount takes some noticeable time. Let's dig deeper into that process.

First, implement timestamps in ploop messages, raise the log level and see what is going on here. Apparently, adding deltas is not instant, takes any time from 0.1 second to almost a second. After some more experiments and thinking it becomes obvious that since ploop kernel driver works with data in delta files directly, bypassing the page cache, it needs to force those files to be written to the disk, and this costly operation happens while container is frozen. Is it possible to do it earlier? Sure, we just need to force write the deltas we just copied before suspending a container. Easy, just call fsync(), or yet better fdatasync(), since we don't really care about metadata being written.

Unfortunately, there is no command line tool to do fsync or fdatasync, so we had to write one and call it from vzmigrate. Is it any better now? Yes indeed, delta adding times went down to from tenths to hundredths of a second.

Except for the top delta, of course, which we migrate using ploop copy. Surely, we can't fsync it before suspending container, because we keep copying it after. Oh wait... actually we can! By adding an fsync before CT suspend, we force the data be written on disk, so the second fsync (which happens after everything is copied) will take less time. This time is shown as "Pcopy after suspend".

The problem is that ploop copy consists of two sides -- the sending one and the receiving one -- which communicate over a pipe (with ssh as a transport). It's the sending side which runs the command to freeze the container, and it's the receiving side which should do fsync, so we need to pass some sort of "do the fsync" command. Yet better, do it without breaking the existing protocol, so nothing bad will happen in case there is an older version of ploop on the receiving side.

The "do the fsync" command ended up being a data block of 4 bytes, you can see the patch here. Older version will write these 4 bytes to disk, which is unnecessary but OK do to, and newer version will recognize it as a need to do fsync.

The other problem is that sending side should wait for fsync to finish, in order to proceed with CT suspend. Unfortunately, there is no way to solve this one with a one-way pipe, so the sending side just waits for a few seconds. Ugly as it is, this is the best way possible (let us know if you can think of something better).

To summarize, what we have added is a couple of fsyncs (it's actually fdatasync() since it is faster), and here are the results:

(Software: vzctl 4.7, ploop-1.11)

              Iteration     1     2     3     4     5     6    AVG

        Suspend + Dump:   0.60  0.60  0.57  0.74  0.59  0.80  0.650
   Pcopy after suspend:   0.41  0.45  0.45  0.49  0.40  0.42  0.437 (-0.4)
        Copy dump file:   0.46  0.44  0.43  0.48  0.47  0.52  0.467
       Undump + Resume:   1.86  1.75  1.67  1.91  1.87  1.84  1.817 (-0.5)
                        ------  ----  ----  ----  ----  ----  -----
  Total suspended time:   3.33  3.24  3.12  3.63  3.35  3.59  3.377 (-1.1)

As you see, both "pcopy after suspend" and "undump + resume" times decreased, shaving off about a second of time, which gives us about 25% improvement. Now, take into account that tests were done on an otherwise idle nodes with mostly idle containers, we suspect that the benefit will be more apparent with I/O load. Let checking if this statement is true will be your homework for today!

Permanent Link3 comments | Leave a comment
How about an OpenVZ CentOS Variant? 03 April 2014 @ 09:54 am
I've used RHEL, CentOS and Fedora for many years... and as many of you already know... back in January, CentOS became a sponsored project of Red Hat. For the upcoming CentOS 7 release they are going beyond just the normal release that is an as-perfect-as-possible clone of RHEL. They have this concept of variants... where Special Interest Groups (SIGs) are formed around making special purpose builds of CentOS... spins or remixs if you will. I don't know a lot about it yet but I think I have the basic concept correct.

Looking at the numbers on I see:

Top host distros
CentOS 56,725
Scientific 2,471
RHEL 869
Debian 576
Fedora 111
Ubuntu 82
Gentoo 54
openSUS 18
ALT Linux 10
Sabayon 6


Top 10 CT distros
centos 245,468
debian 106,350
ubuntu 83,197
OR 8,354
gentoo 7,017
pagoda 4,024
scientific 3,604
fedora 3,173
seedunlimited 1,965

Although reporting is optional, the popularity of CentOS as both an OpenVZ host and an OpenVZ container surely has to do with the fact that the two stable branches of the OpenVZ kernel are derived from RHEL kernels.

Wouldn't be nice if there were a CentOS variant that has the OpenVZ kernel and utils pre-installed? I think so.

While I have made CentOS remixes in the past just for my own personal use... I have not had any official engagement with the CentOS community. I was curious if there were some OpenVZ users out there who are already affiliated with the CentOS Project and who might want to get together in an effort to start a SIG and ultimately an OpenVZ CentOS 7 variant. Anyone? I guess if not, I could make a personal goal of building a CentOS and/or Scientific Linux 6-based remix that includes OpenVZ... as well as working on it after RHEL7 and clones are released... and after such time the OpenVZ Project has released a stable branch based on the RHEL7 kernel.

I will acknowledge up front that some of the top CentOS devs / contributors have historically been fairly nasty to OpenVZ users on the #centos IRC channel. They generally did not want to help someone using a CentOS system running under an OpenVZ kernel... but then again... their reputation is for being obnoxious to many groups of people. :) I don't think we should let that stop us.

Comments, feedback, questions?
Tags: ,
Permanent Link2 comments | Leave a comment
N problems of Linux Containers 05 March 2014 @ 12:22 pm
Kir Kolyshkin
Seven problems with Linux containers is a detailed description of the talk I gave at Southern California Linux Expo (aka SCALE) last week in Los Angeles. Couldn't have written it better myself!
Permanent LinkLeave a comment
CRIU hangout on air 05 February 2014 @ 10:11 am
Kir Kolyshkin
In a light of CRIU 1.1 release that happened last week, we will be doing a hangout on air to tell more about CRIU past and the future, and will be answering your questions. Event page is here, it is going to happen as soon as this Friday, Feb 7th, at 6:00am PST / 9:00am EST / 15:00 CET / 18:00 MSK. Feel free to ask your questions now (go to event page and click on "Play").
Permanent LinkLeave a comment
Why we still use gzip for templates? 28 December 2013 @ 01:47 am
Kir Kolyshkin
OpenVZ precreated OS templates are tarballs of a pre-installed Linux distributions. While there are other ways to create a container, the easiest one is to take such a tarball and extract its contents. This is what takes 99.9% of vzctl create command execution.

To save some space and improve download speeds, those tarballs are compacted with good ol' gzip tool. For example, CentOS 6 template tar.gz is about 200 MB in size, while uncompacted tar would be about 550 MB. But why don't we use more efficient compacting tools, such as bzip2 or xz? Say, the same CentOS 6 tarball, compressed by xz, is as lightweight as 120 MB! Here are the numbers again:

centos-6-x86.tar.gz: 203M
centos-6-x86.tar.xz: 122M
centos-6-x86.tar: 554M

So, why don't we switch to xz which apparently looks way better? Well, there are other criteria to optimize for, except for file size and download speed. In fact, the main optimization target is container creation speed! I just ran a quick non-scientific test on my notebook in order to proof my words, measuring the time it takes to run tar xf on a tarball:

time tar xf tar.gz: ~7 seconds
time tar xf tar.xz: ~13 seconds

See, it takes twice the time if we switch to xz! Note that this ratio doesn't change much when I switched from fast SSD to (relatively slow) rotating hard disk drive:

time tar xf tar.gz: ~8 seconds
time tar xf tar.xz: ~16 seconds

Note, while I call it non-scientific, I still ran each test at least three times, with proper syncs, rms and cache drops in between.

Now, do we want to trade a double increase of container creation time for saving 80 MB of disk space. We sure don't!
Permanent Link10 comments | Leave a comment
Video: Containers and the Cloud 20 November 2013 @ 09:13 pm
James Bottomley, CTO, Server Virtualization for Parallels gave a presentation entitled, "Containers and The Cloud: Do you need another virtual environment?" on Oct 23. The Linux Foundation recently posted it to YouTube.

There is a lot of good information in the video even for us OpenVZ folks. Enjoy:

Permanent Link1 comment | Leave a comment
vzctl 4.6 07 November 2013 @ 10:42 am
Kir Kolyshkin
vzctl 4.6 build hit (and its many mirrors around the world) last week. Let's see what's in store, shall we?

First and foremost, I/O limits, but the feature is already described in great details. What I want to add is the feature was sponsored by GleSYS Internet Services AB, and is one of the first results of OpenVZ partnership program in action. This program is a great opportunity for you to help keep the project up and running, and also experience our expert support service just in case you'd need it. Think of it as a two-way support option. Anyway, I digress. What was it about? Oh yes, vzctl.

Second, improvements to UBC setting in VSwap mode. Previously, if you set RAM and swap, all other UBC parameters not set explicitly were set to unlimited. Now they are just left unset (meaning that the default in-kernel setting is used, whatever it is). Plus, in addition to physpages and swappages, vzctl sets lockedpages and oomguarpages (to RAM), vmguarpages (to RAM+swap).

Plus, there is a new parameter vm_overcommit, and it works in the following way -- if set, it is used as a multiplier to ram+swap to set privvmpages. In layman terms, this is a ratio of real memory (ram+swap) to virtual memory (privvmpages). Again, physpages limits RAM, and physpages+swappages limits real memory used by a container. On the other side, privvmpages limits memory allocated by a container. While it depends on the application, generally not all allocated memory is used -- sometimes allocated memory is 5 or 10 times more than used memory. What vm_overcommit gives is a way to set this gap. For example, command

vzctl set $CTID --ram 2G --swap 4G --vm_overcommit 3 --save

is telling OpenVZ to limit container $CTID with 2 GB of RAM, 4 GB of swap, and (2+4)*3, i.e. 18 GB of virtual memory. That means this container can allocate up to 18 GB of memory, but can't use more than 6 GB. So, vm_overcommit is just a way to set privvmpages implicitly, as a function of physpages and swappages. Oh, and if you are lost in all those *pages, we have extensive documentation at

A first version of vzoversell utility is added. This is a proposed vzmemcheck replacement for VSwap mode. Currently it just summarizes RAM and swap limits for all VSwap containers, and compares it to RAM and swap available on the host. Surely you can oversell RAM (as long as you have enough swap), but sum of all RAM+swap limits should not exceed RAM+swap on the node, and the main purpose of this utility is to check that constraint.

vztmpl-dl got a new --list-orphans option. It lists all local templates that are not available from the download server(s) (and therefore can't be updated by vztmpl-dl). Oh, by the way, since vzctl 4.5 you can use vztmpl-dl --update-all to refresh all templates (i.e. download an updated template, if a newer version is available from a download server). For more details, see vztmpl-dl(8) man page.

vzubc got some love, too. It now skips unlimited UBCs by default, in order to improve the signal to noise ratio. If you want old behavior, (i.e. all UBCs), use -v flag.

Surely, there's a bunch of other fixes and improvements, please read the changelog if you want to know it all. One thing in particular worth mentioning, it's a hack to vzctl console. As you might know, in OpenVZ container's console is sort of eternal, meaning you can attach to it before a container is even started, and it keeps its state even if you detach from it. That creates a minor problem, though -- if someone run, say, vim in console, then detaches and reattaches, vim is not redrawing anything and the console shows nothing. To workaround, one needs to press Ctrl-L (it is also recognized by bash and other software). But it's a bit inconvenient to do that every time after reattach. Although, this is not required if terminal size has changed (i.e. you detach from console, change your xterm size, then run vzctl console again), because in this case vim is noting the change and redraws accordingly. So what vzctl now does after reattach is telling the underlying terminal its size twice -- first the wrong size (with incremented number of rows), then the right size (the one of the terminal vzctl is running in). This forces vim (or whatever is running on container console) to redraw.

Finally, new vzctl (as well as other utilities) are now in our Debian wheezy repo at, so Debian users are now on par with those using RPM-based distros, and can have latest and greatest stuff as soon as it comes out. Same that we did with kernels some time ago.

Enjoy, and don't forget to report bugs to
Permanent LinkLeave a comment
Yay to I/O limits! 29 October 2013 @ 08:12 pm
Kir Kolyshkin

Today we are releasing a somewhat small but very important OpenVZ feature: per-container disk I/O bandwidth and IOPS limiting.

OpenVZ have I/O priority feature for a while, which lets one set a per-container I/O priority -- a number from 0 to 7. This is working in a way that if two similar containers with similar I/O patterns, but different I/O priorities are run on the same system, a container with a prio of 0 (lowest) will have I/O speed of about 2-3 times less than that of a container with a prio of 7 (highest). This works for some scenarios, but not all.

So, I/O bandwidth limiting was introduced in Parallels Cloud Server, and as of today is available in OpenVZ as well. Using the feature is very easy: you set a limit for a container (in megabytes per second), and watch it obeying the limit. For example, here I try doing I/O without any limit set first:

root@host# vzctl enter 777
root@CT:/# cat /dev/urandom | pv -c - >/bigfile
 88MB 0:00:10 [8.26MB/s] [         <=>      ]

Now let's set the I/O limit to 3 MB/s:

root@host# vzctl set 777 --iolimit 3M --save
UB limits were set successfully
Setting iolimit: 3145728 bytes/sec
CT configuration saved to /etc/vz/conf/777.conf
root@host# vzctl enter 777
root@CT:/# cat /dev/urandom | pv -c - >/bigfile3
39.1MB 0:00:10 [   3MB/s] [         <=>     ]

If you run it yourself, you'll notice a spike of speed at the beginning, and then it goes down to the limit. This is so-called burstable limit working, it allows a container to over-use its limit (up to 3x) for a short time.

In the above example we tested writes. Reads work the same way, except when read data are in fact coming from the page cache (such as when you are reading the file which you just wrote). In this case, no actual I/O is performed, therefore no limiting.

Second feature is I/O operations per second, or just IOPS limit. For more info on what is IOPS, go read the linked Wikipedia article -- all I can say here is for traditional rotating disks the hardware capabilities are pretty limited (75 to 150 IOPS is a good guess, or 200 if you have high-end server class HDDs), while for SSDs this is much less of a problem. IOPS limit is set in the same way as iolimit (vzctl set $CTID --iopslimit NN --save), although measuring its impact is more tricky.</o>

Finally, to play with this stuff, you need:

  • vzctl 4.6 (or higher)
  • Kernel 042stab084.3 (or higher)
Note that the kernel with this feature is currently still in testing -- so if you haven't done so, it's time to read about testing kernels.
Permanent Link15 comments | Leave a comment