vzstats in beta 29 April 2013 @ 11:52 am
Kir Kolyshkin
k001
[openvz]
For the last two weeks or so we've been working on vzstats -- a way to get some statistics about OpenVZ usage. The system consists of a server, deployed to http://stats.openvz.org/, and clients installed onto OpenVZ machines (hardware nodes). This is currently in beta testing, with 71 servers participating at the moment. If you want to participate, read http://openvz.org/vzstats and run yum install vzstats on your OpenVZ boxes.

So far we have some interesting results. We are not sure how representative they are -- probably they aren't, much more servers are needed to participate-- but nevertheless they are interested. Let's share a few preliminary findings.

First, it looks like almost no one is using 32-bits on the host system anymore. This is reasonable and expected. Indeed, who needs system limited to 4GB of RAM nowdays?

Second, many hosts stay on latest stable RHEL6-based OpenVZ kernel. This is pretty good and above our expectations.

Third, very few run ploop-based containers. We don't understand why. Maybe we should write more about features you get from ploop, such as instant snapshots and improved live migration.
Permanent Link28 comments | Leave a comment
Post a new comment
Nick Knutov: crocodileknutov on April 29th, 2013 - 07:51 pm
> very few run ploop-based containers. We don't understand why

We don't understand why we should use it.

And they don't look ready for production.
Kir Kolyshkin: nepalk001 on April 29th, 2013 - 08:46 pm
They are ready for production, and are used by lot of commercial customers.

For reasons why, http://openvz.org/Ploop/Why
(Anonymous) on April 29th, 2013 - 08:21 pm
> Third, very few run ploop-based containers. We don't understand why.

- Easy backup, when you just need to backup some files out of the container.
- Copy files from one container to another (HN perspective).
- Just need one Samba on HN to share files inside containers.
- Find the container hosting virtualhost: find /vz/private/*/var/www -type d www.something\*
Kir Kolyshkin: nepalk001 on April 29th, 2013 - 08:52 pm
Just run "vzctl mount $CTID" and get/put those files from/to /vz/root/$CTID

For all containers at once, ignoring errors (such as "container already mounted"):

for CT in $(vzlist -1a); do vzctl mount $CT 2>/dev/null; done

Then if you don't want them mounted, unmount them (again ignoring errors such as "container is running"):

for CT in $(vzlist -1a); do vzctl umount $CT 2>/dev/null; done





Kir Kolyshkin: nepalk001 on April 29th, 2013 - 08:54 pm
Alternative syntax using xargs:

vzlist -1a | xargs -n1 vzctl mount 2>/dev/null
Дмитрий Постригань: snailearly_morning on April 30th, 2013 - 12:20 am
Is there currently a way to resize a ploop container?
Kir Kolyshkin: nepalk001 on April 30th, 2013 - 02:41 am
There was always a way, from the very first release.
Just use --diskspace as you did for simfs+vzquota case.

http://openvz.org/Ploop/Getting_started#Resizing_a_ploop_image
Дмитрий Постригань: snailearly_morning on April 30th, 2013 - 03:33 am
Wow, thanks! For some reason I thought ploop is more like KVM or vmware disk images where you had to deal with partition resize headaches, etc. But then how do you deal with image fragmentation? What happens when I downsize a heavily fragmented container?

I guess I will just need to experiment with this to better understand how it works.

But there is one more issue... Currently, for backups, we rsync the entire /vz/private folder to a remote location. This not only allows us to save i/o (we only need to copy changed files), but also perform differential backups and save tons of disk space on the backup server. I always assumed this becomes impossible with ploop. Am I missing anything here?
Kir Kolyshkin: nepalk001 on May 3rd, 2013 - 01:37 am
With ploop, you can make consistent backups using ploop snapshot feature. See http://wiki.openvz.org/images/f/f3/Ct_in_a_file.pdf presentation, a section about snapshots and backups.

Basically, a backup is:
- create snapshot
- copy this snapshot
- merge this snapshot down

If you don't do merge, you can have real differential backups, just by copying the next snapshot.

Also, see vzctl man page (http://wiki.openvz.org/Man/vzctl.8#Snapshotting) on available snapshot commands. In addition,
vzctl 4.3 will add snapshot-mount and snapshot-umount commands.
(Anonymous) on April 30th, 2013 - 03:29 am
Just installed on a SL6 base node I have at home.

Re: ploop, I can see the benefit of having a FS journal per container. Do you have any tips on monitoring the journal blocking to gauge how big of a problem it is for my setup?

Is there a tool to convert a directory based container into a ploop container? That might help in transitioning existing containers (it wouldn't be too hard to work out myself, but if you want people to move over to it, making it as easy as possible will encourage migration).
Kir Kolyshkin: nepalk001 on May 3rd, 2013 - 01:32 am
> Is there a tool to convert a directory based container into a ploop container?

Yes, it is vzctl convert $CTID. Please set desired --diskspace before doing that.

It is documented in vzctl(8) man page and also here:

http://wiki.openvz.org/Ploop/Getting_started#Converting_an_existing_CT
домовойkyzia on April 30th, 2013 - 06:54 am
With ploop migrating container - take a long time.
Now we use next sheme - rsync working container to another host, then stop it.
The rsync again (take a few time) and then start new container at new place.
Its requered reboot container.

Ploop, as i understand, - stop container when snapshot is creating.
And while ploop copyng snapshot to another host - we have downtime.

Also you need additional work and time to find anything in containers (for example - what files in container eats more space).
Kir Kolyshkin: nepalk001 on May 3rd, 2013 - 01:31 am
домовой: указатель ветраkyzia on June 11th, 2013 - 11:59 am
Thank you for article about ploop! Its really helpful.

After reading it there some new questions:

1. If i'm migrate database production server - there database use much RAM (for example 15GB active RAM ), before switchig process to new place (with, as i understand, vzmigrate --online command ) Openvz kernel must copy active(and inactive?) RAM to another machine? And during this operation processes in old place can not switch to new place? And container in old place not working?

In this case - if vzmigrate can provide the same downtime as old rsync way - it would be perfect.

Here meminfo for example from one our host:

root@:~# cat /proc/meminfo
MemTotal: 22634676 kB
MemFree: 409080 kB
Buffers: 141496 kB
Cached: 10861520 kB
SwapCached: 21796 kB
Active: 15006620 kB
Inactive: 6718756 kB
Active(anon): 9677768 kB
Inactive(anon): 1059764 kB
Active(file): 5328852 kB
Inactive(file): 5658992 kB
Unevictable: 0 kB
........


2. If i remember Vmware have mechanism (called "quiesced io") which can reduce disk io usage when container is backuping. Vzmigrate have the same mechanism? If we migrate machine - with high load disk subsystem - may happen situation when blocks in kernel list have no time to copy in new place?

Thanks a lot for you articles and comments again.
Kir Kolyshkin: nepalk001 on June 11th, 2013 - 10:16 pm
1. We don't have iterative memory migration in OpenVZ yet, but we do have it in Virtuozzo (== commercial OpenVZ). More to say, we have it in CRIU (the future of OpenVZ checkpointing, to be used with kernels 3.x). See http://criu.org/Memory_changes_tracking and http://criu.org/Iterative_migration for more details if you are curious.

But yes, currently while RAM is being migrated the processes are in frozen state.

mechanism (called "quiesced io") which can reduce disk io usage when container is backuping


You can decrease CT I/O priority (vzctl set $CTID --ioprio NNN), but vzmigrate doesn't do that.

If we migrate machine - with high load disk subsystem - may happen situation when blocks in kernel list have no time to copy in new place?


No, this is impossible. We always copy everything, with the last round when CT is frozen.
домовой: указатель ветраkyzia on June 13th, 2013 - 02:55 pm
Thanks!

So (because CRUI with kernel 3.9, as i understand, not stable yet; and we use non-commercial free Openvz) in our case - rsyncing containers without ploop still the best way to copy high load containers.
Dmitry Kopytov on April 30th, 2013 - 01:20 pm
> We don't understand why

Only because you do not recomend (http://openvz.org/Ploop) to use its in production. But if its ready I will try.
Kir Kolyshkin: nepalk001 on May 3rd, 2013 - 01:27 am
thanks for letting know, I removed the warning -- ploop is considered stable now.
CoolCold: pic#23115555coolcold on April 30th, 2013 - 09:07 pm
Every time I see "yum " I can't be master of myself and start crying :(

But I'll try to install that tool. Not using ploop though as being a bit lazy.



CoolCold: pic#23115555coolcold on May 1st, 2013 - 12:35 am
Wrote email into openvz users list with initial debianization.
Kir Kolyshkin: nepalk001 on May 3rd, 2013 - 01:28 am
I will try my best to provide deb packages as well )

Same for vzctl, vzquota and ploop. Oh.
(Anonymous) on May 1st, 2013 - 02:19 pm
Hello from Brazil..

We've been reading about ploop since its first release but also though that it wasn't read for production yet.

Also, based on our reads, ploop would make migrating nodes (with big files that change a lot) faster. That's pretty interesting.

if ready, maybe its time to start "playing" with it.

Congratulations for the good work!
Kir Kolyshkin: nepalk001 on May 3rd, 2013 - 01:29 am
Yes it's ready; I have removed the "warning it's beta" notice from the wiki )
(Anonymous) on May 2nd, 2013 - 01:48 pm
> very few run ploop-based containers. We don't understand why

Because it very well may not be mature enough and we don't want to find out in production.

We got burnt very badly with initial instabilities of the RHEL6 kernel (that we however much needed to upgrade to because of other bugs in previous vanilla branches that were not going to get fixed) and particularly its vswap feature that we promised to our customer to use in their VMs in late 2011 (too early, I now know). We ran every RHEL6 stable update since (30 days before kernel crash was often considered success) and only in August 2012 it became fully stable for us with our workload (another site we manage heavily uses NFS, which was another point of frequent breakage in the kernel). We've reported some issues, some got fixed, eventually we're to the point in the past almost a year when 100+ days of uptime are achieved predictably, i.e. the kernel is actually *stable* from my perspective. I'm so glad we've eventually reached this point, so there is no way we'd use ploop in 2013 I'm afraid.. maybe next year :)

Added a node to vzstats, good idea.
Kir Kolyshkin: nepalk001 on May 3rd, 2013 - 01:29 am
We consider ploop ready for production. Of course we might be wrong and as always regular backups are recommended... )
(Anonymous) on May 6th, 2013 - 06:02 pm
What about ploop qcow2 driver - is there any plans to implement it?
(Anonymous) on May 8th, 2013 - 01:53 am
When Solus supports ploop I will use it. So far they haven't expressed any interest.
pavel_odintsov on May 9th, 2013 - 08:25 pm
Because ploop based containers need fully new approach to VPS deployment (OS reinstall indeed) and backup. But main issue in migration to ploop for us is VDSmanager, it's very strange management panel and they developers do not planned ploop integration into panel.