Log in

Previous Entry | Next Entry

vzstats in beta

For the last two weeks or so we've been working on vzstats -- a way to get some statistics about OpenVZ usage. The system consists of a server, deployed to http://stats.openvz.org/, and clients installed onto OpenVZ machines (hardware nodes). This is currently in beta testing, with 71 servers participating at the moment. If you want to participate, read http://openvz.org/vzstats and run yum install vzstats on your OpenVZ boxes.

So far we have some interesting results. We are not sure how representative they are -- probably they aren't, much more servers are needed to participate-- but nevertheless they are interested. Let's share a few preliminary findings.

First, it looks like almost no one is using 32-bits on the host system anymore. This is reasonable and expected. Indeed, who needs system limited to 4GB of RAM nowdays?

Second, many hosts stay on latest stable RHEL6-based OpenVZ kernel. This is pretty good and above our expectations.

Third, very few run ploop-based containers. We don't understand why. Maybe we should write more about features you get from ploop, such as instant snapshots and improved live migration.


( 28 comments — Leave a comment )
Apr. 29th, 2013 07:51 pm (UTC)
> very few run ploop-based containers. We don't understand why

We don't understand why we should use it.

And they don't look ready for production.
Apr. 29th, 2013 08:46 pm (UTC)
They are ready for production, and are used by lot of commercial customers.

For reasons why, http://openvz.org/Ploop/Why
Apr. 29th, 2013 08:21 pm (UTC)
> Third, very few run ploop-based containers. We don't understand why.

- Easy backup, when you just need to backup some files out of the container.
- Copy files from one container to another (HN perspective).
- Just need one Samba on HN to share files inside containers.
- Find the container hosting virtualhost: find /vz/private/*/var/www -type d www.something\*
Apr. 29th, 2013 08:52 pm (UTC)
Just run "vzctl mount $CTID" and get/put those files from/to /vz/root/$CTID

For all containers at once, ignoring errors (such as "container already mounted"):

for CT in $(vzlist -1a); do vzctl mount $CT 2>/dev/null; done

Then if you don't want them mounted, unmount them (again ignoring errors such as "container is running"):

for CT in $(vzlist -1a); do vzctl umount $CT 2>/dev/null; done

Edited at 2013-04-29 08:52 pm (UTC)
Apr. 29th, 2013 08:54 pm (UTC)
Alternative syntax using xargs:

vzlist -1a | xargs -n1 vzctl mount 2>/dev/null
Apr. 30th, 2013 12:20 am (UTC)
Is there currently a way to resize a ploop container?
Apr. 30th, 2013 02:41 am (UTC)
There was always a way, from the very first release.
Just use --diskspace as you did for simfs+vzquota case.

Apr. 30th, 2013 03:33 am (UTC)
Wow, thanks! For some reason I thought ploop is more like KVM or vmware disk images where you had to deal with partition resize headaches, etc. But then how do you deal with image fragmentation? What happens when I downsize a heavily fragmented container?

I guess I will just need to experiment with this to better understand how it works.

But there is one more issue... Currently, for backups, we rsync the entire /vz/private folder to a remote location. This not only allows us to save i/o (we only need to copy changed files), but also perform differential backups and save tons of disk space on the backup server. I always assumed this becomes impossible with ploop. Am I missing anything here?
May. 3rd, 2013 01:37 am (UTC)
With ploop, you can make consistent backups using ploop snapshot feature. See http://wiki.openvz.org/images/f/f3/Ct_in_a_file.pdf presentation, a section about snapshots and backups.

Basically, a backup is:
- create snapshot
- copy this snapshot
- merge this snapshot down

If you don't do merge, you can have real differential backups, just by copying the next snapshot.

Also, see vzctl man page (http://wiki.openvz.org/Man/vzctl.8#Snapshotting) on available snapshot commands. In addition,
vzctl 4.3 will add snapshot-mount and snapshot-umount commands.
Apr. 30th, 2013 03:29 am (UTC)
Just installed on a SL6 base node I have at home.

Re: ploop, I can see the benefit of having a FS journal per container. Do you have any tips on monitoring the journal blocking to gauge how big of a problem it is for my setup?

Is there a tool to convert a directory based container into a ploop container? That might help in transitioning existing containers (it wouldn't be too hard to work out myself, but if you want people to move over to it, making it as easy as possible will encourage migration).
May. 3rd, 2013 01:32 am (UTC)
> Is there a tool to convert a directory based container into a ploop container?

Yes, it is vzctl convert $CTID. Please set desired --diskspace before doing that.

It is documented in vzctl(8) man page and also here:

Apr. 30th, 2013 06:54 am (UTC)
With ploop migrating container - take a long time.
Now we use next sheme - rsync working container to another host, then stop it.
The rsync again (take a few time) and then start new container at new place.
Its requered reboot container.

Ploop, as i understand, - stop container when snapshot is creating.
And while ploop copyng snapshot to another host - we have downtime.

Also you need additional work and time to find anything in containers (for example - what files in container eats more space).
May. 3rd, 2013 01:31 am (UTC)
Jun. 11th, 2013 11:59 am (UTC)
Thank you for article about ploop! Its really helpful.

After reading it there some new questions:

1. If i'm migrate database production server - there database use much RAM (for example 15GB active RAM ), before switchig process to new place (with, as i understand, vzmigrate --online command ) Openvz kernel must copy active(and inactive?) RAM to another machine? And during this operation processes in old place can not switch to new place? And container in old place not working?

In this case - if vzmigrate can provide the same downtime as old rsync way - it would be perfect.

Here meminfo for example from one our host:

root@:~# cat /proc/meminfo
MemTotal: 22634676 kB
MemFree: 409080 kB
Buffers: 141496 kB
Cached: 10861520 kB
SwapCached: 21796 kB
Active: 15006620 kB
Inactive: 6718756 kB
Active(anon): 9677768 kB
Inactive(anon): 1059764 kB
Active(file): 5328852 kB
Inactive(file): 5658992 kB
Unevictable: 0 kB

2. If i remember Vmware have mechanism (called "quiesced io") which can reduce disk io usage when container is backuping. Vzmigrate have the same mechanism? If we migrate machine - with high load disk subsystem - may happen situation when blocks in kernel list have no time to copy in new place?

Thanks a lot for you articles and comments again.
Jun. 11th, 2013 10:16 pm (UTC)
1. We don't have iterative memory migration in OpenVZ yet, but we do have it in Virtuozzo (== commercial OpenVZ). More to say, we have it in CRIU (the future of OpenVZ checkpointing, to be used with kernels 3.x). See http://criu.org/Memory_changes_tracking and http://criu.org/Iterative_migration for more details if you are curious.

But yes, currently while RAM is being migrated the processes are in frozen state.

mechanism (called "quiesced io") which can reduce disk io usage when container is backuping

You can decrease CT I/O priority (vzctl set $CTID --ioprio NNN), but vzmigrate doesn't do that.

If we migrate machine - with high load disk subsystem - may happen situation when blocks in kernel list have no time to copy in new place?

No, this is impossible. We always copy everything, with the last round when CT is frozen.
Jun. 13th, 2013 02:55 pm (UTC)

So (because CRUI with kernel 3.9, as i understand, not stable yet; and we use non-commercial free Openvz) in our case - rsyncing containers without ploop still the best way to copy high load containers.
Dmitry Kopytov
Apr. 30th, 2013 01:20 pm (UTC)
> We don't understand why

Only because you do not recomend (http://openvz.org/Ploop) to use its in production. But if its ready I will try.
May. 3rd, 2013 01:27 am (UTC)
thanks for letting know, I removed the warning -- ploop is considered stable now.
Apr. 30th, 2013 09:07 pm (UTC)
Every time I see "yum " I can't be master of myself and start crying :(

But I'll try to install that tool. Not using ploop though as being a bit lazy.

May. 1st, 2013 12:35 am (UTC)
Wrote email into openvz users list with initial debianization.
May. 3rd, 2013 01:28 am (UTC)
I will try my best to provide deb packages as well )

Same for vzctl, vzquota and ploop. Oh.
May. 1st, 2013 02:19 pm (UTC)
Hello from Brazil..

We've been reading about ploop since its first release but also though that it wasn't read for production yet.

Also, based on our reads, ploop would make migrating nodes (with big files that change a lot) faster. That's pretty interesting.

if ready, maybe its time to start "playing" with it.

Congratulations for the good work!
May. 3rd, 2013 01:29 am (UTC)
Yes it's ready; I have removed the "warning it's beta" notice from the wiki )
May. 2nd, 2013 01:48 pm (UTC)
> very few run ploop-based containers. We don't understand why

Because it very well may not be mature enough and we don't want to find out in production.

We got burnt very badly with initial instabilities of the RHEL6 kernel (that we however much needed to upgrade to because of other bugs in previous vanilla branches that were not going to get fixed) and particularly its vswap feature that we promised to our customer to use in their VMs in late 2011 (too early, I now know). We ran every RHEL6 stable update since (30 days before kernel crash was often considered success) and only in August 2012 it became fully stable for us with our workload (another site we manage heavily uses NFS, which was another point of frequent breakage in the kernel). We've reported some issues, some got fixed, eventually we're to the point in the past almost a year when 100+ days of uptime are achieved predictably, i.e. the kernel is actually *stable* from my perspective. I'm so glad we've eventually reached this point, so there is no way we'd use ploop in 2013 I'm afraid.. maybe next year :)

Added a node to vzstats, good idea.
May. 3rd, 2013 01:29 am (UTC)
We consider ploop ready for production. Of course we might be wrong and as always regular backups are recommended... )
May. 6th, 2013 06:02 pm (UTC)
What about ploop qcow2 driver - is there any plans to implement it?
May. 8th, 2013 01:53 am (UTC)
When Solus supports ploop I will use it. So far they haven't expressed any interest.
May. 9th, 2013 08:25 pm (UTC)
Because ploop based containers need fully new approach to VPS deployment (OS reinstall indeed) and backup. But main issue in migration to ploop for us is VDSmanager, it's very strange management panel and they developers do not planned ploop integration into panel.

( 28 comments — Leave a comment )

Latest Month

September 2015
Powered by LiveJournal.com
Designed by Tiffany Chow