For the last two weeks or so we've been working on vzstats -- a way to get some statistics about OpenVZ usage. The system consists of a server, deployed to http://stats.openvz.org/, and clients installed onto OpenVZ machines (hardware nodes). This is currently in beta testing, with 71 servers participating at the moment. If you want to participate, read http://openvz.org/vzstats and run
So far we have some interesting results. We are not sure how representative they are -- probably they aren't, much more servers are needed to participate-- but nevertheless they are interested. Let's share a few preliminary findings.
First, it looks like almost no one is using 32-bits on the host system anymore. This is reasonable and expected. Indeed, who needs system limited to 4GB of RAM nowdays?
Second, many hosts stay on latest stable RHEL6-based OpenVZ kernel. This is pretty good and above our expectations.
Third, very few run ploop-based containers. We don't understand why. Maybe we should write more about features you get from ploop, such as instant snapshots and improved live migration.
yum install vzstats on your OpenVZ boxes.So far we have some interesting results. We are not sure how representative they are -- probably they aren't, much more servers are needed to participate-- but nevertheless they are interested. Let's share a few preliminary findings.
First, it looks like almost no one is using 32-bits on the host system anymore. This is reasonable and expected. Indeed, who needs system limited to 4GB of RAM nowdays?
Second, many hosts stay on latest stable RHEL6-based OpenVZ kernel. This is pretty good and above our expectations.
Third, very few run ploop-based containers. We don't understand why. Maybe we should write more about features you get from ploop, such as instant snapshots and improved live migration.


Comments
We don't understand why we should use it.
And they don't look ready for production.
For reasons why, http://openvz.org/Ploop/Why
- Easy backup, when you just need to backup some files out of the container.
- Copy files from one container to another (HN perspective).
- Just need one Samba on HN to share files inside containers.
- Find the container hosting virtualhost: find /vz/private/*/var/www -type d www.something\*
For all containers at once, ignoring errors (such as "container already mounted"):
for CT in $(vzlist -1a); do vzctl mount $CT 2>/dev/null; doneThen if you don't want them mounted, unmount them (again ignoring errors such as "container is running"):
for CT in $(vzlist -1a); do vzctl umount $CT 2>/dev/null; doneEdited at 2013-04-29 08:52 pm (UTC)
vzlist -1a | xargs -n1 vzctl mount 2>/dev/nullJust use --diskspace as you did for simfs+vzquota case.
http://openvz.org/Ploop/Getting_started#Resizing_a_ploop_image
I guess I will just need to experiment with this to better understand how it works.
But there is one more issue... Currently, for backups, we rsync the entire /vz/private folder to a remote location. This not only allows us to save i/o (we only need to copy changed files), but also perform differential backups and save tons of disk space on the backup server. I always assumed this becomes impossible with ploop. Am I missing anything here?
Basically, a backup is:
- create snapshot
- copy this snapshot
- merge this snapshot down
If you don't do merge, you can have real differential backups, just by copying the next snapshot.
Also, see vzctl man page (http://wiki.openvz.org/Man/vzctl.8#Snapshotting) on available snapshot commands. In addition,
vzctl 4.3 will add snapshot-mount and snapshot-umount commands.
Re: ploop, I can see the benefit of having a FS journal per container. Do you have any tips on monitoring the journal blocking to gauge how big of a problem it is for my setup?
Is there a tool to convert a directory based container into a ploop container? That might help in transitioning existing containers (it wouldn't be too hard to work out myself, but if you want people to move over to it, making it as easy as possible will encourage migration).
Yes, it is vzctl convert $CTID. Please set desired --diskspace before doing that.
It is documented in vzctl(8) man page and also here:
http://wiki.openvz.org/Ploop/Getting_started#Converting_an_existing_CT
Now we use next sheme - rsync working container to another host, then stop it.
The rsync again (take a few time) and then start new container at new place.
Its requered reboot container.
Ploop, as i understand, - stop container when snapshot is creating.
And while ploop copyng snapshot to another host - we have downtime.
Also you need additional work and time to find anything in containers (for example - what files in container eats more space).
After reading it there some new questions:
1. If i'm migrate database production server - there database use much RAM (for example 15GB active RAM ), before switchig process to new place (with, as i understand, vzmigrate --online command ) Openvz kernel must copy active(and inactive?) RAM to another machine? And during this operation processes in old place can not switch to new place? And container in old place not working?
In this case - if vzmigrate can provide the same downtime as old rsync way - it would be perfect.
Here meminfo for example from one our host:
root@:~# cat /proc/meminfo
MemTotal: 22634676 kB
MemFree: 409080 kB
Buffers: 141496 kB
Cached: 10861520 kB
SwapCached: 21796 kB
Active: 15006620 kB
Inactive: 6718756 kB
Active(anon): 9677768 kB
Inactive(anon): 1059764 kB
Active(file): 5328852 kB
Inactive(file): 5658992 kB
Unevictable: 0 kB
........
2. If i remember Vmware have mechanism (called "quiesced io") which can reduce disk io usage when container is backuping. Vzmigrate have the same mechanism? If we migrate machine - with high load disk subsystem - may happen situation when blocks in kernel list have no time to copy in new place?
Thanks a lot for you articles and comments again.
But yes, currently while RAM is being migrated the processes are in frozen state.
You can decrease CT I/O priority (vzctl set $CTID --ioprio NNN), but vzmigrate doesn't do that.
No, this is impossible. We always copy everything, with the last round when CT is frozen.
So (because CRUI with kernel 3.9, as i understand, not stable yet; and we use non-commercial free Openvz) in our case - rsyncing containers without ploop still the best way to copy high load containers.
Only because you do not recomend (http://openvz.org/Ploop) to use its in production. But if its ready I will try.
But I'll try to install that tool. Not using ploop though as being a bit lazy.
Same for vzctl, vzquota and ploop. Oh.
We've been reading about ploop since its first release but also though that it wasn't read for production yet.
Also, based on our reads, ploop would make migrating nodes (with big files that change a lot) faster. That's pretty interesting.
if ready, maybe its time to start "playing" with it.
Congratulations for the good work!
Because it very well may not be mature enough and we don't want to find out in production.
We got burnt very badly with initial instabilities of the RHEL6 kernel (that we however much needed to upgrade to because of other bugs in previous vanilla branches that were not going to get fixed) and particularly its vswap feature that we promised to our customer to use in their VMs in late 2011 (too early, I now know). We ran every RHEL6 stable update since (30 days before kernel crash was often considered success) and only in August 2012 it became fully stable for us with our workload (another site we manage heavily uses NFS, which was another point of frequent breakage in the kernel). We've reported some issues, some got fixed, eventually we're to the point in the past almost a year when 100+ days of uptime are achieved predictably, i.e. the kernel is actually *stable* from my perspective. I'm so glad we've eventually reached this point, so there is no way we'd use ploop in 2013 I'm afraid.. maybe next year :)
Added a node to vzstats, good idea.