Top.Mail.Ru
? ?

Entries by tag: linux

An interview with ANK

This is a rare interview with the legendary Alexey Kuznetsov (a.k.a. ANK), who happen to work for Parallels. Alan Cox once said he had thought for a long time that "Kuznetsov" is a collective name for a secret group of Russian programmers -- because no single man can write so much code at once.

An interview is taken by lifehacker.ru and is part of "work places" series. I tried to do my best to translate it to English, but it's far from perfect. I guess this is still a very interesting reading.




Q: Who are you and what you do?

Since mid-90s I was one of Linux maintainers. Back then all the communication was done via conferences and Linux mailing lists. Pretty often I was aggressively arguing with someone there, don't remember for which reasons. Now it's fun to recall. Being a maintainer, I wasn't just making something on my own, but had to control others. Kicking out those who were making rubbish (from my point of view), and supporting those who were making something non-rubbish. All these conflicts, they were exhausting me. Since some moment I started noticing I am becoming "bronzed" [Alexey is referring to superiority complex -- Kir]. You said or did some crap, and then learn that this is becoming the right way now, since ANK said so.

I started to doubt, maybe I am just using my authority to maintain status quo. Every single morning started with a fight with myself, then with the world. In 2003 I got fed by it, so I went away from public, and later switched to a different field of knowledge. At that time I started my first project in Parallels. The task was to implement live migration of containers, and it was very complicated.

Now in Parallels we work on Parallels Cloud Storage project, developing cluster file systems for storing virtual machine images. The technology itself is a few years old already, we did a release recently, and are now working on improving it.

Q: How does your workplace look like?

My workplace is a bunch of computers. But I only work on a notebook, currently it's Lenovo T530. Other computers here are used for various purposes. This display, standing here, I never use it, nor this keyboard. Well, only if something breaks. Here we have different computers, including a Power Mac, an Intel and an AMD. I was using those in different years for different experiments. Once I needed to create a cluster of 3 machines right here at my workplace. One machine here is really old, and its sole purpose is to manage a power switch, so I can reboot all the others when working remotely from home. Here I have two Mac Minis and a Power Mac. They are always on, but I use them rarely, only when I need to see something in Parallels Desktop.

Q: What software do you use?

I don't use anything except for Google Chrome. Well, an editor and a compiler, if they qualify for software. I also store some personal data and notes in Evernote.

I only use a text console. For everything. In fact, on a newer notebooks, when the screen is getting better, the console mode is working worse and worse. So I am working in a graphical environment now, running a few full-screen terminals on top of it. It looks like a good old Unix console. So this is how I live, read email, work.

I do have a GMail account, I use it to read email from my phone. Sometimes it is needed. Or, when I see someone sent me a PDF, I have nothing else to do than to forward this email to where I can open that PDF. Same for PPT. But this is purely theoretical, in practice I never work with PPT.

I use Linux. Currently it is Fedora 13 -- not a newest one, to say at least. I am always using a version that was a base for a corresponding RHEL release. Every few years a new Red Hat [Enterprise Linux] is released, so I install a new system. When I do not change anything for a few years. Say, 5 years. I can't think of any new feature of an OS that would force me to update. I am just using the system as an editor, same as I have used it 20 years ago.

I have a phone, Motorola RAZR Maxx, running Android. I don't like iOS. You can't change anything in there. Not that I like customizations, I like a possibility to customize. I got a Motorola because I hate Samsung. This hatred is absolutely irrational. I had no happiness with any Samsung product, they either did't work for me or they break. I need a phone to make calls and check emails, that is all I need. Everything else is disabled -- to save the battery.

I am also reading news over RSS every day, like a morning paper. Now Feedly, before it was Google Reader, until they closed it. I have a list of bloggers I read, I won't mention their names. I am also reading Russian and foreign media. Lenta.ru, for example. There's that nice english-language service, News 360. It fits for what I like and gives me the relevant news. I am not able to say if it works or not, but the fact is, what it shows to me is really interesting to me. It was showing a lot of sports news at first, but then they disappeared.

I don't use instant messengers like Skype or ICQ, it's just meaningless. If you need something, write an email. If you need it urgently, call. Email and phone covers everything.

Speaking of social networks, I do have a Facebook account with two friends -- my wife and my sister. I view this account only when they post a picture, I don't wander there for no reason.

Q: Is there a use for paper in your work?

It's a mess. I don't have a pen, so when I would need it I could not find it. If I am close to the notebook and I need to write something -- I write to a file. If I don't have a notebook around, I write to my phone. For these situations I recently started to use Google Keep, a service to store small notes. It is convenient so far. Otherwise I use Evernote. Well, I don't have a system for that. But I do have a database of everything on my notebook: perpetual emails, all the files and notes. All this stuff is indexed. Total size is about 10 gigabytes, since I don't have any graphics. Well, if we throw away all the junk from there, almost nothing will remain.

Q: Is there a dream configuration?

What I have here is more than enough for me. This last notebook is probably the best notebook I ever had.

I was getting used to it for a long time, swore a lot. I only use Thinkpads for a long time. They are similar from version to version, but each next one is getting bigger and heavier, physically. This is annoying. This model, they changed the keyboard. I had to get used to it, but now I realize this is the best keyboard I ever had. In general, I am pretty satisfied with ThinkPads. Well, if it would had a Retina screen and be just 1 kilogram less weight -- that would be ideal.

Debian kernel packages

Good news, everyone!
Prof. Farnsworth


Many people use OpenVZ on Debian. In fact, Debian was one of the distribution that come with OpenVZ kernel and tools. Unfortunately, it's not that way anymore, since Debian 7 "Wheezy" dropped OpenVZ kernel. A workaround was to take an RPM-packaged OpenVZ kernel and convert it to .deb using alien tool, but the process is manual and somewhat unnatural.

Finally, now we have a working build system for Debian kernel packages, and a repository for Debian Wheezy with latest and greatest OpenVZ kernels, as well as tools. In fact, we have two: one for stable, one for testing kernels and tools. Kernels debs are built and released at the same time as rpms. Currently we have vzctl/vzquota/ploop in 'wheezy-test' repository only -- once we'll be sure they work as expected, we will move those into stable 'wheezy' repo.

To enable these repos:

cat << EOF > /etc/apt/sources.list.d/openvz.list
deb http://download.openvz.org/debian wheezy main
deb http://download.openvz.org/debian wheezy-test main
EOF
apt-get update


To install the kernel:
apt-get install linux-image-openvz-amd64

More info is available from https://wiki.openvz.org/Installation_on_Debian and http://download.openvz.org/debian/

An OpenVZ Experiment, 1 year later

Some of you may recall that last December I did an experiment where I created 638 OpenVZ containers on an HP Proliant DL380 G5 machine with dual quad-core CPUs and 32GB of RAM. I stopped there because I ran into an error. Well, one of the OpenVZ / Parallels developers suggested a fix back in July both as a comment to my article and as a comment to the bug report... but somehow I overlooked it until I ran across it again the other day when cleaning out my email.

I finally got a chance to give it a try and sure enough it removed the limit I had run into (the sysctl kernel.pid_max default setting being too low) and I verified it by creating 700 containers.

At first I decided to stop there but then I got an email from Kir asking if disk space was going to end up being my real limitation. I'm wondering if Kir has seen other experiments that go to this extreme or if he is simply a good guesser (with some inside information)? Anyway, I decide to bump it up to 1,000 containers. Sure enough, the machine is handling it just fine.

I didn't do a completely new write up, I just wrote a few more comments to the original article and you can find it here:

An OpenVZ Experiment - How many containers?
http://www.montanalinux.org/openvz-experiment.html

Tags:

Containers mini-summit and Linux Symposium

While I am writing this, people are discussing the future of containers in the Linux Kernel at the containers mini-summit which is happening in Ottawa at the moment. You can check some rough notes from the event here. Three guys from OpenVZ team are there: Pavel Emelyanov, Denis Lunev, and Andrey Mirkin.

If you are attending Linux Symposium in Ottawa, note that this Friday, 25th, Andrey Mirkin will talk about containers checkpointing and live migration (12:00, Rockhopper room). It's going to be an interesting talk, do not miss it.

Also, this Wednesday, 23rd, Balbir Singh will lead a BoF on Memory Controller (17:45, Fiordland room). Memory controller is quite important for containers, and while some stuff are already in the mainline kernel, there's still lots to be discussed and developed in the area. You can think of this BoF as an extension to containers mini-summit.
One of the goals of OpenVZ project is to integrate containers functionality into the mainstream Linux kernel. As you know, most of the new kernel code goes through Andrew Morton, the right hand of Linus Torvalds.

I just came across the video of Andrew speaking at the LinuxWorld Expo 2007. Among the other topics, he tells what is going to be in the kernel in a year or so. It is quite interesting to see what he thinks of containers -- to see that part, scroll to 40:58.

Update: here's the transcription of the relevant part, provided by dowdle.

The one prediction I am prepared to make is that over the next 1 to 2 years there'll be quite a lot of focus in the core of the Linux kernel on the project which has many names. Some people call it containerization, others will call it operating system virtualization, other people will call it resource management. It's a whole cloud of different features which have different applications.

It can be used for machine partitioning, to partition workloads amongst one machine, otherwise known as workload management.

Server consolidation. Well, you have a whole bunch of servers which are 30 percent loaded -- move all those things onto one the machine without having to tread on each others toes.

Resource management. A number of people in the high end numerical computing want this; numerical computing area want resource management. Other people who are running world famous web search engines also want resource management in their kernel. In fact, the major, central piece of the whole containerization framework is from an engineer at Google. It's in my tree at present and I'm hoping to get it in at 2.6.24. It's just a framework for containerization. A whole lot of other stuff is going to plug in underneath it, which is under development at present.

So an example of resource management is you might have a particular group of processes, [and] you want to not let it use more than 200 MB of physical memory, and a certain amount of disk bandwidth, network bandwidth, a certain amount of CPU -- so you can just have this little blob and give it maximum amount of resources it can consume, let it run without letting it trash everything else which is running on the machine. So that is a resource management application. People also need this feature for high availability... and I'm still not really sure I understand why.

Also the OpenVZ product, which comes out of the development team in Russia -- that's a mature project that is mainly for web server virtualization, having lots and lots of different instances of the web server on one machine, not have one excessively taking resources away from another. They've been working very hard and very patiently, and with great accommodation on this project. I hope slowly we'll start moving significant parts of the OpenVZ product into the Linux kernel in a way in which it's acceptable to all the other stake holders, so that those guys don't end up carrying such a patch burden.
Last week I went to Cambridge, UK with my colleague Pavel Emelyanov to take part in the LinuxConf Europe and the containers mini-summit, as well as the Linux Kernel Summit session devoted to containers. Pavel, who works in the OpenVZ kernel team, is now working on integrating our technology into the mainstream Linux kernel. To his credit, the memory controller and the PID namespace patch (see my recent blog post), which were integrated into -mm recently, are mostly due to him.

The first event in Cambridge was LinuxConf Europe, where we both presented our talks on containers -- mine was a general introduction to virtualization, containers, and OpenVZ, while Pavel described some intimate details of memory controller (read "beancounters") implementation.

The next day we had to skip the LinuxConf to take part in the containers mini-summit. This was an event for all the containers shareholders to discuss what and how to present the containers topic at the Kernel Summit. Unfortunately, Eric Biederman (Linux Networx) and Paul Menage (Google) came later, and Balbir Singh (IBM) was buzy with VM mini-summit, so we did this mini-summit in two rounds. First round was with Pavel (OpenVZ), Cedric Le Goater (IBM), Oren Laadan (of Zap -- a checkpointing and live migration project), Kamezava Hiroyuki (of Fujitsu Japan, mostly interested in resource management), and Paul (who joined us over Skype). The second round was with Eric, Paul, and Balbir -- the next day in the hall. The results of this mini-summit are a few threads on containers@ mailing list, plus a few documents here.

Finally, there was 30-minute topic on the Kernel Summit devoted to the containers. Paul and Eric have summarized what we have done so far, and what are we going to do next. There was not much discussion, which I think is healthy because now everybody knows about containers and why they are needed. Slides from the talk are available here. Jonathan Corbet (of Linux Weekly News) also provided a summary of the topic (this is still subscriber-only content, but since I'm a subscriber I can share a free link with you).

It feels like we are making good progress and are on the right path to a containers implementation in the Linux kernel. You can see some people helping to make this happen in this photo. Click the image for larger version.

one kernel bug story among 305

A few days ago one of OpenVZ kernel team members, Pavel Emelyanov, posted a one-line patch to fix a bug in Linux kernel. He received the following reply from Andrew Morton, one of the upstream kernel maintainers:


I'm curious. For the past few months, people@openvz.org have discovered (and fixed) an ongoing stream of obscure but serious and quite long-standing bugs.

How are you discovering these bugs?


Andrew added later:


hm, OK, I was visualising some mysterious Russian bugfinding machine or something.

Don't stop ;)


So, here is the story behind that bug.

A few months ago, in the course of OpenVZ kernel testing, our QA (Quality Assurance) team found a strange issue. The thing is, every container (VE) in OpenVZ has a set of resource usage counters (and limits) called beancounters. All the usage counters should be zero when a VE is stopped, since naturally then all the resources are released. The issue was that a resource called kmemsize (a kernel memory used on behalf of given VE) had a usage counter of 78 bytes after the VE was stopped -- which effectively means 78 bytes of kernel memory were lost (or leaked, as programmers say).

Who cares about 78 bytes, especially on a server with 16 gigabytes (17,179,869,184 bytes) of RAM? We do. Pavel checked the beancounters debug information which showed that one struct user object has leaked. He then tried to reproduce that but with no luck.

Bugs that can not be reproduced are tough. The only option left was to audit the kernel source code. That involved finding all the places where struct user object is referenced, and checking the code correctness (the term "correctness" in this context means that every object that is allocated must later be released). It took him 4 hours to do the audit, and he found one place where the reference to an object might be lost (which means it could not later be released). It's the same as if you lend a book to your friend and later forgot whom you gave it to -- you lost the reference and you can't get the book back.

In this case, after the problem was found, fixing it was pretty straightforward. So Pavel wrote a fix and a demo code to trigger the bug, tested the fix and sent it to Linux kernel mailing list.

Why is this particular incident so important?
* It's OpenVZ code (beancounters) which helped to detect the leak in the first place -- as the bug is very hard to trigger (unless you know how) and the leak is small enough that it might not be discovered at all.
* It demonstrates OpenVZ developers dedicated attitude. They never dismiss real bugs as "works for me" or "invalid", and work to find the root cause and fix the problem.
* This bug is in fact a security issue. An ordinary user (actually two users are needed in this case) could exploit the bug and eat all the kernel memory, thus bringing the whole system down. Worse scenarious are possible as well.
* Incidentally, OpenVZ is protected from this security issue -- because kmemsize beancounter (which helped to found it) limits kernel memory usage per Virtual Environment.

Most important of all, this is just one out of 305 kernel patches by our team which were accepted into the mainstream Linux kernel during a one-year period. Almost one patch a day, excluding weekends and holidays. And we are not going to stop! :-)
Recently, I had the opportunity to present at a session of the Gelato Itanium Conference and Expo in San Jose. It was a good fit because they had a special track on virtualization, and OpenVZ (and the Virtuozzo product) is the only stable virtualization technology available now for Itanium servers.

Once again, I was able to talk with Andrew Morton (a kernel hacker, the right hand of Linus Torvalds) and was encouraged about the prospect of OS virtualization and OpenVZ in the Linux kernel. That is something we would really like to see and have been working towards. This article summarizes Andrew’s remarks noting “OpenVZ already has thousands of systems out there” and “as far as containerization standard in mainline goes, ‘most of the stakeholders are playing together quite nicely’”.

Yes, we are and we’ll keep at it so we can realize our goal.

Latest Month

July 2016
S M T W T F S
     12
3456789
10111213141516
17181920212223
24252627282930
31      

Syndicate

RSS Atom

Comments

Powered by LiveJournal.com
Designed by Tiffany Chow