Шамс (shams) wrote in openvz,

  • Mood:

Solar Designer on Owl, OpenVZ and stuff: things you ached to know even if you didn't realize you did

Openwall GNU/*/Linux recently has released version 3.0. Owl project (yeah, you can call it Owl,too, which stands for Openwall Linux) is afloat around ten years, and it's an active and evolving. One of the neat features of Owl 3.0 is OpenVZ support. That was exactly the item which we got excited about, so we asked Solar (and he agreed kindly!) for a short interview.

— When someone is trying to find out what do you do, they mostly come across that you developed John the Ripper, currently lead the Owl project and participate in different open source security projects. Perhaps there are some new fancy things you're busy with, could you tell us a bit about it?

— This is nothing fancy, but besides the things you mentioned, we (some folks on the Openwall team) spend plenty of time on related client-facing work, including setting up and maintaining Owl/OpenVZ systems remotely, with lots of additional software running on top of them. This is one reason why our Live CDs include SSH remote access, after trivial initial configuration by someone with physical access or over IP-KVM. Then we can proceed with installation starting with partitioning of the hard disks while working over SSH. Besides providing income, which is much needed to fund Owl development and our other activities, this also ensures that we're testing our stuff under real-world usage conditions, and it provides stability for the project (we won't unexpectedly “lose interest” in it one day; we're going to continue to need updates for those systems that we're responsible for ourselves).

As to fancier things, I have plenty of potential project ideas, but too little time. So I mostly have to limit my activities to work related to projects already started. You mentioned that I “developed” John the Ripper, which I did, but it shouldn't just stay “developed” for many years more, it needs to evolve further, and there's demand for that (such as from penetration testing firms). This is about coming up with algorithms to produce more optimal candidate password streams, as well as about having them reflect new kinds of weak passwords that people start to use. This is also about efficient use of new CPU types, parallel processing, distributed processing, and use of GPUs. I did only a little bit of work on some of these things lately, being busy with lots of other stuff, but this is something I need to get back to and have specific ideas on.

A luckier project in 2010 was passwdqc, our proactive password/passphrase strength checking tool set (PAM module, library, command-line programs). Unlike John the Ripper, which audits password hash files, passwdqc does not permit weak passwords to be set in the first place. Dmitry V. Levin (of ALT Linux) and I made numerous enhancements to passwdqc: implemented new features, more effective checks (and tested those checks on thousands of real-world passwords), and at the same time made passwdqc even more portable. Proactive password/passphrase strength checking with passwdqc (applied when a user changes their Unix password) is now supported on Linux, FreeBSD, DragonFly BSD, OpenBSD, Solaris, and HP-UX.

— I've read through Owl release notes, slides and even that frightening "Weee SUID ftw!/O, no, no SUID" discussion on Slashdot, so what's so unique in your distro except no SUID (thanks to Openwall guys) and OpenVZ on board (feeling proud of ourselves)?

— It's a combination of things that make our distro unique, although a few things are also unique on their own: the lack of SUID programs in a usable default install, and syslogd credential passing (anti-spoofing). Some other things that make Owl special are:
* Proactive source code reviews of security-critical components of third-party software that we package.
* Security hardening changes in many packages, including some relatively uncommon changes (e.g., environment variable sanitization in glibc, Debian-generated weak key blacklisting in OpenSSH, privilege separation added to telnetd).
* Self-contained yet small: although the distro is small, it contains everything needed to easily rebuild itself from source.
* The included build environment is additionally capable generating ISOs and OpenVZ container templates ("make iso" and "make vztemplate").
* A single CD contains a live system, installable packages, the installer program, as well as full source code and the build environment.
* Old-fashioned text-only installer, simple scripts and configuration files, and reduced bloat overall. The system does not try to outsmart its administrator.
* RPM'ed default kernel, yet manual custom kernel builds are conveniently supported as well (only a few cumulative patch files are applied in the package).
* A good selection of small yet handy sysadmin and diagnostic tools, such as dmidecode, hdparm, ltrace, mdadm, mtree and nc (our ports from OpenBSD), Nmap (and Ncat), pciutils, smartmontools, strace. Of course, most of these are also available for/on mainstream distros, yet given Owl's purpose as a good "base system" for servers it is worth mentioning that we include all of these and more in our otherwise small set of packages, as well as in a default install.

— How big approximately is the user base?

— There's no "Owl user" registration mechanism, so I have no numbers on the size of our user base. It "feels" small, yet sufficient for us to continue with the project. Besides, much of the software that we develop as part of Owl we're also making available separately, and many of our patches, etc. are reused by other distros. For example, our "tcb" suite, which allows for the "passwd" command to work without being SUID, is also reused by ALT Linux's distributions and by Mandriva. This increases its user base a lot.

— What are the most common use cases for the Owl?

— Most common use cases? For us, that's OpenVZ "host systems", as well as OpenVZ container templates built with Owl as the "base system" and with extra software added (sort of "virtual appliances" that we make and install ourselves). We (or rather our client companies) mostly use this to provide "advanced" web hosting.

Here's a project built on top of OpenVZ and Owl, WebEnabled, which is a website development platform, involving multiple backend servers internally, with features such as "one-click" installs of popular web apps and cloning of existing "virtual hosts" (such as for the development/QA/production lifecycle).

We've also experimented with building Owl-based virtual machine hard disk images for a specific project, where the VMs would be run on end-users' desktop/laptop computers. Owl worked well for this purpose.

Since Owl's OpenVZ support is still relatively new (introduced into Owl-current a year ago, but only included in a stable release with Owl 3.0), we're hoping that more people and companies will make use of it in various ways, including in "virtual appliances" that they'd release. These could be VM images that also run OpenVZ containers, or they could be OpenVZ container templates or a mix of both.

Finally, we're aware of a McAfee (now Intel) hardware appliance product that is built upon an Owl-derived system (without OpenVZ, though). We'd be happy to see more hardware appliances make use of Owl as well, including of its OpenVZ integration.

We'd be happy to help more companies make and maintain Owl-based and Owl-derived systems, appliances, etc. Much of this work can be outsourced to us if desired. I think that hosting providers offering Linux-KVM VPS hosting may be interested in making Owl one of their standard distro options. They may specifically advertise the ability for a customer to create multiple OpenVZ containers inside such a VPS, which may provide an extra selling point for their offering vs. the competitors'. I am not aware of hosting providers doing this yet, although I am aware of a user of Owl having tried this out on his own and he was pleased with the result. Thus, I am mentioning this not so much as an answer to your question, but rather to encourage such use by hosting providers.

— I usually pay attention to mascots :) Yours looks like an owl near the boundary post/booth, it's really cool! Was it the first idea? Or there was a contest for a best picture?

— I'm glad that you like it. There was no formal contest. The Owl project logo that we currently use was based on one of several sketches contributed by Gleb Todosov, who also created our CD jewel case artwork. We used the latter up through our 2.0 release, and you can still see a portion of it in use on the first slide in our presentation.

— We're excited that you support OpenVZ in your distro and would like to dig into details. Why did you decide to do that? I guess you tried OpenVZ before, what was your previous experience? What did you use it for?

— Personally, I got interested in container-based virtualization in 1999 or thereabout (although I didn't know the concept would be known under this name). My first exposure to Virtuozzo was during SWsoft's public beta testing, which I think was in 2000. It was possible to register for a free trial account, which I did, and I got onto an SWsoft-hosted free trial system running a 2.4.0-test kernel with early and unreleased Virtuozzo patches. I ended up finding a vulnerability and reporting it to SWsoft.

Fast forward to 2005, the OpenVZ project was about to be announced to the public, and Openwall was contracted by SWsoft to perform a security audit of OpenVZ, which I personally worked on in close coordination with the OpenVZ developers. This was a pleasant experience, and the code quality was better than what I was used to seeing. Most of the issues that were discovered during the audit got fixed promptly. During the audit, I made my own test install of OpenVZ (for fuzzing, etc.) on top of an Owl system on a machine dedicated for the purpose.

OpenVZ was released in 2006. As far as I can tell by looking at file timestamps now, it was August 2006 when we slowly started to deploy it on top of some of our clients' Owl systems - replacing the kernels and adding vz* tools, but keeping the Owl userland on what became OpenVZ host systems. After a long while, we had plenty of systems running Owl and OpenVZ together, and we also had Owl-based userlands running inside "VEs" (the word "container" was not in use in OpenVZ context yet). At some point, we needed to introduce more sysadmins to the team and we also needed more systems of this kind installed to satisfy our clients' needs. It proved painful to explain all the little details (hacks) of making things work right.

We had the idea of integrating OpenVZ into Owl before - I even briefly mentioned it during a 2006 interview - but the difficulty mentioned above might have been the straw that finally made us do it, despite that it meant temporarily giving up some of our kernel hardening patches (not reimplemented for 2.6/OpenVZ kernels yet).

— What do you like in latest releases (already tried VSwap?)? What do you not like (I myself can hardly imagine that after there is no need to set all the UBC manually and get the blind headaches there is something left to improve, but I'm sure I need a fresh look on those things), so is there anything you find obscure or inconvenient?

— Of your "latest releases", we're only making (a lot of) use of the RHEL5 branch kernels ("testing" and/or "stable" on different occasions), with some additional changes (we make our own builds). I guess that you were referring to your newer branches. Unfortunately, we have not tried those out yet. This means that we haven't tried VSwap yet, even though this is a very welcome and anticipated feature (and change from the older UBC model). We're likely to start moving to your RHEL6 branch kernels in Owl-current this year, and we're likely to use them in our next major release (Owl 4.0). We'll be sure to provide feedback as we do this.

As to further improvements that we'd like to see in OpenVZ, it's security hardening, primarily against Linux kernel vulnerabilities that might turn into "container escape" ones in OpenVZ-patched kernels. One fairly obvious enhancement would be to be able to set a container to 32-bit only, 64-bit only, or mixed. The first two settings would disallow the other type of syscalls, thereby significantly reducing the exposure of kernel interfaces to possible attacks from the container's system. Another Owl developer tells me that this specific enhancement was also suggested upstream (LXC, I guess).

— Is there anything you would like to add or tell to millions :) of OVZ blog readers?

— Feedback, please! We need more feedback on Owl; join the owl-users mailing list and post there, please. Successes, failures, comments, opinions, requests, all kinds of feedback is welcome. Also, please contribute to our wiki. So far, it appears that most people who use Owl successfully don't bother contacting us ("no need"), those who run into minor issues and solve/workaround the issues on their own also don't bother notifying us ("no need" or so they think, wrongly), and most of those who run into trouble or are unhappy with Owl for whatever reason either notify individual Owl developers privately or (possibly) give up without telling us of the problem. If you use Owl (or have tried or considered to), please do your part to help correct this "feedback problem".

Tags: interview, open source, openvz, security

Comments for this post were disabled by the author