Currently, our best kernel line is the one that is based on Red Hat Enterprise Linux 6 kernels (RHEL6 for short). This is our most feature-reach, up-to-date yet stable kernel -- i.e. the best. Second-best option is RHEL5-based kernel -- a few years so neither vSwap nor ploop, but still good.

There is a dilemma of either releasing the new kernel version earlier, or delay it for more internal testing. We figured we can do both! Each kernel branch (RHEL6 and RHEL5) comes via two channels -- testing and stable. In terms of yum, we have four kernel repositories defined in openvz.repo file, their names should be self-explanatory:

* openvz-kernel-rhel6
* openvz-kernel-rhel6-testing
* openvz-kernel-rhel5
* openvz-kernel-rhel5-testing

The process of releasing kernels is the following: right after building a kernel, we push it out to the appropriate -testing repository, so it is available as soon as possible. We when do some internal QA on it (that can either be basic or throughout, depending on amount of our changes, and whether we did a rebase to newer RHEL6 kernel). Based on QA report, sometimes we do another build with a few more patches, and repeat the process. Once the kernel looks good to our QA, we put it from testing to stable. In some rare cases (such as when we do one simple but quite important fix), new kernels go right into stable.

So, our users can enjoy being stable, or being up-to-the-moment, or both. In fact, if you have more than a few servers running OpenVZ, we strongly suggest you to dedicate one or two boxes for running -testing kernels, and report any bugs found to OpenVZ bugzilla. This is good for you, because you will be able to catch bugs early, and let us fix them before they hit your production systems. This is good for us, too, because no QA department is big enough to catch all possible bugs in a myriad of hardware and software configurations and use cases.

Enabling -testing repo is easy: just edit openvz.repo, setting enabled=1 under an appropriate [openvz-kernel-...-testing] section.

You might have noticed that we have announced a new kernel branch named rhel5-testing a while ago (back in July, to be more specific). The idea is pretty simple: at the same time as giving the new kernel to our internal QA we are releasing it to rhel5-testing.

Although this change imposes some more work on me (more kernels to release, scripts to run, changelogs to prepare), I'm pleased to say that this model works very well. First, vendors who use our kernels as a base for theirs (for example, OWL) now enjoy earlier access to the sources. Second, new kernels get more testing coverage due to OpenVZ users who choose to use this branch. Finally, it works as a “technology preview”.

Now, let me explain why we have so strange version numbers in the recent rhel5-testing kernels — kernels 028stab07x are intermixed with 028stab070.y. The thing is, we still keep updating 028stab070.y with new fixes and upstream (RHEL) updates, while 028stab07x is a newer “sub-branch” which adds a few new features:

• live migration of containers with NFS and AutoFS mounts
• iotop working in containers and the host system

Because of these new features, these kernels haven't reached the stability yet so we keep releasing those in rhel5-testing. Hopefully soon it will end up being stable enough and we will abandon 028stab070.y in favor of 028stab078 (or so).

Update: this post was mostly written yesterday. Today we have just released 028stab078.1 kernel.

Update 7 Oct 2012: comments disabled due to spam