On Jul 31, 2015, at 1:56 PM, Chip Childers <cchilders(a)cloudfoundry.org> wrote:

This is great news for CF and a good writeup of the reasoning. Thanks Julz!

-chip

Chip Childers | VP Technology | Cloud Foundry Foundation

On Fri, Jul 31, 2015 at 1:49 PM, Julz Friedman <julz.friedman(a)gmail.com> wrote:
Hi cf-dev, I’d like to discuss some exciting changes the Garden team is planning to make in Diego’s container subsystem, Garden.

Garden? What’s that?

Garden is the containerisation layer used by Diego. Garden provides a platform-neutral, lightweight container abstraction that can be backed by multiple backends (most importantly, a Linux backend and a Windows backend). Currently the linux backend is based on our own code which evolved from Warden and which has been used to power Cloud Foundry for many years. Garden enables diego to support buildpack apps and docker apps (via the Linux backend) and windows apps (via the Windows backend).

So: What's changing?

We're planning to use runC [1] as the Linux backend for Garden.

Why?

Garden has always been an unopinionated container system - we like to have the opinionation happen at the higher levels (i.e. in Diego). Docker, on the other hand, is quite an opinionated container technology: it tightly couples containerisation and the user experience (which is one of the reasons docker is so great to use, I’m not knocking docker here!). Recently, docker and others (including IBM and Pivotal) have come together under the Open Container Initiative to spin out an unopinionated common containerisation runtime, “runC”, which gives us a fantastic opportunity to be part of this community while letting us ensure we can retain the flexibility required by our broader use cases. RunC is a reference implementation of the Open Container spec, which means both Docker and Cloud Foundry will be running the same code, and both Docker and Cloud Foundry apps will be using Open Containers.

Using runC as the garden backend has two major advantages. Firstly it lets us reuse some awesome code and be part of the Open Container community. Secondly it means CF applications will be using not only the same kernel primitives as docker apps (as they already are today), but also the exact same runtime container engine. This will minimise incompatibility for our docker lifecycle and result in a first class experience for users, as well as letting us reuse and contribute back to a great open-source code base. We have some remaining features in the Garden Linux backend that we’d like to see in RunC, but we’re excited to engage with the Open Container community to close these gaps.

What about regular CF buildpack apps and the other nice features of Garden?

Moving to runC gives us all the above advantages without compromising our ability to also deliver the buildpack-based platform-centric workflows that make CF great. We will retain the garden abstraction to make it easy for Diego to support both buildpack apps, windows apps and docker apps, and we will maintain a small layer above runC to manage the containers, pull down native warden and docker root filesystems, let us perform live upgrades and so on.

Why not use the full docker-engine as the backend?

Docker-engine has both more capabilities than we need at the layer Garden runs and different opinions than Cloud Foundry currently requires. This means it’s harder for us to maintain (because it’s larger and does more stuff), harder for us to contribute to (for similar reasons) and for some of our use cases (particularly with Diego’s more generic lifecycles) we’d have to actively work around things that would be quite easy to expose if we use runC directly (for example docker-engine intentionally doesn’t support signalling `docker exec`ed processes, which is required by Diego[2]).

Most of the reasons you might want to use docker-engine (e.g. being able to ‘docker push’) make much more sense to expose at the platform level in a multi-host environment (you want to push to the cluster, not a single host) or need to be integrated with multi-tenancy (which again should happen at the platform level - you need access control on storage and networks to integrate with the rest of a multi-tenant platform). For these reasons Cloud Foundry prefers to implement many features at the Diego layer whereas docker-engine implements some of these capabilities at the host layer. As the capabilities for running distributed applications in containers continue to evolve, CF prefers the flexibility to implement the opinions of our developers and community for areas like networking and storage even if those may differ from other orchestration solutions like docker-engine, and in turn Garden needs to retain that flexibility.

We also note that many new features have come to runC first (e.g. criu snapshot restore and - importantly for us - user namespaces were first available in runC before being added to docker-engine; at the time of writing these are still not fully available in docker-engine). We’d like to be able to consume new features as they come out in runC, rather than waiting for them to make it in to docker-engine. We also hope to be contributing new features of our own and this is much easier for us to accomplish against the smaller surface area of runC, and within the open context of the Open Container Initiative.

When will this happen?

Our first goal is completing the work of improving Garden’s security profile around supporting docker apps in production, we're about two weeks out from this according to Tracker and plan to do this with the current code. As soon as we hit this milestone we plan to shift our focus to runC. We have an initial prototype working and will iterate quickly to bring this to production quality and switch over when we feel confident.


I’m excited to hear the community’s views and input on this, so let us know what you think!

Thanks!
- Julz, on behalf the Garden Team

[1]: https://github.com/opencontainers/runc
[2]: https://github.com/docker/docker/pull/9167, https://github.com/docker/docker/pull/9402


_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev