On Jul 31, 2015, at 1:56 PM, Chip Childers <cchilders(a)cloudfoundry.org> wrote:

This is great news for CF and a good writeup of the reasoning. Thanks Julz!

-chip

Chip Childers | VP Technology | Cloud Foundry Foundation

On Fri, Jul 31, 2015 at 1:49 PM, Julz Friedman <julz.friedman(a)gmail.com> wrote:
Hi cf-dev, I’d like to discuss some exciting changes the Garden team is planning to make in Diego’s container subsystem, Garden.

Garden? What’s that?

Garden is the containerisation layer used by Diego. Garden provides a platform-neutral, lightweight container abstraction that can be backed by multiple backends (most importantly, a Linux backend and a Windows backend). Currently the linux backend is based on our own code which evolved from Warden and which has been used to power Cloud Foundry for many years. Garden enables diego to support buildpack apps and docker apps (via the Linux backend) and windows apps (via the Windows backend).

So: What's changing?

We're planning to use runC [1] as the Linux backend for Garden.

Why?

Garden has always been an unopinionated container system - we like to have the opinionation happen at the higher levels (i.e. in Diego). Docker, on the other hand, is quite an opinionated container technology: it tightly couples containerisation and the user experience (which is one of the reasons docker is so great to use, I’m not knocking docker here!). Recently, docker and others (including IBM and Pivotal) have come together under the Open Container Initiative to spin out an unopinionated common containerisation runtime, “runC”, which gives us a fantastic opportunity to be part of this community while letting us ensure we can retain the flexibility required by our broader use cases. RunC is a reference implementation of the Open Container spec, which means both Docker and Cloud Foundry will be running the same code, and both Docker and Cloud Foundry apps will be using Open Containers.

Using runC as the garden backend has two major advantages. Firstly it lets us reuse some awesome code and be part of the Open Container community. Secondly it means CF applications will be using not only the same kernel primitives as docker apps (as they already are today), but also the exact same runtime container engine. This will minimise incompatibility for our docker lifecycle and result in a first class experience for users, as well as letting us reuse and contribute back to a great open-source code base. We have some remaining features in the Garden Linux backend that we’d like to see in RunC, but we’re excited to engage with the Open Container community to close these gaps.

What about regular CF buildpack apps and the other nice features of Garden?

Moving to runC gives us all the above advantages without compromising our ability to also deliver the buildpack-based platform-centric workflows that make CF great. We will retain the garden abstraction to make it easy for Diego to support both buildpack apps, windows apps and docker apps, and we will maintain a small layer above runC to manage the containers, pull down native warden and docker root filesystems, let us perform live upgrades and so on.

Why not use the full docker-engine as the backend?

Docker-engine has both more capabilities than we need at the layer Garden runs and different opinions than Cloud Foundry currently requires. This means it’s harder for us to maintain (because it’s larger and does more stuff), harder for us to contribute to (for similar reasons) and for some of our use cases (particularly with Diego’s more generic lifecycles) we’d have to actively work around things that would be quite easy to expose if we use runC directly (for example docker-engine intentionally doesn’t support signalling `docker exec`ed processes, which is required by Diego[2]).

Most of the reasons you might want to use docker-engine (e.g. being able to ‘docker push’) make much more sense to expose at the platform level in a multi-host environment (you want to push to the cluster, not a single host) or need to be integrated with multi-tenancy (which again should happen at the platform level - you need access control on storage and networks to integrate with the rest of a multi-tenant platform). For these reasons Cloud Foundry prefers to implement many features at the Diego layer whereas docker-engine implements some of these capabilities at the host layer. As the capabilities for running distributed applications in containers continue to evolve, CF prefers the flexibility to implement the opinions of our developers and community for areas like networking and storage even if those may differ from other orchestration solutions like docker-engine, and in turn Garden needs to retain that flexibility.

We also note that many new features have come to runC first (e.g. criu snapshot restore and - importantly for us - user namespaces were first available in runC before being added to docker-engine; at the time of writing these are still not fully available in docker-engine). We’d like to be able to consume new features as they come out in runC, rather than waiting for them to make it in to docker-engine. We also hope to be contributing new features of our own and this is much easier for us to accomplish against the smaller surface area of runC, and within the open context of the Open Container Initiative.

When will this happen?

Our first goal is completing the work of improving Garden’s security profile around supporting docker apps in production, we're about two weeks out from this according to Tracker and plan to do this with the current code. As soon as we hit this milestone we plan to shift our focus to runC. We have an initial prototype working and will iterate quickly to bring this to production quality and switch over when we feel confident.


I’m excited to hear the community’s views and input on this, so let us know what you think!

Thanks!
- Julz, on behalf the Garden Team

[1]: https://github.com/opencontainers/runc
[2]: https://github.com/docker/docker/pull/9167, https://github.com/docker/docker/pull/9402


_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

thanks julz for summarizing all of this. i'm very excited that cloud
foundry will be able to use runc and contribute to the open container
initiative. by joining with the other members and working together, we'll
be able to use the same base runtime as docker, coreos and others. we'll
also preserve the flexibility to do the innovations and user experience we
want for CF users above the core container runtime. this seems like a big
win for everyone.

On Fri, Jul 31, 2015 at 3:06 PM, Deepak Vij (A) <deepak.vij(a)huawei.com>
wrote:

Hi Julz & the whole garden team, it is great to know that Garden
Container is moving towards Open-Container-Project (OCP) App-Container
specifications. Great work.

I am hoping that down the road we will also see App Container Pods
(Co-locating Containers) capabilities enabled as well. A pod is a list of
apps that will be launched together inside a shared execution context (
single Unit of Deployment, migration etc. sharing IP address Space, Storage
etc.). Kubernetes also supports similar Pod concept.

Pod architecture allows me to enable design patters such as Sidecar,
Ambassador & Adaptor. All of this is really helpful from the standpoint of
refactoring the core telecom capabilities such as vEPC (virtual Evolved
packet Core network) and many more NFV/telecom capabilities - Network
Function Virtualization.

- Deepak Vij

----------------------------------------------------------------------

Message: 1
Date: Fri, 31 Jul 2015 18:49:25 +0100
From: Julz Friedman <julz.friedman(a)gmail.com>
To: "Discussions about Cloud Foundry projects and the system overall."
<cf-dev(a)lists.cloudfoundry.org>
Subject: [cf-dev] Garden is Moving!
Message-ID:
<
CAHfHzfOrrdEn_QBZwnoq7qQtXbBW1K2fk-NbbqgLSKaacMcPsw(a)mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Hi cf-dev, I?d like to discuss some exciting changes the Garden team is
planning to make in Diego?s container subsystem, Garden.

Garden? What?s that?

Garden is the containerisation layer used by Diego. Garden provides a
platform-neutral, lightweight container abstraction that can be backed by
multiple backends (most importantly, a Linux backend and a Windows
backend). Currently the linux backend is based on our own code which
evolved from Warden and which has been used to power Cloud Foundry for
many
years. Garden enables diego to support buildpack apps and docker apps (via
the Linux backend) and windows apps (via the Windows backend).

So: What's changing?

We're planning to use runC [1] as the Linux backend for Garden.

Why?

Garden has always been an unopinionated container system - we like to have
the opinionation happen at the higher levels (i.e. in Diego). Docker, on
the other hand, is quite an opinionated container technology: it tightly
couples containerisation and the user experience (which is one of the
reasons docker is so great to use, I?m not knocking docker here!).
Recently, docker and others (including IBM and Pivotal) have come together
under the Open Container Initiative to spin out an unopinionated common
containerisation runtime, ?runC?, which gives us a fantastic opportunity
to
be part of this community while letting us ensure we can retain the
flexibility required by our broader use cases. RunC is a reference
implementation of the Open Container spec, which means both Docker and
Cloud Foundry will be running the same code, and both Docker and Cloud
Foundry apps will be using Open Containers.

Using runC as the garden backend has two major advantages. Firstly it lets
us reuse some awesome code and be part of the Open Container community.
Secondly it means CF applications will be using not only the same kernel
primitives as docker apps (as they already are today), but also the exact
same runtime container engine. This will minimise incompatibility for our
docker lifecycle and result in a first class experience for users, as well
as letting us reuse and contribute back to a great open-source code base.
We have some remaining features in the Garden Linux backend that we?d like
to see in RunC, but we?re excited to engage with the Open Container
community to close these gaps.

What about regular CF buildpack apps and the other nice features of
Garden?

Moving to runC gives us all the above advantages without compromising our
ability to also deliver the buildpack-based platform-centric workflows
that
make CF great. We will retain the garden abstraction to make it easy for
Diego to support both buildpack apps, windows apps and docker apps, and we
will maintain a small layer above runC to manage the containers, pull down
native warden and docker root filesystems, let us perform live upgrades
and
so on.

Why not use the full docker-engine as the backend?

Docker-engine has both more capabilities than we need at the layer Garden
runs and different opinions than Cloud Foundry currently requires. This
means it?s harder for us to maintain (because it?s larger and does more
stuff), harder for us to contribute to (for similar reasons) and for some
of our use cases (particularly with Diego?s more generic lifecycles) we?d
have to actively work around things that would be quite easy to expose if
we use runC directly (for example docker-engine intentionally doesn?t
support signalling `docker exec`ed processes, which is required by
Diego[2]).

Most of the reasons you might want to use docker-engine (e.g. being able
to
?docker push?) make much more sense to expose at the platform level in a
multi-host environment (you want to push to the cluster, not a single
host)
or need to be integrated with multi-tenancy (which again should happen at
the platform level - you need access control on storage and networks to
integrate with the rest of a multi-tenant platform). For these reasons
Cloud Foundry prefers to implement many features at the Diego layer
whereas
docker-engine implements some of these capabilities at the host layer. As
the capabilities for running distributed applications in containers
continue to evolve, CF prefers the flexibility to implement the opinions
of
our developers and community for areas like networking and storage even if
those may differ from other orchestration solutions like docker-engine,
and
in turn Garden needs to retain that flexibility.

We also note that many new features have come to runC first (e.g. criu
snapshot restore and - importantly for us - user namespaces were first
available in runC before being added to docker-engine; at the time of
writing these are still not fully available in docker-engine). We?d like
to
be able to consume new features as they come out in runC, rather than
waiting for them to make it in to docker-engine. We also hope to be
contributing new features of our own and this is much easier for us to
accomplish against the smaller surface area of runC, and within the open
context of the Open Container Initiative.

When will this happen?

Our first goal is completing the work of improving Garden?s security
profile around supporting docker apps in production, we're about two weeks
out from this according to Tracker and plan to do this with the current
code. As soon as we hit this milestone we plan to shift our focus to runC.
We have an initial prototype working and will iterate quickly to bring
this
to production quality and switch over when we feel confident.


I?m excited to hear the community?s views and input on this, so let us
know
what you think!


Thanks!

- Julz, on behalf the Garden Team

[1]: https://github.com/opencontainers/runc

[2]: https://github.com/docker/docker/pull/9167,
https://github.com/docker/docker/pull/9402

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

--
Thank you,

James Bayer

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev