Hi cf-dev!


I’d like to discuss some new security features we’re adding to Garden, and
what they mean for operators.
Tl;dr:

Garden, Cloud Foundry’s container creation and management engine, now has
experimental support for being run without needing to be root. This offers
some quite nice security benefits -- Garden server exploits don’t give an
intruder root access to the host -- so we’d like to transition to making
this non-experimental and, in fact, the default, asap.

This will require action if you still have privileged containers in your
environment (read on..).

What do I have to do?

Most operators will not have to do anything. It is possible to opt-in early
using the* garden.experimental_rootless_mode* bosh property.


*We do not yet suggest operators turn this on*. Please don’t turn this on
yet :).


We will be performing some final work on the feature and then we plan to
first run this on a subset of the servers on PWS to ensure there are no
last minute bugs or performance regressions. Once the garden team is
confident we will invite operators - via a further email to this mailing
list - to use the feature flag to turn this on in their environments and
provide feedback as to any problems. Finally we will make the new rootless
mode the default (see the Timeline below). If you do nothing, we expect -
as long as you don’t use privileged containers (again, see below) -
everything will work fine and things will just become more secure at some
point.

Why are we communicating now?

Turning on rootless mode has one important limitation: *we can no longer
support privileged containers once rootless mode is enabled* (since
privileged containers have elevated permissions that only root can grant).
The default in diego and cf-release/cf-deployment has been to disable
privileged containers for new apps for some time, however it is possible
that some operators will still have some privileged apps running (for
example old apps pushed before the default changed, or environments that
rely on fuse which have chosen not to disable privileged containers).

If you have privileged containers you will need to migrate these
containers. If the reason for the privileged containers is needing to mount
nfs or fuse, you should consider transitioning to the cf-persistence volume
support now in the platform. We believe most operators will already have
transitioned away from privileged containers but if you have not feel free
to reach out to us in the #garden channel on slack or via email and we'll
be very happy to help.

Timeline

Our provisional timeline is below


1.

Garden team completes Rootless track (behind a feature flag) -- *DONE*
2.

Garden requests PWS to deploy Rootless mode on increasing fraction of
PWS while being ready to roll back if there are unexpected issues *(this
month)*
3.

Garden recommends other operators enable Rootless mode and provide
feedback *(1-2 months from now)*
4.

Rootless mode becomes the default, operators that still have privileged
containers can opt back in to Privileged mode but should make plans to
transition any privileged containers.* (2-3 months from now)*
5.

We do not plan at this time to remove the ability to opt back into
privileged mode since this will be used for use cases such as bosh-lite
where privileged containers are still required. It will, however, no longer
be the default in cf-deployment and, 3 months after we make Rootless mode
the default we will begin to treat Privileged mode as a development-only
feature -- in other words our expectation will become that multi-tenant
production environments will not use it. *(3-6 months from now)*


Thanks!

-Julz