FYI..

I had asked some questions to the community on running stateful Docker
containers and Docker service broker which are two cf-community projects. I
got some great responses from the project leads Dr Nic + Vedran and
information from blog posts by Fergie on the Docker Service Broker and
Stateful Docker Containers.

I am summarizing the responses here, and hopefully the community can give
feedback and direction.

Have you used this broker and bosh release and what has been your
experience running Docker containers using this bosh release?

I will move these responses in a git repo as FAQ's if it makes sense, so we
don't lose the excellent responses from the community.

============
What is docker-bosh release?
Ans: One of the fastest ways to get Docker and orchestrate containers with
persistent data on any infrastructure is to deploy this BOSH release.

https://github.com/cloudfoundry-community/docker-boshrelease


What are the features of the docker bosh release?

Ans: From the blog by Fergie:

https://blog.pivotal.io/pivotal-cloud-foundry/products/managing-stateful-docker-containers-with-cloud-foundry-bosh

Why is this CF-BOSH release so “awesomic”?

- It works with a standard CF-BOSH without any modification, you will
only require a stemcell with kernel >= 3.8 (the Ubuntu Trusty ones);
- You can orchestrate multiple Docker containers into multiple virtual
machines;
- You can deploy your containers to your choice of IaaS (AWS, OpenStack,
VSphere, VCHS, CloudStack or Google Compute Engine) using the same
deployment tool;
- It will monitor automatically your containers and restart them in case
of failures;
- It allows to set dependencies between containers running in the same
virtual machine, so if a container fails, when it is restarted it will
restart also all of the dependent containers;
- It will also automatically monitor your virtual machines and recreate
them again in case of a failure;
- It allows to bind host volumes to your Docker containers in an very
easy way;
- It allows easily to resize any data disk attached to a Docker
container without losing any date


What is docker service broker?
Ans:
It is a generic containers service broker for the Cloud Foundry v2 services
API. This can expose and provision service offerings that run inside a
compatible container backend (currently Docker)
https://github.com/cloudfoundry-community/cf-containers-broker

What are the features of Docker Container Service Broker?
Ans: From the blog by Fergie:

https://blog.pivotal.io/pivotal-cloud-foundry/products/docker-service-broker-for-cloud-foundry


This service broker allows Cloud Foundry operators to do a few things. They
can expose and provision services offerings that run inside a compatible
container backend (currently only Docker is supported) and bind
applications to the service. The management tasks that the broker can
perform are:

- Provision a service container with random credentials
- Bind a service container to an application:
- Expose the credentials to access the provisioned service
- Provide a syslog drain service for your application events and logs
- Unbind a service container from an application
- Unprovision a service container
- Expose a service container management dashboard


What version of Docker does it support?

Ans: Docker v 1.9.1 as of 11/24/2015


Who maintains it?

Ans: cf-community. Initiated by Fergie from Pivotal. Turned over to
cf-community. Currently Dr Nic and 2x Stark and Wayne Staff and Vedran
Lerenc from SAP are leading and committing to this project.


Is Elastic Runtime aware of any of these containers or the services or the
swarm cluster?

Ans:

For Docker Services - uses the Docker Service Broker->Docker Daemon ->
Internal Docker Container.

Dr. Nic : No. The abstraction of how a backend primitive service is hidden
from Cloud Foundry - the service broker API is for communicating with
cf-containers-broker, which then talks to docker daemon, which runs a
container, which runs an internal process - e.g. postgresql-server. The
bindings that are returned allow CF apps to communicate with this internal
database service directly. The app does not know that Docker containers are
involved.

Vedran: Not if you don't make it aware.



How does the route to the containers or services get registered or traffic
get routed?

Ans: CC registers the Docker Broker and the broker returns the coordinates
through CC to the app.

Open Question: This is good for Docker Broker (Services) and a CF-app wants
to consume that service.

Scenario: What if I want to run a docker App and there is no need for
Docker Broker? Or do you foresee an NGINX (TCP) Router registered as a
CF-app, and routing requests from Go Router -> NGINX->Docker Container
running the App bound to as services…


Dr. Nic: This is not solved by these projects currently. There are TCP
routing projects going on within the CF team. In a personal project I wrote
my own TCP router based on haproxy & etcd; but that was because the docker
containers themselves were logging details with etcd; not the broker.

Vedran: Only the broker gets registered at the router, then it gets
registered at the CC. When containers are created (on request of the
CC/broker, by the Docker engine, potentially through Swarm), their exposed
ports were automatically mapped in the past by the Docker engine into the
ephemeral port range and are today (by default, that is) mapped by the
broker (to keep stable port mappings as the automatic port mappings are
unstable/may change when the Docker engine/VM crashes/is restarted). The
broker returns the coordinates (host, port, credentials, etc.) through the
CC to the app which then (given access is permitted, e.g. security groups)
can access the container directly.



Can you take logs out of the containers (similar to Diego containers) ?

Ans: Yes

Open Question: Verify

Dr. Nic: Not currently.
Vedran: Yes, by means of [Docker logging drivers](
https://docs.docker.com/engine/reference/logging/overview/), see job spec
https://github.com/cloudfoundry-community/docker-boshrelease/blob/master/jobs/docker/spec#L71-L77


Can you bring your own container/container stem cell?

Ans: Possible...BYO.. but if has to play by the rules of broker (contract)

Dr. Nic:

All docker images are BYO. Many of them were originally authored by Ferdy
and are in Ferdy's github account
https://github.com/search?q=user%3Afrodenas+docker

If you use other people's docker images you may not know what's in it. For
example
https://github.com/cloudfoundry-community/docker-boshrelease/blob/master/examples/docker-bosh-lite.yml#L75references
an image "bosh/elasticsearch" - I have no idea who made it or what's in it.

Personally, for the docker images I use in production I package into BOSH
releases, such as
https://github.com/cloudfoundry-community/postgresql-docker-boshrelease/tree/master/images

This way I know exactly how and who built the docker image and I can
control the version that is being installed into my productions docker vms.

Vedran:
Nope, not if you don't create an image that plays by the rules of the
broker (obeys its contract) and add it to the services section of the
broker configuration/expose it as new service.
I am not aware there is anything formal (text, diagram or otherwise), but
the code as such. On the other hand, the contract is easy to grasp once you
take a look at the service description and an image (from frodenas/ubuntu,
parameters like username/password, etc.). So even without studying the code
(which again is straight-forward/easy to grasp) you can implement new
images very quickly.


Can you connect to an internal Docker Repo?

Ans: Yes

Dr Nic:
https://github.com/cloudfoundry-community/docker-boshrelease/blob/master/examples/docker-bosh-lite.yml#L75
and change the "image: org/label" to "image: my_docker_uri"

Vedran:
Yes, by means of a registry mirror, though I never tried it, see job spec
https://github.com/cloudfoundry-community/docker-boshrelease/blob/master/jobs/docker/spec#L80-L81


How actively is this Bosh release being maintained and worked on?

Ans: Active and being maintained current by the community
https://github.com/cloudfoundry-community/docker-boshrelease/graphs/contributors

What are the dependencies of this release?

Ans: https://rubygems.org/gems/docker-api

Dr. Nic:
The project is dependent on the docker-api rubygem which is not maintained
by Docker Inc; so hopefully it continues to be maintained. Seems like its
maintained continuously so far https://rubygems.org/gems/docker-api



What are the use cases this release will address?

Ans: Services in Docker Containers, Containers with Persistent Data

From Github doc:

- Deploy statically defined Docker containers: see CONTAINERS.md for
deployment instructions.
- Deploy a Docker Service Broker for your Cloud Foundry deployment: see
SERVICE_BROKER.md for deployment instructions.
- Deploy a Docker Swarm cluster: see SWARM.md for deployment
instructions.



Dr. Nic:
We use this bosh release for clients until they replace their required
primitive services with something else; and I'm using it as a backend agent
for a new clustered PostgreSQL product - I have a front end service broker
that talks to backend cf-containers-broker APIs to bring up containers
across multiple AZs that form HA PG clusters. So I will continue
maintaining cf-containers-broker towards these ends.


What are the limitations of this release?

Ans: no HA, no backup, broker is singleton, Swarm is singleton.

Open Question: Is the Docker Service Broker singleton? Can I configure the
underlying containers running the actual service using Docker Clustering
and let the docker daemon manager the cluster?


Vedran:

While the current solution can even scale horizontally (as a whole), there
are limitations (even prohibitive ones) to real productive usage like
singletons (broker, swarm, individual containers), no HA, no backup, no
Docker-compose-like scenarios, no scale-out for the containers themselves,
etc.

Having said that, it’s a great solution to offer quite many services rather
easy/fast to your application developers. They are very inexpensive to run,
as well.
So it’s a great solution to try stuff out or develop it, but when you go
productive, you better have a service implementation with the properties
iterated above.

Yes. The broker is a singleton. If you run Swarm, then also Swarm is a
singleton. You can have multiple Docker engines (with Swarm), so there are
many (to scale the solution horizontally, not for redundancy or individual
service load), but it doesn’t help you as the Docker engines run individual
containers, so in fact they are all singletons.
No, you can’t have a clustered service right now.


​Thank you Dr Nic, Vedran and Fergie.
​

--
Rajesh
t: @rjain15