Hi cf-dev,


In addition to managing the release lifecycle of cf-release, the CF
Relation Integration team has been working on a new way to deploy CF. There
are a lot goals we're trying to accomplish with a new approach, and we
think we have accomplished (or will accomplish) most of them with
cf-deployment <https://github.com/cloudfoundry/cf-deployment>. We just
wanted to give the community a head's up and ask for feedback.

tl;dr:

- The Release Integration team is building cf-deployment
<https://github.com/cloudfoundry/cf-deployment> as a replacement for
manifest generation scripts in cf-release. It uses new BOSH features and
the new BOSH CLI to significantly simplify manifest generation.

- cf-deployment is *still under construction* and should be used primarily
in dev or test environments. We expect it to be ready for general
consumption sometime in March.

- We're working on a migration path from cf-release.

- cf-deployment deploys Diego instead of DEAs.

- Have questions or feedback? Feel free to open github issues or PRs. You
can also get a hold of us in the #cf-deployment
<https://cloudfoundry.slack.com/messages/cf-deployment/> channel of the
Cloud Foundry slack.


First of all, what is cf-deployment?

cf-deployment is, more or less, a BOSH deployment manifest for deploying
Cloud Foundry. It will eventually replace the manifest generation scripts
in cf-release. In fact, it will replace cf-release entirely.

Smaller, composable releases

One thing you might notice if you take a look at cf-deployment is that
we're using quite a few releases
<https://github.com/cloudfoundry/cf-deployment/blob/0994ecee65f5e77cc47bfcec08c23a4dbc1cdfbb/cf-deployment.yml#L1444-L1524>.
Well, cf-release has always been using these releases, but secretly and as
submodules. Those submodules are absent from cf-deployment, because you can
specify releases by their download url in a bosh manifest. The big payoff
here is really for the release teams -- they can iterate more quickly and
test more often against the changes made by other release teams. The effect
to deployers is that in time, we will deprecate cf-release and there will
no longer be a single bosh release that tries to contain all of Cloud
Foundry. Instead, this manifest will encode all of the releases that are
part of the core CF platform and will be the source of truth for how to
deploy CF.

Timelines (sort of)

cf-deployment is still *under construction* and shouldn't be used for
production environments. But we would LOVE for people to start using it for
test and development environments. In the next several weeks, we're going
to make sure that all the release teams are able to move their test
environments over to cf-deployment. After that, the next big milestone for
us will be to develop a clear migration path from cf-release to
cf-deployment. Once we iron that out, the RelInt team will feel comfortable
making cf-deployment the officially supported mechanism for deploying CF
(with BOSH, at least). We expect that there will be a few months where we
support both cf-deployment and cf-release before we deprecate cf-release
entirely.

Making Diego the default backend

One of the primary motivations for cf-deployment is to end-of-life the DEAs
<https://lists.cloudfoundry.org/archives/list/cf-dev(a)lists.cloudfoundry.org/thread/PFXSJMKSQ6UNR2I5U5Q2H2QTXTCAEIJR/>
and make Diego the default backend deployed with any CF. Currently, a
typical CF deployment (as defined by the manifest templates in cf-release
<https://github.com/cloudfoundry/cf-release/tree/master/templates>) deploys
DEAs, but none of the Diego components. Operators looking to include Diego
commonly deploy diego-release with a separate BOSH manifest
<https://github.com/cloudfoundry/diego-release/tree/develop/manifest-generation>
and must manage the two deployments (CF and Diego) separately. This setup
is harder to manage, relies on passing information between the two manifests
<https://github.com/cloudfoundry/diego-release/blob/develop/manifest-generation/config-from-cf.yml>,
and continues to treat Diego like an optional or experimental part of the
CF ecosystem. It's time to bring it into the fold. Accordingly,
cf-deployment includes all the Diego components and does not deploy DEAs.

Simplifying manifest generation

Another part of the mission of cf-deployment is to make it easier for
deployers to build their manifests. The spiff templates in cf-release have
become unwieldy and give first-time deployers a mountain of work to do
before they can see their first successful BOSH deploy of CF. Instead,
cf-deployment uses the new bosh cli
<https://github.com/cloudfoundry/bosh-cli> to achieve the following:

1. Deployers don't need to generate credentials. cf-deployments has a
`variables`
section
<https://github.com/cloudfoundry/cf-deployment/blob/a88d00538aad484c351ac05d3bfe5a8a12111d02/cf-deployment.yml#L1235>
in the manifest describing how all credentials needed by the CF deployment
should be generated. The bosh cli reads this section and generates
credentials for the deployer, and saves them to a file based on the
`--vars-store` parameter. Deployers can save those credentials however they
like. (Future work on this front includes leveraging BOSH config server
<https://github.com/cloudfoundry/bosh-notes/blob/master/config-server.md>
instead).

2. Deployers don't need to use spiff. The bosh cli includes a `-v` flag
that allows the deployer to provide a variable to be templated in to the
manifest. Coupled with credential generation, this means that deployers
need specify only their `system_domain` in order to generate a functional
manifest. Deploying from scratch looks like this:

bosh deploy cf-deployment.yml \

-v system_domain=my-cf.com \

--vars-store credentials.yml \

-d cf

3. Deployers can encode their custom configuration in a simple, repeatable
way. If you haven't seen BOSH ops files yet, allow me to introduce you. Ops
files implement a yaml version of json patch <http://jsonpatch.com/> that
allows deployers to specify mutations to cf-deployment. This empowers
deployers to make arbitrary changes to the manifest, and thus achieve their
desired configuration. For example, if a deployer wants to scale up the
number of API instances, they can run

bosh deploy cf-deployment.yml \

-v system_domain=my-cf.com \

--vars-store credentials.yml \

-d cf \

-o scale-api.yml

With the syntax for scale-api.yml looking like this:

---

- type: replace

path: /instance_groups/name=api/instances

value: 10

Feel free to take a look at the ops files provided in cf-deployment
<https://github.com/cloudfoundry/cf-deployment/tree/develop/operations> for
more examples.

Get in touch

Lastly, the team is eager for feedback on cf-deployment and any kind of
tooling we can build around it. We pay close attention to github issues on
the cf-deployment repo and the #cf-deployment channel in the CF Slack.
We'll try to publish more information to this email list as we discover
other important things that will be valuable to the wider community.


I'm also linking to a doc containing the exact same information
<https://docs.google.com/document/d/1UWNcgDtE4SxNPFumMfHKX5lH72CLGUNH-6Svie_vwN4/edit?usp=sharing>
to allow comments and discussion on specific points. I'll do my best to
answer any questions or concerns both in this email thread or in the doc.


Happy deploying!
David Sabeti
Product Manager, CF Release Integration Team