Hi Guillaume,

Thanks for your question. The best place to start understanding how diego
generates manifests is to look at the manifest generation script in the
diego-release repository:

https://github.com/cloudfoundry-incubator/diego-release/blob/develop/scripts/generate-deployment-manifest

It primarily merges data into the main diego template:

https://github.com/cloudfoundry-incubator/diego-release/blob/develop/manifest-generation/diego.yml

There are several principles operating at the heart of this manifest
generation strategy:

1. The inputs to the script should be a fixed list of stubs with a specific
data structure (not an arbitrary list of stub files which can have all sorts
of crazy structures in their YAML).
2. The YAML structure of the stub should make sense for the purpose of the
stub, i.e. clearly represent the data it contains, it should not represent
the data in some strange contorted format that needs to know where in the
upstream manifest template it will be merged into.
3. Avoid (( merge )) within the body of the template, places that need to
have values filled in explicitly reference the value from the stub that they
need.
4. No nested templates, or "stubs" that have spiff double-parens syntax in
them.
5. Each stub should have a single responsibility (e.g. instance count
overrides, persistent disk overrides, etc.) which should be small and easy
to understand; any two stubs should have orthogonal responsibilities, e.g.
you don't specify some credentials in this stub, and some other credentials
in another stub.
6. The responsibilities of the stubs should be generalizable to any BOSH
release (e.g. "instance count overrides" is a generic enough purpose that
would make sense for any BOSH release.)
7. Have an explicit stub for IaaS settings (basically anything that goes
into a cloud_properties stanza in a BOSH manifest); the main template should
be IaaS agnostic (rather than a template for each IaaS).
8. Cleanliness; do not produce a manifest in the end that has a bunch of
unnecessary junk that BOSH doesn't even use (e.g. "meta"). People tend to
develop the bad habit of depending on this random data.
9. No default property values in the templates; defaults should live in the
job specs, the only properties exposed in the template should be the ones it
makes sense to override.
10. Don't expose every property for override. spiff has hard limitations
that make it impossible to use if you want to construct any arbitrary
deployment manifest for a given release (or set of releases). Given that it
already can't solve every problem, understand that the manifest generation
tooling also doesn't need to solve every possible problem. Advanced users
who want to override obscure properties can use their own advanced tooling.
11. Dovetail with upcoming bosh features:
https://github.com/cloudfoundry/bosh-notes/blob/master/links.md
https://github.com/cloudfoundry/bosh-notes/blob/master/global-networking.md
https://github.com/cloudfoundry/bosh-notes/blob/master/cloud-config.md
https://github.com/cloudfoundry/bosh-notes/blob/master/availability-zones.md

The long-term goal is to make BOSH smarter and ideally obviate the need for
complex tooling to generate these unwieldy, several-thousand-line YAML
manifests. On the surface, both diego-release and cf-release suggest how to
use spiff to generate manifests, but the way the two use spiff is
drastically different. With diego, we've started by simplifying how
manifests are generated so that the full power of spiff isn't needed, and
identifying better abstractions that make sense for BOSH to know about so
that eventually none of the power of spiff is needed.



-----
Amit, CF OSS Release Integration PM
Pivotal Software, Inc.
--
View this message in context: http://cf-dev.70369.x6.nabble.com/cf-dev-CAB-call-for-July-2015-moved-to-7-15-8a-PDT-tp645p986.html
Sent from the CF Dev mailing list archive at Nabble.com.