Thanks Chip. I've added suggested associated edits to the Requirements
document in google docs.

It would be great if the CF community could be informed and ideally asked
for feedback when the formal audit process starts being elaborated. What is
the expected schedule for the audit process to start certifying offerings ?


Guillaume.

On Fri, Nov 4, 2016 at 6:00 PM, Chip Childers <cchilders(a)cloudfoundry.org>
wrote:

These are excellent ideas Guillaume, which we'll use when creating the
formal audit process in the coming year. Thanks so much for taking the time
to document your thoughts!

On Fri, Nov 4, 2016 at 4:02 AM Guillaume Berche <bercheg(a)gmail.com> wrote:

Thanks Shannon.

Public pipelines displaying risk of vulnerability for online services
could be feasible by asserting the version of mandatory Cf core components,
including rootfs and buildpacks.

I had suggested to the buildpack team to extend the CC API so that users
are able to reliably fetch hashes for the stacks and buildpacks, see
https://docs.google.com/document/d/1k7WFhnig2qVg5chLztFerGhUU5KCk
OfQRczcxcVkhbg/edit This could be completed with the hashes for
additional mandantory components (e.g. matching the bosh releases hashes)
to be returned as part of http://apidocs.cloudfoundry.
org/246/info/get_info.html endpoint. Likely this endpoint would return
this info with a proper oauth scope such as cloudcontroller.read.





Guillaume.

On Thu, Nov 3, 2016 at 6:38 PM, Shannon Coen <scoen(a)pivotal.io> wrote:

Great ideas, Guillaume. Especially public pipelines displaying smoke test
results for each provider and displaying risk of vulnerability (comparison
of the providers deployed version of core vs latest version?).

Shannon Coen
Product Manager, Cloud Foundry
Pivotal, Inc.

On Wed, Nov 2, 2016 at 3:57 PM, Guillaume Berche <bercheg(a)gmail.com>
wrote:

Thanks Dieu and Chip for explicitly asking feedback on the 2017
certification process, and leaving time for the CF community to provide
such feedback. I see seldom comments in the google docs document itself, so
I flesh out for now some more comments on cf-dev(a). It might make sense
to track/follow up some of them as comments/suggested edits in google docs,
which I plan to do in the coming days.

Following is some more feedback from the perspective of CF users such as
Orange (as a complement to previous useful feedback from CF
distros/vendors/consulting):
- as a user considering adopting CF as a portable platform backed by
multiple certified vendors, in order to trust the certification label, I
expect the certification process to be transparent, the certification
audits/results/details to be publicly available (including URL and version
of the product or service instance PF URL was assessed), and documentation
to help me distinguish the CF portable core from proprietary added-value
features that may come with vendor lock-in.
- as a CF user picking a certified CF platform, in order to have
confidence that the minimum quality enforced by the CFF teams is
guaranteed, I expect the smoke and acceptance tests to be passing on the
certified platform.
- as a CF user picking a certified CF platform, in order to securely run
production workload, I expect a certified platform to provide up-to-date
CVE patches.

Here are some other examples of risks I would like to mitigate when
picking a certified platform:
- a certified vendor might choose to provide CF on a different OS distro
than what the CFF release engineering team is running in the pipeline
qualifying CF releases [1] (which is leveraging bosh stemcells [2] to
provision OS), therefore exposing me to unexpected bugs/quality issues in
the "CF kernel"
- a certified vendor might not yet publicly provide offerings that can be
purchased, in effect reducing the list of candidate vendors I can pick from
when considering CF as a portable multi-vendor platform
- a certified vendor might not provide timely CVE patches to the "CF
kernel", exposing me to security vulnerabilities that CF I expect CF to
protect me from. The current requirements of components not older than 6
months does not seem strong enough from this perspective.
- a certified vendor leveraging by default proprietary assets others than
the CFF "batteries included" default from the CFF distribution. If
undocumented from in the certification audits, this creates vendor lock-in
for users adopting this certified platform.
- a certified vendor might choose to not enable/support a range of cf
core features (e.g. "cf ssh", "TCP routing", or "route services"), reducing
the scope of the certification.

Some more suggestions to the cert requirements:
- require the outcome/details of the certification audit to be published
for certified platforms (detailing source of distros, version tested,
etc...)
- require the passing smoke tests and acceptance tests test to become
requirements for certified platforms, and configuration of the tests and
reports to be published. Ideally encourage/require public pipelines from
the certified providers as to provide trust that the tests are still
passing after the certification is granted. As a side effect, this would
also encourage certified vendors in contributing to the release engineering
effort (which is currently mostly sponsored by Pivotal as Amit reminded)
and ensure that the pipelines are portable and can be run by the community
without hitting/missing pivotal-specifics. See [3] for related effort
orange went through with the help of the buildpacks team to replicate the
buildpacks-ci within Orange.
- require to document how the certified provider ensures the certified
platform remains secured w.r.t. upcoming CVEs
- make more explicit the CF extension points (either in the document or
as an external reference): services implementing the service broker API,
volume driver provider, network plugin provider, external CC/diego DB or
blobstore, CLI plugins, etc.... This would help clarify the scope of the
certified core and distinguish it from added value, proprietary extensions.

[1] https://runtime.ci.cf-app.com/teams/main/pipelines/cf-release
[2] https://runtime.ci.cf-app.com/teams/main/pipelines/cf-
release/resources/vsphere-stemcell
[3] https://github.com/cloudfoundry/buildpacks-ci/
pulls?q=is%3Apr+author%3Ao-orand

Regards,

Guillaume.

On Tue, Nov 1, 2016 at 7:04 PM, Chip Childers <cchilders(a)cloudfoundry.org

wrote:

Certainly! I didn't mean to close it down on anyone... keep them coming.

On Tue, Nov 1, 2016 at 2:04 PM Troy Topnik <troy.topnik(a)hpe.com> wrote:

Thanks Chip.

I've let the ContainerCF guys know that they should look at this. If we
can keep the discussion open a little longer, we may hear from them.

TT

--
Chip Childers
CTO, Cloud Foundry Foundation
1.267.250.0815




--

Chip Childers
CTO, Cloud Foundry Foundation
1.267.250.0815