Your approach sounds good. What you are doing/plan to do in CI sounds
perfect, as well as your plan for production (namely, run what you have in
CI except cf-acceptance-tests). In the README for cf-acceptance-tests, we
These tests are not intended for use against production systems, they are
intended for acceptance environments for teams developing Cloud Foundry
itself. While these tests attempt to clean up after themselves, there is no
guarantee that they will not mutate state of your system in an undesirable
I'd recommend if you're going to run critical workloads on production, you
should consider having a staging environment where you roll out a CF
upgrade before you roll it out to production.
We are actually already tracking an issue related to buildpacks not being
But as you'll be able to see, it's not the highest priority at the moment.
The README attempts to give some idea of whether test suites are unsafe to
run in certain contexts:https://github.com/cloudfoundry/cf-acceptance-tests#explanation-of-test-suites
And the section on Test Execution explains how you can skip test suites,
tests matching a certain regex, etc:https://github.com/cloudfoundry/cf-acceptance-tests#test-execution
On Tue, May 31, 2016 at 10:15 AM, Graham Bleach <
What do you use to test the behaviour of your production environments?
We are currently not live and are running these in each environment:
- cf-smoke-tests to test core functionality is working
- cf-acceptance-tests to test behaviour in more detail
- our own custom acceptance tests against code we've written, behaviour
we've configured and care about not breaking
- external monitoring against some deployed apps
We need to stop running cf-acceptance-tests in production, because they
sometimes cause problems if they exit prematurely and eg. leave an
unexpected buildpack as the first buildpack in the list. So we could run
those tests only in our CI environment every time we change something.
However we'd like to identify behaviour changes that aren't caused by our
changes and don't occur in our CI environment. For example, we recently
uncovered a problem with an infrastructure product that we only noticed by
running smoke-tests in production - that error didn't happen in other
environments. We're worried about the coverage we'd lose by not running the
One option that seems appealing to us is to try to work out a way of
running just the "safe" acceptance tests. For our purposes, "safe" tests
probably means ones that don't need to run as admin, that could run in
their own org - the isolation features of CF probably protect us enough
against impact to people using CF.
But it doesn't seem obvious how to currently run such a subset of the
acceptance-tests and do so in a way that's likely to be stable in the
future, so I asked this question.