Testing behaviour of a production CF environment


Graham Bleach
 

Hello,

What do you use to test the behaviour of your production environments?

We are currently not live and are running these in each environment:
- cf-smoke-tests to test core functionality is working
- cf-acceptance-tests to test behaviour in more detail
- our own custom acceptance tests against code we've written, behaviour
we've configured and care about not breaking
- external monitoring against some deployed apps

We need to stop running cf-acceptance-tests in production, because they
sometimes cause problems if they exit prematurely and eg. leave an
unexpected buildpack as the first buildpack in the list. So we could run
those tests only in our CI environment every time we change something.

However we'd like to identify behaviour changes that aren't caused by our
changes and don't occur in our CI environment. For example, we recently
uncovered a problem with an infrastructure product that we only noticed by
running smoke-tests in production - that error didn't happen in other
environments. We're worried about the coverage we'd lose by not running the
tests.

One option that seems appealing to us is to try to work out a way of
running just the "safe" acceptance tests. For our purposes, "safe" tests
probably means ones that don't need to run as admin, that could run in
their own org - the isolation features of CF probably protect us enough
against impact to people using CF.

But it doesn't seem obvious how to currently run such a subset of the
acceptance-tests and do so in a way that's likely to be stable in the
future, so I asked this question.

Graham

Join cf-dev@lists.cloudfoundry.org to automatically receive all group messages.