Hi Graham,

Something like that would be nice. However, technically, every test needs
admin because in their Before hooks they use the admin to create an org,
quota, etc. Sounds like what you want is to run tests that don't require
"admin" *in a way that might affect other users*. Even if we could split
up the tests right now along those lines, it will be a hard thing to
enforce moving forward since CATS is such a moving target, and touched by
so many different teams.

In principle, I'd like CATS to be able to guarantee it will clean up after
itself, do things with minimal affect on other users, and even be something
that could be safe to run in a production environment. And for such a
thing, I'd like to have some automated way to ensure CATS adheres to this
contract. Right now nothing like this exists, so we don't make the
guarantee that it's safe to run against prod. That said, it would be nice
to have that option in the future, so I'd like to keep CATS as
self-contained as possible right now. So for now, the best thing we can do
is identify tests, setup, or patterns that definitely would be bad for a
prod environment and fix them.

Have you been able to identify a complete list of problematic tests? We
can tackle them one-by-one and try to figure out which projects need
issues/PRs opened against them. E.g. I think for the buildpacks issue, it
might be desirable to limit the scope of a buildpack to a space or org,
though it would require some thought to figure out how to make sense of
buildpack priority.

Best,
Amit

On Wed, Jun 1, 2016 at 3:44 AM, Graham Bleach <
graham.bleach(a)digital.cabinet-office.gov.uk> wrote:

Hi Amit,

Thanks for your reply.

On 1 June 2016 at 04:06, Amit Gupta <agupta(a)pivotal.io> wrote:

The README attempts to give some idea of whether test suites are unsafe
to run in certain contexts:


https://github.com/cloudfoundry/cf-acceptance-tests#explanation-of-test-suites

And the section on Test Execution explains how you can skip test suites,
tests matching a certain regex, etc:

https://github.com/cloudfoundry/cf-acceptance-tests#test-execution

We've been looking through the test suites and can't see a straightforward
way to run only the "safe" tests that won't affect normal users / don't
require an admin user.

For instance, the apps suite includes both tests we'd like to run, testing
core user-facing behaviour and the admin buildpack lifecycle test with the
issue you linked to. I don't think skipping based on regexes on test names
works well, both because the regex will become long quite quickly and
because cf-acceptance-tests is a moving target - each time we upgraded to a
new release we'd need to review which tests were added and update our
regexes.

I wondered if other people would be interested in having a way to only run
the "non-admin" tests? If so, perhaps re-organising the suites to enable
that would be a welcome change?

Regards,
Graham