|
Hi all,
The CAPI team would like to take ownership of the CF acceptance tests and include them as part of CAPI release.
This solves several pain points we've experienced over the last few months, mainly due to the strong coupling between CATs and the Cloud Controller API.
Our plan is to submodule the CATs repo in to CAPI release, and bump CATs on a successful run through the CAPI ci pipeline. At this point, CAPI release will be bumped in cf-release develop, which other teams will consume for their own testing purposes.
We hope to make these changes in the near future as we wrap up our release extraction from cf-release. We'd like to know if any teams have any concerns about this before we proceed, so do let us know as soon as possible so we can address them.
Thanks,
Utako, CF CAPI Team
|
|
|
Michael Fraenkel <michael.fraenkel@...>
Can you explain why CATs should live under CAPI?
CATs now represents acceptance test for all of the Cloud Foundry
function driven mainly via the CLI.
Tieing this to any one release seems a bit artificial.
CATs is about testing the aggregation of all the various releases not
just CAPI.
I am more surprised that it wasn't suggested to be its own release which
dictated the versions for all releases that had passed.
- Michael
toggle quoted message
Show quoted text
On 3/22/16 3:05 AM, Utako Ueda wrote: Hi all,
The CAPI team would like to take ownership of the CF acceptance tests
<https://github.com/cloudfoundry/cf-acceptance-tests/> and include
them as part of CAPI release
<http://github.com/cloudfoundry/capi-release>.
This solves several pain points we've experienced over the last few
months, mainly due to the strong coupling between CATs and the Cloud
Controller API.
Our plan is to submodule the CATs repo in to CAPI release, and bump
CATs on a successful run through the CAPI ci pipeline. At this point,
CAPI release will be bumped in cf-release develop, which other teams
will consume for their own testing purposes.
We hope to make these changes in the near future as we wrap up our
release extraction from cf-release. We'd like to know if any teams
have any concerns about this before we proceed, so do let us know as
soon as possible so we can address them.
Thanks,
Utako, CF CAPI Team
|
|
|
Mike Youngstrom <youngm@...>
I agree with Michael. I think the CATs and Smoke tests should be in their own release(s). Mike On Wed, Mar 23, 2016 at 5:08 AM, Michael Fraenkel < michael.fraenkel(a)gmail.com> wrote: Can you explain why CATs should live under CAPI?
CATs now represents acceptance test for all of the Cloud Foundry function
driven mainly via the CLI.
Tieing this to any one release seems a bit artificial.
CATs is about testing the aggregation of all the various releases not just
CAPI.
I am more surprised that it wasn't suggested to be its own release which
dictated the versions for all releases that had passed.
- Michael
On 3/22/16 3:05 AM, Utako Ueda wrote:
Hi all,
The CAPI team would like to take ownership of the CF acceptance tests
<https://github.com/cloudfoundry/cf-acceptance-tests/> and include them
as part of CAPI release <http://github.com/cloudfoundry/capi-release>.
This solves several pain points we've experienced over the last few
months, mainly due to the strong coupling between CATs and the Cloud
Controller API.
Our plan is to submodule the CATs repo in to CAPI release, and bump CATs
on a successful run through the CAPI ci pipeline. At this point, CAPI
release will be bumped in cf-release develop, which other teams will
consume for their own testing purposes.
We hope to make these changes in the near future as we wrap up our release
extraction from cf-release. We'd like to know if any teams have any
concerns about this before we proceed, so do let us know as soon as
possible so we can address them.
Thanks,
Utako, CF CAPI Team
|
|
|
We had a series of meetings to figure out a good path for CATs that ranged from multiple teams to smaller groups. We met most recently to address concerns that include the following: • Workflow and Release Bumping Issues: CATs relies on calls to specific versions of CC API to run successfully. The CAPI CI currently bumps cloud_controller_ng within capi-release, and on a successful run of CATs (which lives on cf-release develop), capi-release then gets bumped in cf-release develop. This leads to a "chicken and egg" scenario, in that when changes are made to both CATs and CC, both cf-release and capi pipelines are broken until a manual fix is made. • In a world where cf-release no longer exists, how do we ensure CATs uses the correct versions of the CF CLI and CC API? tl;dr: Though not all of the CATs test the CC API explicitly, they all make use of the CC API and therefore make CC a choke point. Having CATs as a separate release may be a good idea, but doesn't necessarily address the strong coupling issues. We'd like to know if there are specific cases in which this will be detrimental to other projects' workflows before moving forward. On Wed, Mar 23, 2016 at 4:08 AM, Michael Fraenkel < michael.fraenkel(a)gmail.com> wrote: Can you explain why CATs should live under CAPI?
CATs now represents acceptance test for all of the Cloud Foundry function driven mainly via the CLI.
Tieing this to any one release seems a bit artificial. CATs is about testing the aggregation of all the various releases not just
CAPI.
I am more surprised that it wasn't suggested to be its own release which
dictated the versions for all releases that had passed.
- Michael
On 3/22/16 3:05 AM, Utako Ueda wrote:
Hi all,
The CAPI team would like to take ownership of the CF acceptance tests
<https://github.com/cloudfoundry/cf-acceptance-tests/> and include them
as part of CAPI release <http://github.com/cloudfoundry/capi-release>.
This solves several pain points we've experienced over the last few
months, mainly due to the strong coupling between CATs and the Cloud
Controller API.
Our plan is to submodule the CATs repo in to CAPI release, and bump CATs
on a successful run through the CAPI ci pipeline. At this point, CAPI
release will be bumped in cf-release develop, which other teams will
consume for their own testing purposes.
We hope to make these changes in the near future as we wrap up our release
extraction from cf-release. We'd like to know if any teams have any
concerns about this before we proceed, so do let us know as soon as
possible so we can address them.
Thanks,
Utako, CF CAPI Team
|
|
|
Mike Youngstrom <youngm@...>
I don't have any show stopping cases. But, my 2 main concerns are:
* Conceptually speaking if the CATS are for all will making them part of
capi hinder other teams from contributing to it?
* Having the CATs part of CAPI will require me to fork CAPI when I need to
fix/customize a test specific to our environment, something we frequently
need to do. Long term I'd prefer to fork a smaller more single purpose
release when these situations arise.
Mike
toggle quoted message
Show quoted text
On Wed, Mar 23, 2016 at 12:43 PM, Utako Ueda <uueda(a)pivotal.io> wrote: We had a series of meetings to figure out a good path for CATs that ranged
from multiple teams to smaller groups. We met most recently to address
concerns that include the following:
• Workflow and Release Bumping Issues: CATs relies on calls to specific
versions of CC API to run successfully. The CAPI CI currently bumps
cloud_controller_ng within capi-release, and on a successful run of CATs
(which lives on cf-release develop), capi-release then gets bumped in
cf-release develop. This leads to a "chicken and egg" scenario, in that
when changes are made to both CATs and CC, both cf-release and capi
pipelines are broken until a manual fix is made.
• In a world where cf-release no longer exists, how do we ensure CATs uses
the correct versions of the CF CLI and CC API?
tl;dr: Though not all of the CATs test the CC API explicitly, they all
make use of the CC API and therefore make CC a choke point. Having CATs as
a separate release may be a good idea, but doesn't necessarily address the
strong coupling issues.
We'd like to know if there are specific cases in which this will be
detrimental to other projects' workflows before moving forward.
On Wed, Mar 23, 2016 at 4:08 AM, Michael Fraenkel <
michael.fraenkel(a)gmail.com> wrote:
Can you explain why CATs should live under CAPI?
CATs now represents acceptance test for all of the Cloud Foundry function
driven mainly via the CLI.
Tieing this to any one release seems a bit artificial.
CATs is about testing the aggregation of all the various releases not
just CAPI.
I am more surprised that it wasn't suggested to be its own release which
dictated the versions for all releases that had passed.
- Michael
On 3/22/16 3:05 AM, Utako Ueda wrote:
Hi all,
The CAPI team would like to take ownership of the CF acceptance tests
<https://github.com/cloudfoundry/cf-acceptance-tests/> and include them
as part of CAPI release <http://github.com/cloudfoundry/capi-release>.
This solves several pain points we've experienced over the last few
months, mainly due to the strong coupling between CATs and the Cloud
Controller API.
Our plan is to submodule the CATs repo in to CAPI release, and bump CATs
on a successful run through the CAPI ci pipeline. At this point, CAPI
release will be bumped in cf-release develop, which other teams will
consume for their own testing purposes.
We hope to make these changes in the near future as we wrap up our
release extraction from cf-release. We'd like to know if any teams have any
concerns about this before we proceed, so do let us know as soon as
possible so we can address them.
Thanks,
Utako, CF CAPI Team
|
|
|
Michael Fraenkel <michael.fraenkel@...>
All the teams have to deal with a workflow and bumping. I don't believe
there is a team that doesn't have to also deploy CF with Diego (except
mine) and run CATs.
The issue I believe you are highlighting is something different. CAPI
needs a CAPI acceptance for its new features which have no CLI
counterpart. In the past CATs contained that. I think now that CATs now
includes DATs and RATs and xATs, its not part of one team. If CAPI needs
a place to test out new functions using cf curl, that can either live in
CATs or it goes some where else. CATs should be what an end user will
attempt which is hopefully not driving CF via 'cf curl'.
Having CATs live under any one release is a problem for any other team
that uses it and provides code. Any changes are gated by someone merging
a commit and making it through a pipeline that includes other changes
that maybe should not be consumed by others since its not part of the
"release" because we will get additional components from the CAPI
release that may not be ready.
- Michael
toggle quoted message
Show quoted text
On 3/23/16 2:43 PM, Utako Ueda wrote: We had a series of meetings to figure out a good path for CATs that
ranged from multiple teams to smaller groups. We met most recently to
address concerns that include the following:
• Workflow and Release Bumping Issues: CATs relies on calls to
specific versions of CC API to run successfully. The CAPI CI currently
bumps cloud_controller_ng within capi-release, and on a successful run
of CATs (which lives on cf-release develop), capi-release then gets
bumped in cf-release develop. This leads to a "chicken and egg"
scenario, in that when changes are made to both CATs and CC, both
cf-release and capi pipelines are broken until a manual fix is made.
• In a world where cf-release no longer exists, how do we ensure CATs
uses the correct versions of the CF CLI and CC API?
tl;dr: Though not all of the CATs test the CC API explicitly, they all
make use of the CC API and therefore make CC a choke point. Having
CATs as a separate release may be a good idea, but doesn't necessarily
address the strong coupling issues.
We'd like to know if there are specific cases in which this will be
detrimental to other projects' workflows before moving forward.
On Wed, Mar 23, 2016 at 4:08 AM, Michael Fraenkel
<michael.fraenkel(a)gmail.com <mailto:michael.fraenkel(a)gmail.com>> wrote:
Can you explain why CATs should live under CAPI?
CATs now represents acceptance test for all of the Cloud Foundry
function driven mainly via the CLI.
Tieing this to any one release seems a bit artificial.
CATs is about testing the aggregation of all the various releases
not just CAPI.
I am more surprised that it wasn't suggested to be its own release
which dictated the versions for all releases that had passed.
- Michael
On 3/22/16 3:05 AM, Utako Ueda wrote:
Hi all,
The CAPI team would like to take ownership of the CF acceptance
tests <https://github.com/cloudfoundry/cf-acceptance-tests/> and
include them as part of CAPI release
<http://github.com/cloudfoundry/capi-release>.
This solves several pain points we've experienced over the last
few months, mainly due to the strong coupling between CATs and
the Cloud Controller API.
Our plan is to submodule the CATs repo in to CAPI release, and
bump CATs on a successful run through the CAPI ci pipeline. At
this point, CAPI release will be bumped in cf-release develop,
which other teams will consume for their own testing purposes.
We hope to make these changes in the near future as we wrap up
our release extraction from cf-release. We'd like to know if any
teams have any concerns about this before we proceed, so do let
us know as soon as possible so we can address them.
Thanks,
Utako, CF CAPI Team
|
|
|
To address your concerns, Mike:
I think there's a solution here that won't hinder people from contributing
to CATs. Contributing to CATs won't necessarily change from the perspective
of those who already have push access. Your changes would make their way
through the CAPI pipeline first, get bumped within capi-release in a manner
similar to how we currently bump cloud_controller_ng, before getting bumped
to cf-release develop. This means you could potentially fork just the CATs
repo and push to it. Our CI pipeline would take the responsibility of
pointing capi-release to the correct SHA of its submoduled CATs.
toggle quoted message
Show quoted text
On Wed, Mar 23, 2016 at 12:03 PM, Mike Youngstrom <youngm(a)gmail.com> wrote: I don't have any show stopping cases. But, my 2 main concerns are:
* Conceptually speaking if the CATS are for all will making them part of
capi hinder other teams from contributing to it?
* Having the CATs part of CAPI will require me to fork CAPI when I need to
fix/customize a test specific to our environment, something we frequently
need to do. Long term I'd prefer to fork a smaller more single purpose
release when these situations arise.
Mike
On Wed, Mar 23, 2016 at 12:43 PM, Utako Ueda <uueda(a)pivotal.io> wrote:
We had a series of meetings to figure out a good path for CATs that
ranged from multiple teams to smaller groups. We met most recently to
address concerns that include the following:
• Workflow and Release Bumping Issues: CATs relies on calls to specific
versions of CC API to run successfully. The CAPI CI currently bumps
cloud_controller_ng within capi-release, and on a successful run of CATs
(which lives on cf-release develop), capi-release then gets bumped in
cf-release develop. This leads to a "chicken and egg" scenario, in that
when changes are made to both CATs and CC, both cf-release and capi
pipelines are broken until a manual fix is made.
• In a world where cf-release no longer exists, how do we ensure CATs
uses the correct versions of the CF CLI and CC API?
tl;dr: Though not all of the CATs test the CC API explicitly, they all
make use of the CC API and therefore make CC a choke point. Having CATs as
a separate release may be a good idea, but doesn't necessarily address the
strong coupling issues.
We'd like to know if there are specific cases in which this will be
detrimental to other projects' workflows before moving forward.
On Wed, Mar 23, 2016 at 4:08 AM, Michael Fraenkel <
michael.fraenkel(a)gmail.com> wrote:
Can you explain why CATs should live under CAPI?
CATs now represents acceptance test for all of the Cloud Foundry function
driven mainly via the CLI.
Tieing this to any one release seems a bit artificial.
CATs is about testing the aggregation of all the various releases not
just CAPI.
I am more surprised that it wasn't suggested to be its own release which
dictated the versions for all releases that had passed.
- Michael
On 3/22/16 3:05 AM, Utako Ueda wrote:
Hi all,
The CAPI team would like to take ownership of the CF acceptance tests
<https://github.com/cloudfoundry/cf-acceptance-tests/> and include them
as part of CAPI release <http://github.com/cloudfoundry/capi-release>.
This solves several pain points we've experienced over the last few
months, mainly due to the strong coupling between CATs and the Cloud
Controller API.
Our plan is to submodule the CATs repo in to CAPI release, and bump CATs
on a successful run through the CAPI ci pipeline. At this point, CAPI
release will be bumped in cf-release develop, which other teams will
consume for their own testing purposes.
We hope to make these changes in the near future as we wrap up our
release extraction from cf-release. We'd like to know if any teams have any
concerns about this before we proceed, so do let us know as soon as
possible so we can address them.
Thanks,
Utako, CF CAPI Team
|
|
|
Matthew Sykes <matthew.sykes@...>
This has as much to do with finding the "right" place for the CATS as it
does anything else. The CATS are the end-to-end acceptance tests that are
associated with what used to be called `cf-release` - the thing that
(before we spread our bits across 20+ releases) actually represented what
Cloud Foundry was. Not only does it reflect the target environment, it
tries to ensure that the primary developer experience, the cf cli,
continues to function correctly.
Now that all of the things that used to be part of the coherent, versioned,
collection of components have been spread to the wind, it's probably best
to make the CATS an independent release too.
Basically, I also believe that it's inappropriate to pull them into the
CAPI releases; they have as much to do with the CLI, Diego, LAM, and
Routing teams as they do with CAPI.
toggle quoted message
Show quoted text
On Thu, Mar 24, 2016 at 11:48 AM, Utako Ueda <uueda(a)pivotal.io> wrote: To address your concerns, Mike:
I think there's a solution here that won't hinder people from contributing
to CATs. Contributing to CATs won't necessarily change from the perspective
of those who already have push access. Your changes would make their way
through the CAPI pipeline first, get bumped within capi-release in a manner
similar to how we currently bump cloud_controller_ng, before getting bumped
to cf-release develop. This means you could potentially fork just the CATs
repo and push to it. Our CI pipeline would take the responsibility of
pointing capi-release to the correct SHA of its submoduled CATs.
On Wed, Mar 23, 2016 at 12:03 PM, Mike Youngstrom <youngm(a)gmail.com>
wrote:
I don't have any show stopping cases. But, my 2 main concerns are:
* Conceptually speaking if the CATS are for all will making them part of
capi hinder other teams from contributing to it?
* Having the CATs part of CAPI will require me to fork CAPI when I need
to fix/customize a test specific to our environment, something we
frequently need to do. Long term I'd prefer to fork a smaller more single
purpose release when these situations arise.
Mike
On Wed, Mar 23, 2016 at 12:43 PM, Utako Ueda <uueda(a)pivotal.io> wrote:
We had a series of meetings to figure out a good path for CATs that
ranged from multiple teams to smaller groups. We met most recently to
address concerns that include the following:
• Workflow and Release Bumping Issues: CATs relies on calls to specific
versions of CC API to run successfully. The CAPI CI currently bumps
cloud_controller_ng within capi-release, and on a successful run of CATs
(which lives on cf-release develop), capi-release then gets bumped in
cf-release develop. This leads to a "chicken and egg" scenario, in that
when changes are made to both CATs and CC, both cf-release and capi
pipelines are broken until a manual fix is made.
• In a world where cf-release no longer exists, how do we ensure CATs
uses the correct versions of the CF CLI and CC API?
tl;dr: Though not all of the CATs test the CC API explicitly, they all
make use of the CC API and therefore make CC a choke point. Having CATs as
a separate release may be a good idea, but doesn't necessarily address the
strong coupling issues.
We'd like to know if there are specific cases in which this will be
detrimental to other projects' workflows before moving forward.
On Wed, Mar 23, 2016 at 4:08 AM, Michael Fraenkel <
michael.fraenkel(a)gmail.com> wrote:
Can you explain why CATs should live under CAPI?
CATs now represents acceptance test for all of the Cloud Foundry
function driven mainly via the CLI.
Tieing this to any one release seems a bit artificial.
CATs is about testing the aggregation of all the various releases not
just CAPI.
I am more surprised that it wasn't suggested to be its own release
which dictated the versions for all releases that had passed.
- Michael
On 3/22/16 3:05 AM, Utako Ueda wrote:
Hi all,
The CAPI team would like to take ownership of the CF acceptance tests
<https://github.com/cloudfoundry/cf-acceptance-tests/> and include
them as part of CAPI release
<http://github.com/cloudfoundry/capi-release>.
This solves several pain points we've experienced over the last few
months, mainly due to the strong coupling between CATs and the Cloud
Controller API.
Our plan is to submodule the CATs repo in to CAPI release, and bump
CATs on a successful run through the CAPI ci pipeline. At this point, CAPI
release will be bumped in cf-release develop, which other teams will
consume for their own testing purposes.
We hope to make these changes in the near future as we wrap up our
release extraction from cf-release. We'd like to know if any teams have any
concerns about this before we proceed, so do let us know as soon as
possible so we can address them.
Thanks,
Utako, CF CAPI Team
--
Matthew Sykes
matthew.sykes(a)gmail.com
|
|
|
