Hi all,

In the spirit of community, I'm going to ask this question outright and in the open - is anyone (*looks at VMware MAPBU*) working on a CRD and accompanying controller for CF on Kubernetes?

Anyone with a non-trivial amount of Cloud Foundry experience knows that whilst the imperative interface of the CF CLI is great for exploratory work, it's far from ideal for production environments. I make a point of telling folks in CF training courses that no human should be using the CLI in production, other than for debugging.

A declarative interface for CF has always been desirable, and its omission is going to become even more stark once running CF on Kubernetes is the norm. We've had cf-mgmt which is close, but involves more steps than is ideal.

It'd be great to have one or more CRDs that represent the state of a Cloud Foundry, and accompanying controllers that converge upon that state. When I talk about state, I mean things like orgs, spaces, service broker registrations, service instances, roles and so on.

Four years ago I started work on a project called cf-converger to diff and converge a Cloud Foundry instance on a state declared in YAML. I got distracted with running a business and doing billable work, so whilst the concept was proven, it never really got to a state of usefulness. Plus, I seemed to spend half my coding time just writing excessively-verbose chained method calls on the official Java CF client :D 

I see such a solution as an inevitability in the Kubernetes-based future. Whilst I'm personally tempted to resurrect the project, it makes very little sense for a company like EngineerBetter to divert resources from billable work to a project that is needed, but can't be monetised.

Are any of the big players working on something similar? Is it something we should all be thinking and talking about? Is there any point in the community working on such a solution, or are one of the bigger companies secretly working on something that would supersede any community effort?

Regards,

Daniel 'Deejay' Jones - CEO

+44 (0)79 8000 9153

@DanielJonesEB

EngineerBetter Ltd - More than cloud platform specialists

Hi Daniel, 
The CF-K8s Networking team is currently working on a solution using Route CRD- to introduce a “Route” custom resource and have Cloud Controller make/update these directly as part of the `cf map-route` and `cf unmap-route` workflows. Happy to discuss this further in our Cloud-Foundry Slack- #networking group. Regards, Keshav Sharma Product Manager | CF-K8s-Networking VMware

On Wed, Apr 15, 2020 at 3:01 AM Daniel Jones <daniel.jones@...> wrote:

Hi all,

In the spirit of community, I'm going to ask this question outright and in the open - is anyone (*looks at VMware MAPBU*) working on a CRD and accompanying controller for CF on Kubernetes?

Anyone with a non-trivial amount of Cloud Foundry experience knows that whilst the imperative interface of the CF CLI is great for exploratory work, it's far from ideal for production environments. I make a point of telling folks in CF training courses that no human should be using the CLI in production, other than for debugging.

A declarative interface for CF has always been desirable, and its omission is going to become even more stark once running CF on Kubernetes is the norm. We've had cf-mgmt which is close, but involves more steps than is ideal.

It'd be great to have one or more CRDs that represent the state of a Cloud Foundry, and accompanying controllers that converge upon that state. When I talk about state, I mean things like orgs, spaces, service broker registrations, service instances, roles and so on.

Four years ago I started work on a project called cf-converger to diff and converge a Cloud Foundry instance on a state declared in YAML. I got distracted with running a business and doing billable work, so whilst the concept was proven, it never really got to a state of usefulness. Plus, I seemed to spend half my coding time just writing excessively-verbose chained method calls on the official Java CF client :D 

I see such a solution as an inevitability in the Kubernetes-based future. Whilst I'm personally tempted to resurrect the project, it makes very little sense for a company like EngineerBetter to divert resources from billable work to a project that is needed, but can't be monetised.

Are any of the big players working on something similar? Is it something we should all be thinking and talking about? Is there any point in the community working on such a solution, or are one of the bigger companies secretly working on something that would supersede any community effort?

Regards,

Daniel 'Deejay' Jones - CEO

+44 (0)79 8000 9153

@DanielJonesEB

EngineerBetter Ltd - More than cloud platform specialists

Zach, am I correct in thinking that the current approach is the have clients interact with CloudController, have CloudController persist to CCDB as per usual, and then CloudController post stuff to CRDs in the Kube API?

Yes, this is the current approach for the CRDs that Cloud Controller interacts with now. There are also reconciliation loops that we run for these resources analogous to how we've always reconciled CCDB state with Diego.

I think this method has developed organically from our goal of preserving in-place workflows via having the same API surface, allowing cli to function without changes, while trying to consume existing functionality in k8s. I think the longer term goal here is to review that usage and be more intentional about how we expect CRDs to fit into the system as a whole.

-Zach

On Thu, Apr 16, 2020 at 4:10 AM Daniel Jones <daniel.jones@...> wrote:

Thanks for that folks - especially for such a long and detailed response, Zach. Much appreciated.

It's great that folks are already thinking about this.

Zach, am I correct in thinking that the current approach is the have clients interact with CloudController, have CloudController persist to CCDB as per usual, and then CloudController post stuff to CRDs in the Kube API?

I'd kinda imagined things going the other way around, introducing the CRDs as a new user interface, and have the controllers do the diff and then post/put to CloudController. The intention here would be saving Kubes-native folks (and CI servers!) from having to deal with the imperative interface. I was working on the assumption that all CF concepts would be exposed as CRDs.

From a CAPI perspective, I've been thinking more about representing apps and push as CRDs, and I believe the Eirini team has some thoughts in this area as well.

I'd always dodged this area in cf-converger as it was the most complicated, and the biggest break from the current CF experience :)

For example when we look at orgs/spaces - we have to wonder if those are even resources we want to represent as CRDs, when k8s already has a namespace construct.

I can imagine that the logical concept of orgs may one day map onto distinct Kubernetes namespaces, but I hadn't imagined that orgs would in any way disappear, or not be configurable via the Kubernetes approach.

It'd be great to hear what others think on all of this.

Thanks again for sharing, Keshav and Zach.

Regards,

Daniel 'Deejay' Jones - CEO

+44 (0)79 8000 9153

@DanielJonesEB

EngineerBetter Ltd - More than cloud platform specialists

On Wed, 15 Apr 2020 at 23:45, Zach Robinson <zrobinson@...> wrote:

Hey Daniel,

Thanks for starting a convo. The timing couldn't be better. I can share some of the thinking going on in both CAPI and VMware land. 

But first and foremost I want to address the question: "Is there any point in the community working on such a solution, or are one of the bigger companies secretly working on something that would supersede any community effort?" - yes, yes, a thousand times yes to community collaboration!

The topic of CRDs is kind of huge. It's really great to hear your focus is on things like orgs/spaces. From a CAPI perspective, I've been thinking more about representing apps and push as CRDs, and I believe the Eirini team has some thoughts in this area as well. There's a lot of ground to cover clearly.

In terms of things that are in flight now.

• Keshav kindly shared that networking is looking to provide Route CRDs.
• Networking is also exploring providing Security Groups as CRDs.
• CAPI is integrating kpack into CF to provide buildpack staging. This is done by generating kpack CRDs.

Currently these CRDs are being created and owned by the Cloud Controller, which enables us to present backward compatible APIs to our CLI and UI like stratos, while moving the implementation down into controllers fronted by CRDs.  There's an obvious potential to interact with k8s directly in the future, but the current use case is to be used by CC. Except for kpack which is an excellent stand alone tool already :).  

These efforts are great and are allowing us to begin a path towards using existing k8s tooling and projects to provide Cloud Foundry outcomes, but obviously there's a lot further to go.

From VMware's side, over the last couple weeks, there has been an effort to explore what it might look like to use CRDs entirely as a source of truth for *all* CF data, in place of a SQL database, as a learning exercise, not necessarily a desired outcome. The primary goal of that effort was to start the convo that we're having now, and to be able to contribute in a meaningful way.  Note there are no VMware-based plans to dump a fully fledged proposal. We need to start from a premise as a community of understanding how/why/and which resources are valuable as CRDs.

For example when we look at orgs/spaces - we have to wonder if those are even resources we want to represent as CRDs, when k8s already has a namespace construct.

I think we'll need to start some dedicated working document to collaborate in. I had planned on facilitating that after some of the investigation that VMware has wraps up, but I don't think that's a reason to hold off if you or somebody else in the community has thoughts they want to start getting down. CAPI and other teams can contribute to an existing doc if that makes sense. Any other thoughts you'd prefer for next steps?

Looking forward to a continued conversation.

-Zach

On Wed, Apr 15, 2020 at 9:56 AM Keshav Sharma <ksharma@...> wrote:

Hi Daniel, 
The CF-K8s Networking team is currently working on a solution using Route CRD- to introduce a “Route” custom resource and have Cloud Controller make/update these directly as part of the `cf map-route` and `cf unmap-route` workflows. Happy to discuss this further in our Cloud-Foundry Slack- #networking group. Regards, Keshav Sharma Product Manager | CF-K8s-Networking VMware

On Wed, Apr 15, 2020 at 3:01 AM Daniel Jones <daniel.jones@...> wrote:

Hi all,

In the spirit of community, I'm going to ask this question outright and in the open - is anyone (*looks at VMware MAPBU*) working on a CRD and accompanying controller for CF on Kubernetes?

Anyone with a non-trivial amount of Cloud Foundry experience knows that whilst the imperative interface of the CF CLI is great for exploratory work, it's far from ideal for production environments. I make a point of telling folks in CF training courses that no human should be using the CLI in production, other than for debugging.

A declarative interface for CF has always been desirable, and its omission is going to become even more stark once running CF on Kubernetes is the norm. We've had cf-mgmt which is close, but involves more steps than is ideal.

It'd be great to have one or more CRDs that represent the state of a Cloud Foundry, and accompanying controllers that converge upon that state. When I talk about state, I mean things like orgs, spaces, service broker registrations, service instances, roles and so on.

Four years ago I started work on a project called cf-converger to diff and converge a Cloud Foundry instance on a state declared in YAML. I got distracted with running a business and doing billable work, so whilst the concept was proven, it never really got to a state of usefulness. Plus, I seemed to spend half my coding time just writing excessively-verbose chained method calls on the official Java CF client :D 

I see such a solution as an inevitability in the Kubernetes-based future. Whilst I'm personally tempted to resurrect the project, it makes very little sense for a company like EngineerBetter to divert resources from billable work to a project that is needed, but can't be monetised.

Are any of the big players working on something similar? Is it something we should all be thinking and talking about? Is there any point in the community working on such a solution, or are one of the bigger companies secretly working on something that would supersede any community effort?

Regards,

Daniel 'Deejay' Jones - CEO

+44 (0)79 8000 9153

@DanielJonesEB

EngineerBetter Ltd - More than cloud platform specialists

Hey Daniel,

Wanted to follow up. There's now a doc to start building out our understanding of CRDs here https://docs.google.com/document/d/1mIMH4uEtIJkZjVa5mKWJoIAIeVyM4gVqO-Cx5JtYgtE/edit?usp=sharing

Shared with the mailing list under topic "Exploring CRDs in CF".  Would love to get your thoughts and feedback.

-Zach

On Thu, Apr 16, 2020 at 4:09 PM Zach Robinson <zrobinson@...> wrote:

Zach, am I correct in thinking that the current approach is the have clients interact with CloudController, have CloudController persist to CCDB as per usual, and then CloudController post stuff to CRDs in the Kube API?

Yes, this is the current approach for the CRDs that Cloud Controller interacts with now. There are also reconciliation loops that we run for these resources analogous to how we've always reconciled CCDB state with Diego.

I think this method has developed organically from our goal of preserving in-place workflows via having the same API surface, allowing cli to function without changes, while trying to consume existing functionality in k8s. I think the longer term goal here is to review that usage and be more intentional about how we expect CRDs to fit into the system as a whole.

-Zach

On Thu, Apr 16, 2020 at 4:10 AM Daniel Jones <daniel.jones@...> wrote:

Thanks for that folks - especially for such a long and detailed response, Zach. Much appreciated.

It's great that folks are already thinking about this.

Zach, am I correct in thinking that the current approach is the have clients interact with CloudController, have CloudController persist to CCDB as per usual, and then CloudController post stuff to CRDs in the Kube API?

I'd kinda imagined things going the other way around, introducing the CRDs as a new user interface, and have the controllers do the diff and then post/put to CloudController. The intention here would be saving Kubes-native folks (and CI servers!) from having to deal with the imperative interface. I was working on the assumption that all CF concepts would be exposed as CRDs.

From a CAPI perspective, I've been thinking more about representing apps and push as CRDs, and I believe the Eirini team has some thoughts in this area as well.

I'd always dodged this area in cf-converger as it was the most complicated, and the biggest break from the current CF experience :)

For example when we look at orgs/spaces - we have to wonder if those are even resources we want to represent as CRDs, when k8s already has a namespace construct.

I can imagine that the logical concept of orgs may one day map onto distinct Kubernetes namespaces, but I hadn't imagined that orgs would in any way disappear, or not be configurable via the Kubernetes approach.

It'd be great to hear what others think on all of this.

Thanks again for sharing, Keshav and Zach.

Regards,

Daniel 'Deejay' Jones - CEO

+44 (0)79 8000 9153

@DanielJonesEB

EngineerBetter Ltd - More than cloud platform specialists

On Wed, 15 Apr 2020 at 23:45, Zach Robinson <zrobinson@...> wrote:

Hey Daniel,

Thanks for starting a convo. The timing couldn't be better. I can share some of the thinking going on in both CAPI and VMware land. 

But first and foremost I want to address the question: "Is there any point in the community working on such a solution, or are one of the bigger companies secretly working on something that would supersede any community effort?" - yes, yes, a thousand times yes to community collaboration!

The topic of CRDs is kind of huge. It's really great to hear your focus is on things like orgs/spaces. From a CAPI perspective, I've been thinking more about representing apps and push as CRDs, and I believe the Eirini team has some thoughts in this area as well. There's a lot of ground to cover clearly.

In terms of things that are in flight now.

• Keshav kindly shared that networking is looking to provide Route CRDs.
• Networking is also exploring providing Security Groups as CRDs.
• CAPI is integrating kpack into CF to provide buildpack staging. This is done by generating kpack CRDs.

Currently these CRDs are being created and owned by the Cloud Controller, which enables us to present backward compatible APIs to our CLI and UI like stratos, while moving the implementation down into controllers fronted by CRDs.  There's an obvious potential to interact with k8s directly in the future, but the current use case is to be used by CC. Except for kpack which is an excellent stand alone tool already :).  

These efforts are great and are allowing us to begin a path towards using existing k8s tooling and projects to provide Cloud Foundry outcomes, but obviously there's a lot further to go.

From VMware's side, over the last couple weeks, there has been an effort to explore what it might look like to use CRDs entirely as a source of truth for *all* CF data, in place of a SQL database, as a learning exercise, not necessarily a desired outcome. The primary goal of that effort was to start the convo that we're having now, and to be able to contribute in a meaningful way.  Note there are no VMware-based plans to dump a fully fledged proposal. We need to start from a premise as a community of understanding how/why/and which resources are valuable as CRDs.

For example when we look at orgs/spaces - we have to wonder if those are even resources we want to represent as CRDs, when k8s already has a namespace construct.

I think we'll need to start some dedicated working document to collaborate in. I had planned on facilitating that after some of the investigation that VMware has wraps up, but I don't think that's a reason to hold off if you or somebody else in the community has thoughts they want to start getting down. CAPI and other teams can contribute to an existing doc if that makes sense. Any other thoughts you'd prefer for next steps?

Looking forward to a continued conversation.

-Zach

On Wed, Apr 15, 2020 at 9:56 AM Keshav Sharma <ksharma@...> wrote:

Hi Daniel, 
The CF-K8s Networking team is currently working on a solution using Route CRD- to introduce a “Route” custom resource and have Cloud Controller make/update these directly as part of the `cf map-route` and `cf unmap-route` workflows. Happy to discuss this further in our Cloud-Foundry Slack- #networking group. Regards, Keshav Sharma Product Manager | CF-K8s-Networking VMware

On Wed, Apr 15, 2020 at 3:01 AM Daniel Jones <daniel.jones@...> wrote:

Hi all,

In the spirit of community, I'm going to ask this question outright and in the open - is anyone (*looks at VMware MAPBU*) working on a CRD and accompanying controller for CF on Kubernetes?

Anyone with a non-trivial amount of Cloud Foundry experience knows that whilst the imperative interface of the CF CLI is great for exploratory work, it's far from ideal for production environments. I make a point of telling folks in CF training courses that no human should be using the CLI in production, other than for debugging.

A declarative interface for CF has always been desirable, and its omission is going to become even more stark once running CF on Kubernetes is the norm. We've had cf-mgmt which is close, but involves more steps than is ideal.

It'd be great to have one or more CRDs that represent the state of a Cloud Foundry, and accompanying controllers that converge upon that state. When I talk about state, I mean things like orgs, spaces, service broker registrations, service instances, roles and so on.

Four years ago I started work on a project called cf-converger to diff and converge a Cloud Foundry instance on a state declared in YAML. I got distracted with running a business and doing billable work, so whilst the concept was proven, it never really got to a state of usefulness. Plus, I seemed to spend half my coding time just writing excessively-verbose chained method calls on the official Java CF client :D 

I see such a solution as an inevitability in the Kubernetes-based future. Whilst I'm personally tempted to resurrect the project, it makes very little sense for a company like EngineerBetter to divert resources from billable work to a project that is needed, but can't be monetised.

Are any of the big players working on something similar? Is it something we should all be thinking and talking about? Is there any point in the community working on such a solution, or are one of the bigger companies secretly working on something that would supersede any community effort?

Regards,

Daniel 'Deejay' Jones - CEO

+44 (0)79 8000 9153

@DanielJonesEB

EngineerBetter Ltd - More than cloud platform specialists

Hey Daniel,

Wanted to follow up. There's now a doc to start building out our understanding of CRDs here https://docs.google.com/document/d/1mIMH4uEtIJkZjVa5mKWJoIAIeVyM4gVqO-Cx5JtYgtE/edit?usp=sharing

Shared with the mailing list under topic "Exploring CRDs in CF".  Would love to get your thoughts and feedback.

-Zach

On Thu, Apr 16, 2020 at 4:09 PM Zach Robinson <zrobinson@...> wrote:

Zach, am I correct in thinking that the current approach is the have clients interact with CloudController, have CloudController persist to CCDB as per usual, and then CloudController post stuff to CRDs in the Kube API?

Yes, this is the current approach for the CRDs that Cloud Controller interacts with now. There are also reconciliation loops that we run for these resources analogous to how we've always reconciled CCDB state with Diego.

I think this method has developed organically from our goal of preserving in-place workflows via having the same API surface, allowing cli to function without changes, while trying to consume existing functionality in k8s. I think the longer term goal here is to review that usage and be more intentional about how we expect CRDs to fit into the system as a whole.

-Zach

On Thu, Apr 16, 2020 at 4:10 AM Daniel Jones <daniel.jones@...> wrote:

Thanks for that folks - especially for such a long and detailed response, Zach. Much appreciated.

It's great that folks are already thinking about this.

Zach, am I correct in thinking that the current approach is the have clients interact with CloudController, have CloudController persist to CCDB as per usual, and then CloudController post stuff to CRDs in the Kube API?

I'd kinda imagined things going the other way around, introducing the CRDs as a new user interface, and have the controllers do the diff and then post/put to CloudController. The intention here would be saving Kubes-native folks (and CI servers!) from having to deal with the imperative interface. I was working on the assumption that all CF concepts would be exposed as CRDs.

From a CAPI perspective, I've been thinking more about representing apps and push as CRDs, and I believe the Eirini team has some thoughts in this area as well.

I'd always dodged this area in cf-converger as it was the most complicated, and the biggest break from the current CF experience :)

For example when we look at orgs/spaces - we have to wonder if those are even resources we want to represent as CRDs, when k8s already has a namespace construct.

I can imagine that the logical concept of orgs may one day map onto distinct Kubernetes namespaces, but I hadn't imagined that orgs would in any way disappear, or not be configurable via the Kubernetes approach.

It'd be great to hear what others think on all of this.

Thanks again for sharing, Keshav and Zach.

Regards,

Daniel 'Deejay' Jones - CEO

+44 (0)79 8000 9153

@DanielJonesEB

EngineerBetter Ltd - More than cloud platform specialists

On Wed, 15 Apr 2020 at 23:45, Zach Robinson <zrobinson@...> wrote:

Hey Daniel,

Thanks for starting a convo. The timing couldn't be better. I can share some of the thinking going on in both CAPI and VMware land. 

But first and foremost I want to address the question: "Is there any point in the community working on such a solution, or are one of the bigger companies secretly working on something that would supersede any community effort?" - yes, yes, a thousand times yes to community collaboration!

The topic of CRDs is kind of huge. It's really great to hear your focus is on things like orgs/spaces. From a CAPI perspective, I've been thinking more about representing apps and push as CRDs, and I believe the Eirini team has some thoughts in this area as well. There's a lot of ground to cover clearly.

In terms of things that are in flight now.

• Keshav kindly shared that networking is looking to provide Route CRDs.
• Networking is also exploring providing Security Groups as CRDs.
• CAPI is integrating kpack into CF to provide buildpack staging. This is done by generating kpack CRDs.

Currently these CRDs are being created and owned by the Cloud Controller, which enables us to present backward compatible APIs to our CLI and UI like stratos, while moving the implementation down into controllers fronted by CRDs.  There's an obvious potential to interact with k8s directly in the future, but the current use case is to be used by CC. Except for kpack which is an excellent stand alone tool already :).  

These efforts are great and are allowing us to begin a path towards using existing k8s tooling and projects to provide Cloud Foundry outcomes, but obviously there's a lot further to go.

From VMware's side, over the last couple weeks, there has been an effort to explore what it might look like to use CRDs entirely as a source of truth for *all* CF data, in place of a SQL database, as a learning exercise, not necessarily a desired outcome. The primary goal of that effort was to start the convo that we're having now, and to be able to contribute in a meaningful way.  Note there are no VMware-based plans to dump a fully fledged proposal. We need to start from a premise as a community of understanding how/why/and which resources are valuable as CRDs.

For example when we look at orgs/spaces - we have to wonder if those are even resources we want to represent as CRDs, when k8s already has a namespace construct.

I think we'll need to start some dedicated working document to collaborate in. I had planned on facilitating that after some of the investigation that VMware has wraps up, but I don't think that's a reason to hold off if you or somebody else in the community has thoughts they want to start getting down. CAPI and other teams can contribute to an existing doc if that makes sense. Any other thoughts you'd prefer for next steps?

Looking forward to a continued conversation.

-Zach

On Wed, Apr 15, 2020 at 9:56 AM Keshav Sharma <ksharma@...> wrote:

Hi Daniel, 
The CF-K8s Networking team is currently working on a solution using Route CRD- to introduce a “Route” custom resource and have Cloud Controller make/update these directly as part of the `cf map-route` and `cf unmap-route` workflows. Happy to discuss this further in our Cloud-Foundry Slack- #networking group. Regards, Keshav Sharma Product Manager | CF-K8s-Networking VMware

On Wed, Apr 15, 2020 at 3:01 AM Daniel Jones <daniel.jones@...> wrote:

Hi all,

In the spirit of community, I'm going to ask this question outright and in the open - is anyone (*looks at VMware MAPBU*) working on a CRD and accompanying controller for CF on Kubernetes?

Anyone with a non-trivial amount of Cloud Foundry experience knows that whilst the imperative interface of the CF CLI is great for exploratory work, it's far from ideal for production environments. I make a point of telling folks in CF training courses that no human should be using the CLI in production, other than for debugging.

A declarative interface for CF has always been desirable, and its omission is going to become even more stark once running CF on Kubernetes is the norm. We've had cf-mgmt which is close, but involves more steps than is ideal.

It'd be great to have one or more CRDs that represent the state of a Cloud Foundry, and accompanying controllers that converge upon that state. When I talk about state, I mean things like orgs, spaces, service broker registrations, service instances, roles and so on.

Four years ago I started work on a project called cf-converger to diff and converge a Cloud Foundry instance on a state declared in YAML. I got distracted with running a business and doing billable work, so whilst the concept was proven, it never really got to a state of usefulness. Plus, I seemed to spend half my coding time just writing excessively-verbose chained method calls on the official Java CF client :D 

I see such a solution as an inevitability in the Kubernetes-based future. Whilst I'm personally tempted to resurrect the project, it makes very little sense for a company like EngineerBetter to divert resources from billable work to a project that is needed, but can't be monetised.

Are any of the big players working on something similar? Is it something we should all be thinking and talking about? Is there any point in the community working on such a solution, or are one of the bigger companies secretly working on something that would supersede any community effort?

Regards,

Daniel 'Deejay' Jones - CEO

+44 (0)79 8000 9153

@DanielJonesEB

EngineerBetter Ltd - More than cloud platform specialists

Just as inspiration, and maybe spike implémentation, a déclarative approach based on terraform HCL is probably a good way to start

https://github.com/cloudfoundry-community/terraform-provider-cf/blob/master/README.md

Le mar. 28 avr. 2020 à 19:18, Zach Robinson <zrobinson@...> a écrit :

Hey Daniel,

Wanted to follow up. There's now a doc to start building out our understanding of CRDs here https://docs.google.com/document/d/1mIMH4uEtIJkZjVa5mKWJoIAIeVyM4gVqO-Cx5JtYgtE/edit?usp=sharing

Shared with the mailing list under topic "Exploring CRDs in CF".  Would love to get your thoughts and feedback.

-Zach

On Thu, Apr 16, 2020 at 4:09 PM Zach Robinson <zrobinson@...> wrote:

Zach, am I correct in thinking that the current approach is the have clients interact with CloudController, have CloudController persist to CCDB as per usual, and then CloudController post stuff to CRDs in the Kube API?

Yes, this is the current approach for the CRDs that Cloud Controller interacts with now. There are also reconciliation loops that we run for these resources analogous to how we've always reconciled CCDB state with Diego.

I think this method has developed organically from our goal of preserving in-place workflows via having the same API surface, allowing cli to function without changes, while trying to consume existing functionality in k8s. I think the longer term goal here is to review that usage and be more intentional about how we expect CRDs to fit into the system as a whole.

-Zach

On Thu, Apr 16, 2020 at 4:10 AM Daniel Jones <daniel.jones@...> wrote:

Thanks for that folks - especially for such a long and detailed response, Zach. Much appreciated.

It's great that folks are already thinking about this.

Zach, am I correct in thinking that the current approach is the have clients interact with CloudController, have CloudController persist to CCDB as per usual, and then CloudController post stuff to CRDs in the Kube API?

I'd kinda imagined things going the other way around, introducing the CRDs as a new user interface, and have the controllers do the diff and then post/put to CloudController. The intention here would be saving Kubes-native folks (and CI servers!) from having to deal with the imperative interface. I was working on the assumption that all CF concepts would be exposed as CRDs.

From a CAPI perspective, I've been thinking more about representing apps and push as CRDs, and I believe the Eirini team has some thoughts in this area as well.

I'd always dodged this area in cf-converger as it was the most complicated, and the biggest break from the current CF experience :)

For example when we look at orgs/spaces - we have to wonder if those are even resources we want to represent as CRDs, when k8s already has a namespace construct.

I can imagine that the logical concept of orgs may one day map onto distinct Kubernetes namespaces, but I hadn't imagined that orgs would in any way disappear, or not be configurable via the Kubernetes approach.

It'd be great to hear what others think on all of this.

Thanks again for sharing, Keshav and Zach.

Regards,

Daniel 'Deejay' Jones - CEO

+44 (0)79 8000 9153

@DanielJonesEB

EngineerBetter Ltd - More than cloud platform specialists

On Wed, 15 Apr 2020 at 23:45, Zach Robinson <zrobinson@...> wrote:

Hey Daniel,

Thanks for starting a convo. The timing couldn't be better. I can share some of the thinking going on in both CAPI and VMware land. 

But first and foremost I want to address the question: "Is there any point in the community working on such a solution, or are one of the bigger companies secretly working on something that would supersede any community effort?" - yes, yes, a thousand times yes to community collaboration!

The topic of CRDs is kind of huge. It's really great to hear your focus is on things like orgs/spaces. From a CAPI perspective, I've been thinking more about representing apps and push as CRDs, and I believe the Eirini team has some thoughts in this area as well. There's a lot of ground to cover clearly.

In terms of things that are in flight now.

• Keshav kindly shared that networking is looking to provide Route CRDs.
• Networking is also exploring providing Security Groups as CRDs.
• CAPI is integrating kpack into CF to provide buildpack staging. This is done by generating kpack CRDs.

Currently these CRDs are being created and owned by the Cloud Controller, which enables us to present backward compatible APIs to our CLI and UI like stratos, while moving the implementation down into controllers fronted by CRDs.  There's an obvious potential to interact with k8s directly in the future, but the current use case is to be used by CC. Except for kpack which is an excellent stand alone tool already :).  

These efforts are great and are allowing us to begin a path towards using existing k8s tooling and projects to provide Cloud Foundry outcomes, but obviously there's a lot further to go.

From VMware's side, over the last couple weeks, there has been an effort to explore what it might look like to use CRDs entirely as a source of truth for *all* CF data, in place of a SQL database, as a learning exercise, not necessarily a desired outcome. The primary goal of that effort was to start the convo that we're having now, and to be able to contribute in a meaningful way.  Note there are no VMware-based plans to dump a fully fledged proposal. We need to start from a premise as a community of understanding how/why/and which resources are valuable as CRDs.

For example when we look at orgs/spaces - we have to wonder if those are even resources we want to represent as CRDs, when k8s already has a namespace construct.

I think we'll need to start some dedicated working document to collaborate in. I had planned on facilitating that after some of the investigation that VMware has wraps up, but I don't think that's a reason to hold off if you or somebody else in the community has thoughts they want to start getting down. CAPI and other teams can contribute to an existing doc if that makes sense. Any other thoughts you'd prefer for next steps?

Looking forward to a continued conversation.

-Zach

On Wed, Apr 15, 2020 at 9:56 AM Keshav Sharma <ksharma@...> wrote:

Hi Daniel, 
The CF-K8s Networking team is currently working on a solution using Route CRD- to introduce a “Route” custom resource and have Cloud Controller make/update these directly as part of the `cf map-route` and `cf unmap-route` workflows. Happy to discuss this further in our Cloud-Foundry Slack- #networking group. Regards, Keshav Sharma Product Manager | CF-K8s-Networking VMware

On Wed, Apr 15, 2020 at 3:01 AM Daniel Jones <daniel.jones@...> wrote:

Hi all,

In the spirit of community, I'm going to ask this question outright and in the open - is anyone (*looks at VMware MAPBU*) working on a CRD and accompanying controller for CF on Kubernetes?

Anyone with a non-trivial amount of Cloud Foundry experience knows that whilst the imperative interface of the CF CLI is great for exploratory work, it's far from ideal for production environments. I make a point of telling folks in CF training courses that no human should be using the CLI in production, other than for debugging.

A declarative interface for CF has always been desirable, and its omission is going to become even more stark once running CF on Kubernetes is the norm. We've had cf-mgmt which is close, but involves more steps than is ideal.

It'd be great to have one or more CRDs that represent the state of a Cloud Foundry, and accompanying controllers that converge upon that state. When I talk about state, I mean things like orgs, spaces, service broker registrations, service instances, roles and so on.

Four years ago I started work on a project called cf-converger to diff and converge a Cloud Foundry instance on a state declared in YAML. I got distracted with running a business and doing billable work, so whilst the concept was proven, it never really got to a state of usefulness. Plus, I seemed to spend half my coding time just writing excessively-verbose chained method calls on the official Java CF client :D 

I see such a solution as an inevitability in the Kubernetes-based future. Whilst I'm personally tempted to resurrect the project, it makes very little sense for a company like EngineerBetter to divert resources from billable work to a project that is needed, but can't be monetised.

Are any of the big players working on something similar? Is it something we should all be thinking and talking about? Is there any point in the community working on such a solution, or are one of the bigger companies secretly working on something that would supersede any community effort?

Regards,

Daniel 'Deejay' Jones - CEO

+44 (0)79 8000 9153

@DanielJonesEB

EngineerBetter Ltd - More than cloud platform specialists