Date   

Introducing Log-Cache - A Horizontally Scalable Self Pruning Cache for Observability (built by #loggregator)

Adam Hevenor
 

Problem Summary

The current implementation of recent logs and container metrics on CF Application Runtime is slow and increases latency as operators scale up their deployments. Over the past year we have patched with concurrent processing and circuit breakers but the problem masked a larger problem we set out to fix this summer with an improved persistance architecture

Solution Summary

Creating an improved persistance layer has lead us to discover many additional advantages of offering a new restful API. To demonstrate these advantages we developed a log-cache-cli plugin that effectively replaces the cf logs command with a cf tail command. The tail command borrows many of it's flags and UX from the popular unix tail command, and allows you to follow, and window the cache in ways logs and logs --recent did not. 

Using Log Cache and Service Level Objectives

We have a couple of months of operation experience with this new release and we are working through some final packaging steps to include this in cf-deployment. For now you can include it in cf-deployment using an ops file and learn about the operation track record on our Product FAQ

Feedback and next steps

This feature is paving the way for a new developer UX as well as some new loggregator improvements we are considering and I will post soon. If you have feedback or ideas about using Log-Cache and/or the CLI we'd love to hear from you here or in slack. 


Re: CF CLI v6.35.0 Release Today - service instance sharing; client credentials

Dr Nic Williams <drnicwilliams@...>
 

I’m not sure what I’m asking is what you’re saying; I think you’re confirming the current implementation - a service offering cannot be made shareable unless the vendor is aware of the concept of shareable services and ships a future version to modifies their /v2/catalog JSON. I’m asking for admins to be able to make up their own decisions without requiring service brokers to be modified.

Question: when a service instance is shared with another space, does a broker API get invoked? Or would a broker only discover this ( if it cared) when the new binding API requests come in with different org/space GUIDs?


From: cf-dev@... <cf-dev@...> on behalf of Dr Nic Williams <drnicwilliams@...>
Sent: Monday, March 12, 2018 6:18:18 PM
To: cf-dev@...
Subject: Re: [cf-dev] CF CLI v6.35.0 Release Today - service instance sharing; client credentials
 
Jay, to confirm, an admin will be able to share a service broker/service offering even if it’s not explicitly supported by the /v2/catalog?


From: cf-dev@... <cf-dev@...> on behalf of Jay Badenhope <jbadenhope@...>
Sent: Monday, March 12, 2018 6:16:51 PM
To: cf-dev@...
Subject: Re: [cf-dev] CF CLI v6.35.0 Release Today - service instance sharing; client credentials
 
Hi Dr Nic,
Building on Denise's response, we also empower the admin to enable/restrict sharing. There are two settings that must be true in order to enable service instance sharing:
1. At the global level: "To enable service instance sharing, an administrator must enable the `service_instance_sharing` flag." https://docs.cloudfoundry.org/devguide/services/sharing-instances.html#enabling
2. At the service level, as you mentioned, "Service brokers must explicitly enable service instance sharing by setting a flag in their service-level metadata object." https://docs.cloudfoundry.org/services/enable-sharing.html#enabling

Matthias,
I'm going to connect with my UAA colleagues and make sure we have a good answer to your question.

Jay

On Mon, Mar 12, 2018 at 3:24 AM, <dyu@...> wrote:
The decision to have service authors opt in was to account for the fact that some services may not be shareable out-of-the-box, primarily due to security considerations. Some brokers may currently be designed to only issue global read+write permissions, but authors may want to change their service permissions model if shareability is now on the cards, for example, read+write for SpaceDevs in the original space, but read-only for spaces that received the instance via sharing.




--

Jay Badenhope

Product Manager
Pivotal Labs
+1.510.517.6973 
LinkedIn | Twitter



Re: CF CLI v6.35.0 Release Today - service instance sharing; client credentials

Dr Nic Williams <drnicwilliams@...>
 

Jay, to confirm, an admin will be able to share a service broker/service offering even if it’s not explicitly supported by the /v2/catalog?


From: cf-dev@... <cf-dev@...> on behalf of Jay Badenhope <jbadenhope@...>
Sent: Monday, March 12, 2018 6:16:51 PM
To: cf-dev@...
Subject: Re: [cf-dev] CF CLI v6.35.0 Release Today - service instance sharing; client credentials
 
Hi Dr Nic,
Building on Denise's response, we also empower the admin to enable/restrict sharing. There are two settings that must be true in order to enable service instance sharing:
1. At the global level: "To enable service instance sharing, an administrator must enable the `service_instance_sharing` flag." https://docs.cloudfoundry.org/devguide/services/sharing-instances.html#enabling
2. At the service level, as you mentioned, "Service brokers must explicitly enable service instance sharing by setting a flag in their service-level metadata object." https://docs.cloudfoundry.org/services/enable-sharing.html#enabling

Matthias,
I'm going to connect with my UAA colleagues and make sure we have a good answer to your question.

Jay

On Mon, Mar 12, 2018 at 3:24 AM, <dyu@...> wrote:
The decision to have service authors opt in was to account for the fact that some services may not be shareable out-of-the-box, primarily due to security considerations. Some brokers may currently be designed to only issue global read+write permissions, but authors may want to change their service permissions model if shareability is now on the cards, for example, read+write for SpaceDevs in the original space, but read-only for spaces that received the instance via sharing.




--

Jay Badenhope

Product Manager
Pivotal Labs
+1.510.517.6973 
LinkedIn | Twitter



Re: CF CLI v6.35.0 Release Today - service instance sharing; client credentials

Jay Badenhope <jbadenhope@...>
 

Hi Dr Nic,
Building on Denise's response, we also empower the admin to enable/restrict sharing. There are two settings that must be true in order to enable service instance sharing:
1. At the global level: "To enable service instance sharing, an administrator must enable the `service_instance_sharing` flag." https://docs.cloudfoundry.org/devguide/services/sharing-instances.html#enabling
2. At the service level, as you mentioned, "Service brokers must explicitly enable service instance sharing by setting a flag in their service-level metadata object." https://docs.cloudfoundry.org/services/enable-sharing.html#enabling

Matthias,
I'm going to connect with my UAA colleagues and make sure we have a good answer to your question.

Jay

On Mon, Mar 12, 2018 at 3:24 AM, <dyu@...> wrote:
The decision to have service authors opt in was to account for the fact that some services may not be shareable out-of-the-box, primarily due to security considerations. Some brokers may currently be designed to only issue global read+write permissions, but authors may want to change their service permissions model if shareability is now on the cards, for example, read+write for SpaceDevs in the original space, but read-only for spaces that received the instance via sharing.




--

Jay Badenhope

Product Manager
Pivotal Labs
+1.510.517.6973 
LinkedIn | Twitter



Feedback request: Disable logging Client IP’s in the Gorouter logs for compliance with the EU General Data Protection Regulation (GDPR)

Shubha Anjur Tupil
 

Hello everyone, 

In lieu of The EU General Data Protection Regulation (GDPR), the routing team is investigating adding manifest properties to allow an operator to disable logging the client IP's in the X-Forwarded-For header in the access and error logs for Gorouter.

Enforcement of The EU General Data Protection Regulation (GDPR) (https://www.eugdpr.org/) begins May 28th and imposes steep fines. This law says that companies will be fined if they are capturing PII. The Gorouter currently captures Client IP addresses that are included in that definition. We are exploring manifest properties to allow operators to disable logging the originating client IP.

We need help weighing the options given the use-cases.

Use-cases:

#1. L3 Load Balancer e.g. Amazon NLB in front of the Gorouter 
The client IP is logged and the X-Forwarded-For header might have the originating client IP if an intermediary component or the originating client is adding the header.

#2. L7 Load Balancer e.g. Amazon NLB in front of the Gorotuer 
The client IP is the IP of the load balancer in front of the Gorouter, but the X-Forwarded-For header has the originating client IP.

Option1: We have one manifest property to disable logging both the X-Forwarded-For header and the Client IP in the Gorouter logs.

For use-case #1 this works, and there is only one manifest property to disable all PII being logged by the Gorouter.

For use-case #2 this results in the operator losing the LB IP which might be helpful for troubleshooting.

Option2: We have two separate manifest properties to disable logging the X-Forwarded-For header and/or the Client IP in the Gorouter logs. This is generally more flexible at the cost of user experience.

For use-case #1 this would mean that an operator would have to set two manifest properties instead of one. Both these properties would need to be exposed in Ops Man for PCF installations. It leads to a more cumbersome user experience, adding to the already long list of options.

For use-case #2 this results in a better outcome for an operator in that they still get the information on the LB the request came from, while the originating client IP will not be logged.

Based on the information we have, we are not sure which experience is better and which use-case to optimize for. Some questions we have

  1. What type of LB is more popular in CF environments; L3 or L7? This might help us optimize for Use-case #2 and go with Option #2.
  2. Is there a compelling reason from an operator perspective for installations with a L7 load balancer, making it important to have the Client IP (i.e. they would absolutely want to have Option #2 or maybe don’t care and Option #1 would be ok)?
  3. Does the user experience benefit by having just one option outweigh the benefit of having the client IP for an operator with an L7 Load Balancer? (Flexibility over Experience)

We would love to hear from you if you have thoughts on this. 

Thanks, 

Shannon Coen & Shubha Anjur Tupil
Product Managers, CF Routing



Re: FW: 2018 Cloud Foundry PaaS Certification Program

Eric Malm <emalm@...>
 

Hi, Deepak,

Since these seem to be issues with running the Garden-runC components on your particular infrastructure, I think it would be most productive for your team to file one or more issues about them at https://github.com/cloudfoundry/garden-runc-release.

Best,
Eric, CF Diego PM

On Mon, Mar 12, 2018 at 1:40 PM, Deepak Vij <deepak.vij@...> wrote:

Hi folks, sending this email on behalf of Amit. Any help in resolving the following issue would be highly appreciated. Thanks.

 

Regards,

Deepak Vij

 

From: Amit Roushan
Sent: Monday, March 12, 2018 12:00 PM
To: Deepak Vij (A) <deepak.vij@...>
Subject: RE: 2018 Cloud Foundry PaaS Certification Program

 

HI Chip,

 

                As Huawei uses saltstack for its service deployment. So we are porting Bosh scripts onto salt.

                We have finished all modules deployment but ran into one “mount” related issue for “Garden” module. described below .

 

1.       Instead of “vcap”, we are maintaining “paas” user for CF module deployment. “paas” user has sudoer permission as well.

Now during “garden” module deployment, “greenskeeper” is expecting “vcap” user and it is hard coded in code.

https://github.com/cloudfoundry/garden-runc-release/blob/master/src/greenskeeper/cmd/greenskeeper/main.go#L20

               

                                Currently we have changed it with “paas” user and generated new binary.

 

                                Query:  Is there any plan to make it configurable for other user.

 

       

2.       After above workaround solution, We are facing some problem in grootfs. It is reporting some “mount” related issue.

Following are the related information:

 

System Info :

       

 

 

Issue reported by grootfs:

 

               

When we tried to run mount command manually. Following issue reported in dmesg.

 

 

 

Thanks

-Amit



FW: 2018 Cloud Foundry PaaS Certification Program

Deepak Vij
 

Hi folks, sending this email on behalf of Amit. Any help in resolving the following issue would be highly appreciated. Thanks.

 

Regards,

Deepak Vij

 

From: Amit Roushan
Sent: Monday, March 12, 2018 12:00 PM
To: Deepak Vij (A) <deepak.vij@...>
Subject: RE: 2018 Cloud Foundry PaaS Certification Program

 

HI Chip,

 

                As Huawei uses saltstack for its service deployment. So we are porting Bosh scripts onto salt.

                We have finished all modules deployment but ran into one “mount” related issue for “Garden” module. described below .

 

1.       Instead of “vcap”, we are maintaining “paas” user for CF module deployment. “paas” user has sudoer permission as well.

Now during “garden” module deployment, “greenskeeper” is expecting “vcap” user and it is hard coded in code.

https://github.com/cloudfoundry/garden-runc-release/blob/master/src/greenskeeper/cmd/greenskeeper/main.go#L20

               

                                Currently we have changed it with “paas” user and generated new binary.

 

                                Query:  Is there any plan to make it configurable for other user.

 

       

2.       After above workaround solution, We are facing some problem in grootfs. It is reporting some “mount” related issue.

Following are the related information:

 

System Info :

       

 

 

Issue reported by grootfs:

 

               

When we tried to run mount command manually. Following issue reported in dmesg.

 

 

 

Thanks

-Amit


SapMachine is available in the Cloud Foundry Java Buildpack

Rene Schuenemann
 
Edited

Hello Folks,

we are happy to announce, that the SapMachine is now available as a standard Java Runtime Environment in the Cloud Foundry Java Buildpack!

The SapMachine project is a downstream version of the OpenJDK project. It is used to build and maintain a SAP supported version of OpenJDK for SAP customers who wish to use OpenJDK in their production environments.
With the SapMachine project we can quickly react on customer problems with new and fixed versions, without having to wait on the OpenJDK project.
We will also bring over features from our commercially licensed, closed source SAP JVM into the SapMachine which can't be integrated upstream in the short-term.

The SapMachine project is currently available for Linux x64, supporting both glibc and musl libc (Alpine Linux).
For more details please visit us at https://sapmachine.io.

Starting from version v4.9 of the Cloud Foundry Java Buildpack (https://github.com/cloudfoundry/java-buildpack/releases/tag/v4.9), the SapMachine can be used as a standard Java Runtime Environment.
In order to use the SapMachine JRE, set the following environment variable for your Cloud Foundry application:

cf set-env <app_name> JBP_CONFIG_COMPONENTS '{jres: ["JavaBuildpack::Jre::SapMachineJRE"]}' 
cf restage <app_name>


For more detailed instructions pleases check out the documentation at https://github.com/cloudfoundry/java-buildpack/blob/master/docs/jre-sap_machine_jre.md
Want to get in touch with us? Simply mail to sapmachine at sap.com or join our Slack Channel https://join.slack.com/t/sapmachine/signup

The SapMachine Team


Re: CF CLI v6.35.0 Release Today - service instance sharing; client credentials

dyu@...
 

The decision to have service authors opt in was to account for the fact that some services may not be shareable out-of-the-box, primarily due to security considerations. Some brokers may currently be designed to only issue global read+write permissions, but authors may want to change their service permissions model if shareability is now on the cards, for example, read+write for SpaceDevs in the original space, but read-only for spaces that received the instance via sharing.


Re: CF CLI v6.35.0 Release Today - service instance sharing; client credentials

Matthias Winzeler
 

Hi Jay

The service accounts feature sounds very exciting. How does it work exactly? 

I guess I have to create a user on UAA with client credentials grant type (i.e. with uaac) first.
How can I then permit this client to orgs and spaces so that he can interact with CF like a user?

-Matthias

2018-03-10 2:15 GMT+01:00 Dr Nic Williams <drnicwilliams@...>:


> Service brokers must explicitly enable service instance sharing by setting a flag in their service-level metadata object. This allows service instances, of any service plan, to be shared across organizations and spaces. The "shareable" flag must be set to true in the service-level metadata to enable service instance sharing. If the flag is set to false or is absent, sharing is disabled.


From: cf-dev@... <cf-dev@...> on behalf of Dr Nic Williams <drnicwilliams@...>
Sent: Saturday, March 10, 2018 11:14:01 AM
To: cf-dev@...
Subject: Re: [cf-dev] CF CLI v6.35.0 Release Today - service instance sharing; client credentials
 
Reading the links thru to info in services supporting sharing - what were the ideas behind requiring service implementors to opt in; rather than leave this to CF administrators or space/org administrators to enable?


From: cf-dev@... <cf-dev@...> on behalf of Jay Badenhope <jbadenhope@...>
Sent: Saturday, March 10, 2018 8:53:18 AM
To: cf-dev@...
Subject: [cf-dev] CF CLI v6.35.0 Release Today - service instance sharing; client credentials
 
The CF CLI team released version 6.35.0 today. Yay!


Service Instance Sharing

This cf CLI feature includes two new commands, share-service and unshare-service, to enable you to share service instances between spaces in the same or different orgs. Additional details here. We welcome your feedback on the new implementation.

To help you track where a service instance is shared to or from, we refactored and updated the service command.

Service Account Authentication (Client Credentials)

It is now possible to authenticate with only a client ID and a secret using the auth command with a new --client-credentials flag. Before this release, users could only log in as a user (i.e. username & password with either default client id, or custom client id & secret). That meant "fake" users needed to be prepared for CI environments and scripts ("tiles" self-registration).

push Fixes and Enhancements

  • v2-push no longer accepted (previous release merged v2-push into push)
  • Fixes problem where existing routes with customer hosts were ignored on push, #1321

Other Fixes and Enhancements

  • Fixed problem where wildcards weren't allowed in routes section of app manifest, deployment #399

Plugin Updates

Going forward, we ask that every plugin name matches its command name so it can be installed and uninstalled with the same name.

  • Updated Event Alerts Plugin to 0.0.1, #198, then removed that plugin, #211
  • Updated top Plugin to 0.9.3, #210
  • Updated service-use to 1.2.2 with matching command and plugin names #213

Refactored commands

  • services to enable an upcoming feature
  • service (see above)
  • logout to enable clearing of client credentials for Service Account Authentication (see above); will now also show user name during logout for consistency with other commands

Release contributors: An Yu, Nick Wei, Sebastian Vidrio, Anande Gaitonde, Jay Badenhope, and special guest Kevin Middleton. Thanks also to our partners on the CAPI, SAPI, and UAA teams.


--

Jay Badenhope

Product Manager, CF CLI
Pivotal Labs
LinkedIn | Twitter





--
Matthias Winzeler
Mattenenge 8
3011 Bern
mailto: matthias.winzeler@...


Re: CF CLI v6.35.0 Release Today - service instance sharing; client credentials

Dr Nic Williams <drnicwilliams@...>
 

Specifically referencing https://docs.cloudfoundry.org/services/enable-sharing.html#enabling

> Service brokers must explicitly enable service instance sharing by setting a flag in their service-level metadata object. This allows service instances, of any service plan, to be shared across organizations and spaces. The "shareable" flag must be set to true in the service-level metadata to enable service instance sharing. If the flag is set to false or is absent, sharing is disabled.


From: cf-dev@... <cf-dev@...> on behalf of Dr Nic Williams <drnicwilliams@...>
Sent: Saturday, March 10, 2018 11:14:01 AM
To: cf-dev@...
Subject: Re: [cf-dev] CF CLI v6.35.0 Release Today - service instance sharing; client credentials
 
Reading the links thru to info in services supporting sharing - what were the ideas behind requiring service implementors to opt in; rather than leave this to CF administrators or space/org administrators to enable?


From: cf-dev@... <cf-dev@...> on behalf of Jay Badenhope <jbadenhope@...>
Sent: Saturday, March 10, 2018 8:53:18 AM
To: cf-dev@...
Subject: [cf-dev] CF CLI v6.35.0 Release Today - service instance sharing; client credentials
 
The CF CLI team released version 6.35.0 today. Yay!


Service Instance Sharing

This cf CLI feature includes two new commands, share-service and unshare-service, to enable you to share service instances between spaces in the same or different orgs. Additional details here. We welcome your feedback on the new implementation.

To help you track where a service instance is shared to or from, we refactored and updated the service command.

Service Account Authentication (Client Credentials)

It is now possible to authenticate with only a client ID and a secret using the auth command with a new --client-credentials flag. Before this release, users could only log in as a user (i.e. username & password with either default client id, or custom client id & secret). That meant "fake" users needed to be prepared for CI environments and scripts ("tiles" self-registration).

push Fixes and Enhancements

  • v2-push no longer accepted (previous release merged v2-push into push)
  • Fixes problem where existing routes with customer hosts were ignored on push, #1321

Other Fixes and Enhancements

  • Fixed problem where wildcards weren't allowed in routes section of app manifest, deployment #399

Plugin Updates

Going forward, we ask that every plugin name matches its command name so it can be installed and uninstalled with the same name.

  • Updated Event Alerts Plugin to 0.0.1, #198, then removed that plugin, #211
  • Updated top Plugin to 0.9.3, #210
  • Updated service-use to 1.2.2 with matching command and plugin names #213

Refactored commands

  • services to enable an upcoming feature
  • service (see above)
  • logout to enable clearing of client credentials for Service Account Authentication (see above); will now also show user name during logout for consistency with other commands

Release contributors: An Yu, Nick Wei, Sebastian Vidrio, Anande Gaitonde, Jay Badenhope, and special guest Kevin Middleton. Thanks also to our partners on the CAPI, SAPI, and UAA teams.


--

Jay Badenhope

Product Manager, CF CLI
Pivotal Labs
LinkedIn | Twitter



Re: CF CLI v6.35.0 Release Today - service instance sharing; client credentials

Dr Nic Williams <drnicwilliams@...>
 

Reading the links thru to info in services supporting sharing - what were the ideas behind requiring service implementors to opt in; rather than leave this to CF administrators or space/org administrators to enable?


From: cf-dev@... <cf-dev@...> on behalf of Jay Badenhope <jbadenhope@...>
Sent: Saturday, March 10, 2018 8:53:18 AM
To: cf-dev@...
Subject: [cf-dev] CF CLI v6.35.0 Release Today - service instance sharing; client credentials
 
The CF CLI team released version 6.35.0 today. Yay!


Service Instance Sharing

This cf CLI feature includes two new commands, share-service and unshare-service, to enable you to share service instances between spaces in the same or different orgs. Additional details here. We welcome your feedback on the new implementation.

To help you track where a service instance is shared to or from, we refactored and updated the service command.

Service Account Authentication (Client Credentials)

It is now possible to authenticate with only a client ID and a secret using the auth command with a new --client-credentials flag. Before this release, users could only log in as a user (i.e. username & password with either default client id, or custom client id & secret). That meant "fake" users needed to be prepared for CI environments and scripts ("tiles" self-registration).

push Fixes and Enhancements

  • v2-push no longer accepted (previous release merged v2-push into push)
  • Fixes problem where existing routes with customer hosts were ignored on push, #1321

Other Fixes and Enhancements

  • Fixed problem where wildcards weren't allowed in routes section of app manifest, deployment #399

Plugin Updates

Going forward, we ask that every plugin name matches its command name so it can be installed and uninstalled with the same name.

  • Updated Event Alerts Plugin to 0.0.1, #198, then removed that plugin, #211
  • Updated top Plugin to 0.9.3, #210
  • Updated service-use to 1.2.2 with matching command and plugin names #213

Refactored commands

  • services to enable an upcoming feature
  • service (see above)
  • logout to enable clearing of client credentials for Service Account Authentication (see above); will now also show user name during logout for consistency with other commands

Release contributors: An Yu, Nick Wei, Sebastian Vidrio, Anande Gaitonde, Jay Badenhope, and special guest Kevin Middleton. Thanks also to our partners on the CAPI, SAPI, and UAA teams.


--

Jay Badenhope

Product Manager, CF CLI
Pivotal Labs
LinkedIn | Twitter



CF CLI v6.35.0 Release Today - service instance sharing; client credentials

Jay Badenhope <jbadenhope@...>
 

The CF CLI team released version 6.35.0 today. Yay!


Service Instance Sharing

This cf CLI feature includes two new commands, share-service and unshare-service, to enable you to share service instances between spaces in the same or different orgs. Additional details here. We welcome your feedback on the new implementation.

To help you track where a service instance is shared to or from, we refactored and updated the service command.

Service Account Authentication (Client Credentials)

It is now possible to authenticate with only a client ID and a secret using the auth command with a new --client-credentials flag. Before this release, users could only log in as a user (i.e. username & password with either default client id, or custom client id & secret). That meant "fake" users needed to be prepared for CI environments and scripts ("tiles" self-registration).

push Fixes and Enhancements

  • v2-push no longer accepted (previous release merged v2-push into push)
  • Fixes problem where existing routes with customer hosts were ignored on push, #1321

Other Fixes and Enhancements

  • Fixed problem where wildcards weren't allowed in routes section of app manifest, deployment #399

Plugin Updates

Going forward, we ask that every plugin name matches its command name so it can be installed and uninstalled with the same name.

  • Updated Event Alerts Plugin to 0.0.1, #198, then removed that plugin, #211
  • Updated top Plugin to 0.9.3, #210
  • Updated service-use to 1.2.2 with matching command and plugin names #213

Refactored commands

  • services to enable an upcoming feature
  • service (see above)
  • logout to enable clearing of client credentials for Service Account Authentication (see above); will now also show user name during logout for consistency with other commands

Release contributors: An Yu, Nick Wei, Sebastian Vidrio, Anande Gaitonde, Jay Badenhope, and special guest Kevin Middleton. Thanks also to our partners on the CAPI, SAPI, and UAA teams.


--

Jay Badenhope

Product Manager, CF CLI
Pivotal Labs
LinkedIn | Twitter



Re: Automating Security Groups and Brokered Services

Mueller, Florian
 

Hi all,

 

There is already a related issue in the OSB repo:

https://github.com/openservicebrokerapi/servicebroker/issues/386

 

The scope of this issue is a bit broader but the final solution might be very similar.

I would suggest adding your requirements also to this issue.

 

 

Regards,

 

Florian

 

 

From: <cf-dev@...> on behalf of Guillaume Berche <bercheg@...>
Reply-To: "cf-dev@..." <cf-dev@...>
Date: Friday, 9.
March 2018 at 01:18
To: "cf-dev@..." <cf-dev@...>
Subject: Re: [cf-dev] Automating Security Groups and Brokered Services

 

Thanks Sabha. This proposal also makes sense to me. This would remove the need for brokers to open ASG and be given admin permission.

A link to the proposal doc would be useful.

+1 for sharing this proposal to the OSB api team (could not yet find a related issue in the OSB repo)

Note that we currently plan to also leverage this mechanism for service brokers that return empty credentials in service bindings, and rather provide on-demand egress access to intranets or internet subnets. Service plan visibility enables restricting access to the intranet/internet plans per org.


Guillaume.

 



Guillaume.

 

On Thu, Mar 8, 2018 at 8:47 PM, Matt Cholick <cholick@...> wrote:

That's great Guillaume, thanks!

 

That's exactly what I was after Sabha, thanks for sharing.

 

Looping in Matt McNeeney for the OSBPAI perspective:
is this concept generic enough to fit into OSBAPI spec? Is anything like Sabha's proposal anything you've thought about or on your radar?

 

 


Re: Automating Security Groups and Brokered Services

Sabha Parameswaran <sabhap@...>
 

On Thu, Mar 8, 2018 at 4:17 PM, Guillaume Berche <bercheg@...> wrote:
Thanks Sabha. This proposal also makes sense to me. This would remove the need for brokers to open ASG and be given admin permission.

A link to the proposal doc would be useful.

+1 for sharing this proposal to the OSB api team (could not yet find a related issue in the OSB repo)

Note that we currently plan to also leverage this mechanism for service brokers that return empty credentials in service bindings, and rather provide on-demand egress access to intranets or internet subnets. Service plan visibility enables restricting access to the intranet/internet plans per org.

Guillaume.



Guillaume.

On Thu, Mar 8, 2018 at 8:47 PM, Matt Cholick <cholick@...> wrote:
That's great Guillaume, thanks!

That's exactly what I was after Sabha, thanks for sharing.

Looping in Matt McNeeney for the OSBPAI perspective:
is this concept generic enough to fit into OSBAPI spec? Is anything like Sabha's proposal anything you've thought about or on your radar?





--
Sabha Parameswaran
Platform Engineering, Cloud Foundry
Pivotal, Inc.


Re: Automating Security Groups and Brokered Services

Basavaraju, Jagadish
 

Hi Matt,

                Service Fabrik does the below & creates the application security group on the fly post successful creation of the service instance[1]. In case of SF step #2 in the below is omitted and security group per instance is created/bound to the space post successful instance creation (via hook added in last operation), as this makes it easier for clean-up of the same during instance deletion. However with broker taking on the responsibility of creating the ASG, the dependency is added back from broker onto the upstream CF, which is not ideal for a cross platform broker. SF handles this via having plugins (platform managers) to which such platform specific operations are delegated.

 

+1, for the below proposal from Sabha. This is what even wanted & decouple broker from the platform.

 

[1] https://github.com/cloudfoundry-incubator/service-fabrik-broker/wiki/Architecture#create-bosh-service-instance

 

Regards,

Jagadish

 

 

From: <cf-dev@...> on behalf of Sabha Parameswaran <sabhap@...>
Reply-To: "cf-dev@..." <cf-dev@...>
Date: Friday, 9 March 2018 at 12:20 AM
To: "cf-dev@..." <cf-dev@...>
Subject: Re: [cf-dev] Automating Security Groups and Brokered Services

 

I proposed an enhancement to let the service broker api to return the service endpoint info as part of the bind call so Cloud Controller or whatever is running the platform can create the ASG or its equivalent to let the app communicate to that endpoint. This lets the service broker be transparent to where its running and what platform or the credentials of the platform while the platform manage the required policy/security handling.

 

sorry for the long content:

Problem Statement

As more services get consumed by the applications (same application or across multiple apps) in a given space or boundary, it becomes a painful exercise in figuring out the endpoints and creating a customized ASG or its equivalent (which can change as more services get added/consumed). Platforms like CF do not allow application developer to change or administer App Security Groups and requires an Admin to open things up. With proliferation of Service Brokers and associated Services in the Services Marketplace, this becomes a burden every App Developer has to undergo.

Other Solutions

It is possible to build yet another service broker that can act as an intermediary to create and manage the Application Security Groups on behalf of set of a services used by the application. But this requires the Service Broker to be actively coupled with the Platform, requires administrative privileges, endpoint information about the platform while also requiring the Service Broker to be tightly tied to the underlying services (to know which to allow, what endpoints/ports etc.). This just makes it more complex and inefficient and breaks the model of Service Broker not requiring anything to know about the consuming Platform (CF or Kubernetes).

Proposed Solution

 

Enhance the Service Broker to return additional metadata on `bind` service call about the service (DNS name or IP, Subnet, Port ranges and any additional metadata). This would allow automatic creation of ASG or its equivalent for the various Platforms based on the services bound to a given application or consumed in an associated space or boundary on a bind call and update/delete the ASG on unbind of the service.

 

Reason for the Service Broker enhancement

Even if an ASG can be created automatically based purely on a service bind information returned by the Service Broker to the Platform, it is harder in some cases to lock down or derive the endpoint information. For instance, in case of APM service brokers, the bind information might contain only a license key without information on the remote service endpoint. The APM agents bundled via buildpacks or application might have some well known endpoints to reach out to.

 

Service Broker Enhancement

 

The service broker would return an additional json element that contains endpoint metadata as part of the ‘bind’ call. This metadata portion can contain details about the endpoint (ip or dns names or cidr), ports (list of ports, port ranges), protocol, the service instance guid and anything related to help the control/consuming platform create the appropriate network security policy



{
     "credentials": {
       "uri": "mycustomservice.instance-name",
       "username": "myuser",
       "password": "pass",
       "host": "myhost",

       "database": "dbname"
     },

     "service_endpoints": [ {

       "endpoint": "service.test.domain.com",
       "port": 3306,

       "protocol": "all",

       "sid": "653ba025-aa43-4e66-941a-4fea786d3755"
        ...

     }] ,

  ....
   }



Workflow

  1.  
  2. Admin
  3. or user with requisite privileges registers the Service Broker with the Platform
  4.  
  5.  
  6. Admin
  7. user provides an allowed set of network ranges, ports or dns names to be used as the permitted superset of network endpoints to be associated with the Service Broker to the Platform.
  8.  
  9.  

10.   

    1.  
    2. This
    3. would be specific to the Platform and not part of any service broker contact. This allows the administrator to lock down allowed Vs. disallowed services across the platform per service broker.
    4.  
  1.  
  1.  
  2. The
  3. Platform saves the information in its Policy Engine against the Service.
  4.  
  5.  
  6. User
  7. creates an instance of a service from the Service Broker and binds to it.
  8.  
  9.  
  10. Service
  11. Broker provides the endpoint metadata during the bind call to the Platform
  12.  
  13.  
  14. Platform
  15. verifies if the provided endpoint matches or falls within the permitted set of endpoints associated with the service broker to decide to allow or not using its Policy Engine.
  16.  
  17.  
  18. Proceed
  19. with auto creation of the ASG or its equivalent based on the outcome for the app container.
  20.  
  21.  
  22. Allow
  23. or disallow the application container from communicating with the service.
  24.  
  25.  
  26. On
  27. unbind, update or delete the ASG for the associated app.
  28.  
  29.  
  30. Block
  31. outbound communication with removal of the ASG.
  32.  



Workflow for App Security Creation


Can share the doc if there is more interest.

 

-Sabha


 

On Thu, Mar 8, 2018 at 10:21 AM, Guillaume Berche <bercheg@...> wrote:

Hi Matt,

We developed at Orange the following broker which matches your description,
https://github.com/orange-cloudfoundry/sec-group-broker-filter

Feedback is welcome.

 

Regards,

Guillaume.

 

On Thu, Mar 8, 2018 at 3:10 PM, Matt Cholick <cholick@...> wrote:

In a default deny situation, where the operator doesn't want to open up a foundation-wide security group at service installation time, it would be useful to create and bind a security group on the fly (that allows communication to the service deployment) at service instance creation time.


1. Developer creates service

2. Broker checks if if the ASG allowing communication exists

3. If not, broker binds the ASG to the app's space

4. Rest of flow works as normal

The developer would need to restart the app after service bind anyway, so the security group would get applied as part of that flow.

 

Has anyone built something this as an open source library? Have run across some folks that are interested in this as a cross-cutting broker behavior, to keep their traffic rules as restrictive as possible.

-Matt Cholick

 



 

--

Sabha Parameswaran

Platform Engineering, Cloud Foundry

Pivotal, Inc.


Re: Notice: Known stability issue running Diego cells on BOSH stemcell 3541.2 and later; advise use of 3468.26 instead

Jon Price
 

And we had just finished ours…

 

Jon

 

From: cf-dev@... <cf-dev@...> On Behalf Of Mike Youngstrom
Sent: Thursday, March 8, 2018 4:39 PM
To: cf-dev@...
Subject: Re: [cf-dev] Notice: Known stability issue running Diego cells on BOSH stemcell 3541.2 and later; advise use of 3468.26 instead

 

Thanks Eric for these posts.  You caught us literally moments before we started updating to cf-deployment 1.16.0.

 

Mike

 

On Thu, Mar 8, 2018 at 5:29 PM, Eric Malm <emalm@...> wrote:

Update: the ever-nimble BOSH team has just released stemcell 3541.8, which successfully resolves this issue.

 

Best,

Eric

 

On Thu, Mar 8, 2018 at 1:03 PM, Eric Malm <emalm@...> wrote:

Hi, all,

 

FYI, the core CF dev teams have observed an issue running Diego cells on BOSH stemcell 3541.2 and later, and at the moment advise you to roll back to the 3468 stemcell line for those VMs. The next cf-deployment release will also revert to that stemcell line. The BOSH team is addressing the issue in https://www.pivotaltracker.com/story/show/155716654, and we expect the fix to be incorporated into the next stemcell in the 3541 line.

 

Problem: Diego cell VMs affected by this issue will suddenly be unable to make new Garden containers, although existing containers will continue to run. Within at most 10 minutes, the cell also will cease to accept new CF app instances or tasks, preventing execution failures but reducing available capacity for placement. This failure to make containers can cause `cf push` to fail or app instances not to be restarted, especially if this issue incapacitates enough cells in the environment.

 

Symptoms: App developers may observe that `cf push` or `cf restart` fails either with a container-creation error that includes "permission denied" or because of insufficient resources. CF operators may observe a persistently elevated number of Diego cell reps reporting a UnhealthyCell metric of 1 or reporting failures to create Garden containers, or increased rates of Task or LRP auction failures from the Diego auctioneer component.

 

Mitigations: In addition to rolling back to the 3468 stemcell line, restarting or recreating the affected cell VM with the BOSH CLI, running `sudo monit restart all` on it, or running `sudo chown vcap:vcap /var/vcap/data/garden /var/vcap/data/rep` and waiting at most 10 minutes will all suffice to mitigate the immediate effects of the issue. The last option, while the most manual, does have the benefit of not disrupting app instances and tasks that are already running on the Diego cell VM.

 

More details: Because of the delicate filesystem dance that Garden must perform to create container root filesystems, the Garden and Diego rep jobs on the Diego cell rely on certain directories on the /var/vcap/data volume to have very particular user ownership and access permissions.

 

In the 3541.2 stemcell, the bosh-agent added new behavior to change the ownership on those directories to root:vcap when it restarts, which is incompatible with what the garden and rep jobs require. Normally this is not a problem, as the garden and rep ctl scripts override that ownership when BOSH invokes them. If the bosh-agent is restarted for any reason, though, it will reset it without restarting those BOSH jobs, and Garden will then be unable to create the root filesystems for new containers.

 

Those of you who use PWS may have noticed an incident last Friday evening when `cf push` was unavailable for about half an hour. This incident was the result of an extended update to its BOSH director, which caused all of the bosh-agent processes on the VMs to restart and then to apply the permission change above on all of the Diego cells simultaneously. We resolved it by applying the `chown` mitigation above in the short term and then rolling back to the latest 3468 stemcell. A+ to the batch `bosh ssh` capability available in the v2 BOSH CLI, by the way.

 

Please let me know if you have any questions, and we apologize for any inconvenience this issue may have caused you.

 

Thanks,

Eric Malm, CF Diego PM

 

 


Re: Notice: Known stability issue running Diego cells on BOSH stemcell 3541.2 and later; advise use of 3468.26 instead

Mike Youngstrom
 

Thanks Eric for these posts.  You caught us literally moments before we started updating to cf-deployment 1.16.0.

Mike

On Thu, Mar 8, 2018 at 5:29 PM, Eric Malm <emalm@...> wrote:
Update: the ever-nimble BOSH team has just released stemcell 3541.8, which successfully resolves this issue.

Best,
Eric

On Thu, Mar 8, 2018 at 1:03 PM, Eric Malm <emalm@...> wrote:
Hi, all,

FYI, the core CF dev teams have observed an issue running Diego cells on BOSH stemcell 3541.2 and later, and at the moment advise you to roll back to the 3468 stemcell line for those VMs. The next cf-deployment release will also revert to that stemcell line. The BOSH team is addressing the issue in https://www.pivotaltracker.com/story/show/155716654, and we expect the fix to be incorporated into the next stemcell in the 3541 line.

Problem: Diego cell VMs affected by this issue will suddenly be unable to make new Garden containers, although existing containers will continue to run. Within at most 10 minutes, the cell also will cease to accept new CF app instances or tasks, preventing execution failures but reducing available capacity for placement. This failure to make containers can cause `cf push` to fail or app instances not to be restarted, especially if this issue incapacitates enough cells in the environment.

Symptoms: App developers may observe that `cf push` or `cf restart` fails either with a container-creation error that includes "permission denied" or because of insufficient resources. CF operators may observe a persistently elevated number of Diego cell reps reporting a UnhealthyCell metric of 1 or reporting failures to create Garden containers, or increased rates of Task or LRP auction failures from the Diego auctioneer component.

Mitigations: In addition to rolling back to the 3468 stemcell line, restarting or recreating the affected cell VM with the BOSH CLI, running `sudo monit restart all` on it, or running `sudo chown vcap:vcap /var/vcap/data/garden /var/vcap/data/rep` and waiting at most 10 minutes will all suffice to mitigate the immediate effects of the issue. The last option, while the most manual, does have the benefit of not disrupting app instances and tasks that are already running on the Diego cell VM.

More details: Because of the delicate filesystem dance that Garden must perform to create container root filesystems, the Garden and Diego rep jobs on the Diego cell rely on certain directories on the /var/vcap/data volume to have very particular user ownership and access permissions.

In the 3541.2 stemcell, the bosh-agent added new behavior to change the ownership on those directories to root:vcap when it restarts, which is incompatible with what the garden and rep jobs require. Normally this is not a problem, as the garden and rep ctl scripts override that ownership when BOSH invokes them. If the bosh-agent is restarted for any reason, though, it will reset it without restarting those BOSH jobs, and Garden will then be unable to create the root filesystems for new containers.

Those of you who use PWS may have noticed an incident last Friday evening when `cf push` was unavailable for about half an hour. This incident was the result of an extended update to its BOSH director, which caused all of the bosh-agent processes on the VMs to restart and then to apply the permission change above on all of the Diego cells simultaneously. We resolved it by applying the `chown` mitigation above in the short term and then rolling back to the latest 3468 stemcell. A+ to the batch `bosh ssh` capability available in the v2 BOSH CLI, by the way.

Please let me know if you have any questions, and we apologize for any inconvenience this issue may have caused you.

Thanks,
Eric Malm, CF Diego PM




Re: Notice: Known stability issue running Diego cells on BOSH stemcell 3541.2 and later; advise use of 3468.26 instead

Eric Malm <emalm@...>
 

Update: the ever-nimble BOSH team has just released stemcell 3541.8, which successfully resolves this issue.

Best,
Eric

On Thu, Mar 8, 2018 at 1:03 PM, Eric Malm <emalm@...> wrote:
Hi, all,

FYI, the core CF dev teams have observed an issue running Diego cells on BOSH stemcell 3541.2 and later, and at the moment advise you to roll back to the 3468 stemcell line for those VMs. The next cf-deployment release will also revert to that stemcell line. The BOSH team is addressing the issue in https://www.pivotaltracker.com/story/show/155716654, and we expect the fix to be incorporated into the next stemcell in the 3541 line.

Problem: Diego cell VMs affected by this issue will suddenly be unable to make new Garden containers, although existing containers will continue to run. Within at most 10 minutes, the cell also will cease to accept new CF app instances or tasks, preventing execution failures but reducing available capacity for placement. This failure to make containers can cause `cf push` to fail or app instances not to be restarted, especially if this issue incapacitates enough cells in the environment.

Symptoms: App developers may observe that `cf push` or `cf restart` fails either with a container-creation error that includes "permission denied" or because of insufficient resources. CF operators may observe a persistently elevated number of Diego cell reps reporting a UnhealthyCell metric of 1 or reporting failures to create Garden containers, or increased rates of Task or LRP auction failures from the Diego auctioneer component.

Mitigations: In addition to rolling back to the 3468 stemcell line, restarting or recreating the affected cell VM with the BOSH CLI, running `sudo monit restart all` on it, or running `sudo chown vcap:vcap /var/vcap/data/garden /var/vcap/data/rep` and waiting at most 10 minutes will all suffice to mitigate the immediate effects of the issue. The last option, while the most manual, does have the benefit of not disrupting app instances and tasks that are already running on the Diego cell VM.

More details: Because of the delicate filesystem dance that Garden must perform to create container root filesystems, the Garden and Diego rep jobs on the Diego cell rely on certain directories on the /var/vcap/data volume to have very particular user ownership and access permissions.

In the 3541.2 stemcell, the bosh-agent added new behavior to change the ownership on those directories to root:vcap when it restarts, which is incompatible with what the garden and rep jobs require. Normally this is not a problem, as the garden and rep ctl scripts override that ownership when BOSH invokes them. If the bosh-agent is restarted for any reason, though, it will reset it without restarting those BOSH jobs, and Garden will then be unable to create the root filesystems for new containers.

Those of you who use PWS may have noticed an incident last Friday evening when `cf push` was unavailable for about half an hour. This incident was the result of an extended update to its BOSH director, which caused all of the bosh-agent processes on the VMs to restart and then to apply the permission change above on all of the Diego cells simultaneously. We resolved it by applying the `chown` mitigation above in the short term and then rolling back to the latest 3468 stemcell. A+ to the batch `bosh ssh` capability available in the v2 BOSH CLI, by the way.

Please let me know if you have any questions, and we apologize for any inconvenience this issue may have caused you.

Thanks,
Eric Malm, CF Diego PM



Re: Automating Security Groups and Brokered Services

Guillaume Berche
 

Thanks Sabha. This proposal also makes sense to me. This would remove the need for brokers to open ASG and be given admin permission.

A link to the proposal doc would be useful.

+1 for sharing this proposal to the OSB api team (could not yet find a related issue in the OSB repo)

Note that we currently plan to also leverage this mechanism for service brokers that return empty credentials in service bindings, and rather provide on-demand egress access to intranets or internet subnets. Service plan visibility enables restricting access to the intranet/internet plans per org.

Guillaume.



Guillaume.

On Thu, Mar 8, 2018 at 8:47 PM, Matt Cholick <cholick@...> wrote:
That's great Guillaume, thanks!

That's exactly what I was after Sabha, thanks for sharing.

Looping in Matt McNeeney for the OSBPAI perspective:
is this concept generic enough to fit into OSBAPI spec? Is anything like Sabha's proposal anything you've thought about or on your radar?


1541 - 1560 of 9389