Date   

Re: Feature Narrative - Isolated Loggregator Components

Mike Youngstrom
 

Thanks for posting this Adam.  I love the direction and improvements the Loggregator team has been working towards lately!  Isolation segment support will be yet another plus.

Mike

On Tue, Mar 13, 2018 at 12:26 PM, Adam Hevenor <ahevenor@...> wrote:
One of the benefits we see of having re-engineered the persistance layer of Loggregator (see log-cache announcement) is a simpler architecture which can be deployed to multiple times within Cloud Foundry Application Runtime. This makes the concept of isolation segments one that works nicely with Loggregator, which is a feature we have received consistent feedback on over the last year. 

For more details about the problems this solved and work proposed please read the following Feature Narrative



Re: Designing a new routing tier for Cloud Foundry

Shannon Coen
 


Re: Announcing DNS-based service discovery for container networking

Michael Maximilien
 

Congrats to the team and to you Usha for getting this done with a heterogeneous group. Looks really good to me, well done. 

Yet another feature that’s helping CF push the envelop of cloud application development and management, while remaining simple and easy to use. Cheers,

Max

On Thu, Mar 15, 2018 at 9:54 AM Usha Ramachandran <uramachandran@...> wrote:

Hello everyone,


We are excited to announce the availability of DNS-based polyglot service discovery on Cloud Foundry. With this feature, Cloud Foundry apps in all languages and frameworks can leverage container-to-container networking.


Many Cloud Foundry teams came together to deliver this feature. A big thank you to CAPI, Routing, Diego, BOSH, CLI and Release Integration teams for their participation in conversations and cross-team development.


To find out more about the feature and the use cases, read this blog post. To try it out, follow our instructions to deploy with cf-deployment v1.17.0 or later.


Polyglot service discovery is currently an experimental feature. It is  deployed to Pivotal Web Services, where we will watch it and make generally available if all goes well.


We look forward to your feedback! Please reach out to us on #container-networking on Cloud Foundry slack if you have any questions!


Thank you,

CF Networking Team



--
dr.max Sent from my iPhone http://maximilien.org


CF routing-release 0.174.0

Shubha Anjur Tupil
 

Hi, 

We cut a release yesterday evening. The release included a fix for an issue we discovered on PWS while rolling out a feature to enable TLS between Gorouter and backend applications. 

- When a connection to a TLS enabled backend fails, Gorouter tries to send a request to another backend of the app before returning a response to the client. In an environment where some backends are TLS enabled and some are not, if the Gorouter first chooses a TLS enabled backend and fails, and if it subsequently chooses a non TLS backend, then it will appropriately use a plain text request. [details](https://www.pivotaltracker.com/story/show/155981374)


Regards, 
Shubha & Shannon
Product Manager, Routing


Announcing DNS-based service discovery for container networking

Usha Ramachandran
 

Hello everyone,


We are excited to announce the availability of DNS-based polyglot service discovery on Cloud Foundry. With this feature, Cloud Foundry apps in all languages and frameworks can leverage container-to-container networking.


Many Cloud Foundry teams came together to deliver this feature. A big thank you to CAPI, Routing, Diego, BOSH, CLI and Release Integration teams for their participation in conversations and cross-team development.


To find out more about the feature and the use cases, read this blog post. To try it out, follow our instructions to deploy with cf-deployment v1.17.0 or later.


Polyglot service discovery is currently an experimental feature. It is  deployed to Pivotal Web Services, where we will watch it and make generally available if all goes well.


We look forward to your feedback! Please reach out to us on #container-networking on Cloud Foundry slack if you have any questions!


Thank you,

CF Networking Team




Re: CF CLI v6.35.0 Release Today - service instance sharing; client credentials

Gowrisankar M
 

Hello All,

CF services sharing when it will be available for user-provided services?

BRs,Gowrisankar

On Thu, Mar 15, 2018 at 12:53 AM, Matthias Winzeler <matthias.winzeler@...> wrote:
Hi Tian

Thanks for the information - this sounds good. 
We're looking forward for clients with user permission since this will solve the issue with SAML federated users who want to interact with the API interactively (currently not possible, so we create local users for them).

-Matthias

2018-03-14 20:02 GMT+01:00 Tian Wang <tiwang@...>:
Hi Matthias,

The service accounts feature today works for global scopes on clients created in UAA such as `cloud_controller.admin` and `cloud_controller.admin_read_only`.

The current plan is to integrate down the line with the Perm project which will provide the ability to assign clients with space and org fine-grained authorizations, so that they can act essentially the same as a user.

Regards,

Tian

On Sat, Mar 10, 2018 at 12:52 AM, Matthias Winzeler <matthias.winzeler@...> wrote:
Hi Jay

The service accounts feature sounds very exciting. How does it work exactly? 

I guess I have to create a user on UAA with client credentials grant type (i.e. with uaac) first.
How can I then permit this client to orgs and spaces so that he can interact with CF like a user?

-Matthias

2018-03-10 2:15 GMT+01:00 Dr Nic Williams <drnicwilliams@...>:

> Service brokers must explicitly enable service instance sharing by setting a flag in their service-level metadata object. This allows service instances, of any service plan, to be shared across organizations and spaces. The "shareable" flag must be set to true in the service-level metadata to enable service instance sharing. If the flag is set to false or is absent, sharing is disabled.


From: cf-dev@... <cf-dev@...> on behalf of Dr Nic Williams <drnicwilliams@...>
Sent: Saturday, March 10, 2018 11:14:01 AM
To: cf-dev@...
Subject: Re: [cf-dev] CF CLI v6.35.0 Release Today - service instance sharing; client credentials
 
Reading the links thru to info in services supporting sharing - what were the ideas behind requiring service implementors to opt in; rather than leave this to CF administrators or space/org administrators to enable?


From: cf-dev@... <cf-dev@...> on behalf of Jay Badenhope <jbadenhope@...>
Sent: Saturday, March 10, 2018 8:53:18 AM
To: cf-dev@...
Subject: [cf-dev] CF CLI v6.35.0 Release Today - service instance sharing; client credentials
 
The CF CLI team released version 6.35.0 today. Yay!


Service Instance Sharing

This cf CLI feature includes two new commands, share-service and unshare-service, to enable you to share service instances between spaces in the same or different orgs. Additional details here. We welcome your feedback on the new implementation.

To help you track where a service instance is shared to or from, we refactored and updated the service command.

Service Account Authentication (Client Credentials)

It is now possible to authenticate with only a client ID and a secret using the auth command with a new --client-credentials flag. Before this release, users could only log in as a user (i.e. username & password with either default client id, or custom client id & secret). That meant "fake" users needed to be prepared for CI environments and scripts ("tiles" self-registration).

push Fixes and Enhancements

  • v2-push no longer accepted (previous release merged v2-push into push)
  • Fixes problem where existing routes with customer hosts were ignored on push, #1321

Other Fixes and Enhancements

  • Fixed problem where wildcards weren't allowed in routes section of app manifest, deployment #399

Plugin Updates

Going forward, we ask that every plugin name matches its command name so it can be installed and uninstalled with the same name.

  • Updated Event Alerts Plugin to 0.0.1, #198, then removed that plugin, #211
  • Updated top Plugin to 0.9.3, #210
  • Updated service-use to 1.2.2 with matching command and plugin names #213

Refactored commands

  • services to enable an upcoming feature
  • service (see above)
  • logout to enable clearing of client credentials for Service Account Authentication (see above); will now also show user name during logout for consistency with other commands

Release contributors: An Yu, Nick Wei, Sebastian Vidrio, Anande Gaitonde, Jay Badenhope, and special guest Kevin Middleton. Thanks also to our partners on the CAPI, SAPI, and UAA teams.


--

Jay Badenhope

Product Manager, CF CLI
Pivotal Labs
LinkedIn | Twitter





--





--



CF CLI v6.35.1 Release Today - patch for services issue

Jay Badenhope <jbadenhope@...>
 

Fixed regressions
  • Corrected issue so `service`, `services`, and `share-service` commands no longer fail with a JSON unmarshal error for certain service broker configurations; was also reported as Github issue #1342

Other fixes
  • Changed all warnings, including experimental warnings for v3 commands, so they now output to STDERR to make debugging easier
Thanks for reading.
Jay
--

Jay Badenhope

Product Manager, CF CLI
Pivotal Labs
+1.510.517.6973 
LinkedIn | Twitter



Re: CF CLI v6.35.0 Release Today - service instance sharing; client credentials

Matthias Winzeler
 

Hi Tian

Thanks for the information - this sounds good. 
We're looking forward for clients with user permission since this will solve the issue with SAML federated users who want to interact with the API interactively (currently not possible, so we create local users for them).

-Matthias

2018-03-14 20:02 GMT+01:00 Tian Wang <tiwang@...>:

Hi Matthias,

The service accounts feature today works for global scopes on clients created in UAA such as `cloud_controller.admin` and `cloud_controller.admin_read_only`.

The current plan is to integrate down the line with the Perm project which will provide the ability to assign clients with space and org fine-grained authorizations, so that they can act essentially the same as a user.

Regards,

Tian

On Sat, Mar 10, 2018 at 12:52 AM, Matthias Winzeler <matthias.winzeler@...> wrote:
Hi Jay

The service accounts feature sounds very exciting. How does it work exactly? 

I guess I have to create a user on UAA with client credentials grant type (i.e. with uaac) first.
How can I then permit this client to orgs and spaces so that he can interact with CF like a user?

-Matthias

2018-03-10 2:15 GMT+01:00 Dr Nic Williams <drnicwilliams@...>:

> Service brokers must explicitly enable service instance sharing by setting a flag in their service-level metadata object. This allows service instances, of any service plan, to be shared across organizations and spaces. The "shareable" flag must be set to true in the service-level metadata to enable service instance sharing. If the flag is set to false or is absent, sharing is disabled.


From: cf-dev@... <cf-dev@...> on behalf of Dr Nic Williams <drnicwilliams@...>
Sent: Saturday, March 10, 2018 11:14:01 AM
To: cf-dev@...
Subject: Re: [cf-dev] CF CLI v6.35.0 Release Today - service instance sharing; client credentials
 
Reading the links thru to info in services supporting sharing - what were the ideas behind requiring service implementors to opt in; rather than leave this to CF administrators or space/org administrators to enable?


From: cf-dev@... <cf-dev@...> on behalf of Jay Badenhope <jbadenhope@...>
Sent: Saturday, March 10, 2018 8:53:18 AM
To: cf-dev@...
Subject: [cf-dev] CF CLI v6.35.0 Release Today - service instance sharing; client credentials
 
The CF CLI team released version 6.35.0 today. Yay!


Service Instance Sharing

This cf CLI feature includes two new commands, share-service and unshare-service, to enable you to share service instances between spaces in the same or different orgs. Additional details here. We welcome your feedback on the new implementation.

To help you track where a service instance is shared to or from, we refactored and updated the service command.

Service Account Authentication (Client Credentials)

It is now possible to authenticate with only a client ID and a secret using the auth command with a new --client-credentials flag. Before this release, users could only log in as a user (i.e. username & password with either default client id, or custom client id & secret). That meant "fake" users needed to be prepared for CI environments and scripts ("tiles" self-registration).

push Fixes and Enhancements

  • v2-push no longer accepted (previous release merged v2-push into push)
  • Fixes problem where existing routes with customer hosts were ignored on push, #1321

Other Fixes and Enhancements

  • Fixed problem where wildcards weren't allowed in routes section of app manifest, deployment #399

Plugin Updates

Going forward, we ask that every plugin name matches its command name so it can be installed and uninstalled with the same name.

  • Updated Event Alerts Plugin to 0.0.1, #198, then removed that plugin, #211
  • Updated top Plugin to 0.9.3, #210
  • Updated service-use to 1.2.2 with matching command and plugin names #213

Refactored commands

  • services to enable an upcoming feature
  • service (see above)
  • logout to enable clearing of client credentials for Service Account Authentication (see above); will now also show user name during logout for consistency with other commands

Release contributors: An Yu, Nick Wei, Sebastian Vidrio, Anande Gaitonde, Jay Badenhope, and special guest Kevin Middleton. Thanks also to our partners on the CAPI, SAPI, and UAA teams.


--

Jay Badenhope

Product Manager, CF CLI
Pivotal Labs
LinkedIn | Twitter





--





--
Matthias Winzeler
Mattenenge 8
3011 Bern
mailto: matthias.winzeler@...


REMINDER: CAB call for March is Wednesday 03/21 @ 8a PST

Michael Maximilien
 

FYI...
 
Reminder that the CAB call for March is scheduled for next Wednesday 03/21 @ 8a PST.
 
WIP agenda and zoom info in [1] but summary here:
  1. CFF updates.
  2. Highlights from different PMCs and teams.
  3. Community talks:
a. MS SQL Service Broker proposal [3] by Jared Gordon of Pivotal and team (Pivotal and Microsoft)
b. CF-Dev [4] proposal [5] by Stephen Levine of Pivotal
 
I will send one more reminder next week on this list. Zoom soon.
 
Best,

PS: next month the CAB call will be during CF Summit in Boston and we will try to get a room so we can do it live! If you will be at summit, plan to join us.
 
------
dr.max
ibm ☁ 
silicon valley, ca
 


Re: CF CLI v6.35.0 Release Today - service instance sharing; client credentials

Tian Wang
 

Hi Matthias,

The service accounts feature today works for global scopes on clients created in UAA such as `cloud_controller.admin` and `cloud_controller.admin_read_only`.

The current plan is to integrate down the line with the Perm project which will provide the ability to assign clients with space and org fine-grained authorizations, so that they can act essentially the same as a user.

Regards,

Tian

On Sat, Mar 10, 2018 at 12:52 AM, Matthias Winzeler <matthias.winzeler@...> wrote:
Hi Jay

The service accounts feature sounds very exciting. How does it work exactly? 

I guess I have to create a user on UAA with client credentials grant type (i.e. with uaac) first.
How can I then permit this client to orgs and spaces so that he can interact with CF like a user?

-Matthias

2018-03-10 2:15 GMT+01:00 Dr Nic Williams <drnicwilliams@...>:

> Service brokers must explicitly enable service instance sharing by setting a flag in their service-level metadata object. This allows service instances, of any service plan, to be shared across organizations and spaces. The "shareable" flag must be set to true in the service-level metadata to enable service instance sharing. If the flag is set to false or is absent, sharing is disabled.


From: cf-dev@... <cf-dev@...> on behalf of Dr Nic Williams <drnicwilliams@...>
Sent: Saturday, March 10, 2018 11:14:01 AM
To: cf-dev@...
Subject: Re: [cf-dev] CF CLI v6.35.0 Release Today - service instance sharing; client credentials
 
Reading the links thru to info in services supporting sharing - what were the ideas behind requiring service implementors to opt in; rather than leave this to CF administrators or space/org administrators to enable?


From: cf-dev@... <cf-dev@...> on behalf of Jay Badenhope <jbadenhope@...>
Sent: Saturday, March 10, 2018 8:53:18 AM
To: cf-dev@...
Subject: [cf-dev] CF CLI v6.35.0 Release Today - service instance sharing; client credentials
 
The CF CLI team released version 6.35.0 today. Yay!


Service Instance Sharing

This cf CLI feature includes two new commands, share-service and unshare-service, to enable you to share service instances between spaces in the same or different orgs. Additional details here. We welcome your feedback on the new implementation.

To help you track where a service instance is shared to or from, we refactored and updated the service command.

Service Account Authentication (Client Credentials)

It is now possible to authenticate with only a client ID and a secret using the auth command with a new --client-credentials flag. Before this release, users could only log in as a user (i.e. username & password with either default client id, or custom client id & secret). That meant "fake" users needed to be prepared for CI environments and scripts ("tiles" self-registration).

push Fixes and Enhancements

  • v2-push no longer accepted (previous release merged v2-push into push)
  • Fixes problem where existing routes with customer hosts were ignored on push, #1321

Other Fixes and Enhancements

  • Fixed problem where wildcards weren't allowed in routes section of app manifest, deployment #399

Plugin Updates

Going forward, we ask that every plugin name matches its command name so it can be installed and uninstalled with the same name.

  • Updated Event Alerts Plugin to 0.0.1, #198, then removed that plugin, #211
  • Updated top Plugin to 0.9.3, #210
  • Updated service-use to 1.2.2 with matching command and plugin names #213

Refactored commands

  • services to enable an upcoming feature
  • service (see above)
  • logout to enable clearing of client credentials for Service Account Authentication (see above); will now also show user name during logout for consistency with other commands

Release contributors: An Yu, Nick Wei, Sebastian Vidrio, Anande Gaitonde, Jay Badenhope, and special guest Kevin Middleton. Thanks also to our partners on the CAPI, SAPI, and UAA teams.


--

Jay Badenhope

Product Manager, CF CLI
Pivotal Labs
LinkedIn | Twitter





--



Windows Acceptance Tests

Ashley Hathaway
 

Hello all!


This is a heads up that the Garden Windows team has completed simplifying our test suite by moving Windows Acceptance Tests (WATs) into CF Acceptance Tests (CATs). CATs can now be configured to run tests against a CF deployment with Windows cells.


The WATs repo will be moved into the cloudfoundry-attic in 60 days (mid-May). We recommend within this time if you can move off WATs please do so.


For any feedback or thoughts please feel free to reach out. We're also available on the CF #garden-windows Slack


giphy.gif


Thanks!

The Garden Windows Team



Re: The CF App Autoscaler: An Update

Michael Maximilien
 

Nice nice.

Congrats App AutoScaler team.

I second Julz ask to try it out and give feedback. And for regular updates, feel free to come to the CF Extensions monthly calls (watch this mailing list for announces) to get updates from this team and others.

Well done App AutoScaler. Cheers,

Max


On Wed, Mar 14, 2018 at 4:38 AM Julz Friedman <julz.friedman@...> wrote:

Hi cf-dev,


I wanted to give everyone a quick update on the progress of the application autoscaler project as we’ve recently hit some nice milestones.


WHAT IS THIS?


The app autoscaler is a cloud foundry extension (part of the Extensions PMC) that automatically scales cloud foundry applications based on load.


Why would I want it?


Do you like paying for more instances than you need? Do you like not having enough instances when load unexpectedly increases? If you, like me, want as many instances as you need and no more then you, like me, will like App Autoscaler.


Cool, who uses this already?


The big news - and the reason I'm writing this email - is that I'm excited to announce the first GA production deployment of the App Autoscaler: the app autoscaler is now used in production on the SAP Cloud. App Autoscaler’s design was initially based on the BlueMix App Autoscaler project which has been running in production and at scale for quite a while (on BlueMix, unsurprisingly). We’re hoping to be able to match all the features of the BlueMix autoscaler using the open-source autoscaler component in the next couple of months at which point we will deploy the open-source autoscaler to BlueMix also.


Is Autoscaler ready for me to use on my cloud?


I would say cautiously yes. The team feels that autoscaler is ready to go. We want to wait until App Autoscaler is running in two production environments before officially declaring a 1.0 version, but the current version is stable and is working well at scale.


Who is contributing to the App Autoscaler effort?


I'm proud to say that App Autoscaler has been a truly cross-foundation project, and we’ve had committers from IBM, SAP and Fujitsu. We work using a distributed commit model (and when we say distributed we mean distributed -- at various points we’ve had team members based in Australia, India, Seattle, China and the UK!).


LET ME TRY IT OUT!


Ok! You can get started with the bosh release at https://github.com/cloudfoundry-incubator/app-autoscaler-release. We're working on enabling deploying as a Cloud Foundry app (and removing the consul dependency) as an easier deployment model for some environments. Kick the tires, give us feedback (we're on slack in #autoscaler) and - it goes without saying - contributions welcome!


Anything else?


On behalf of the autoscaler team, thanks!


Julz

Garden & App Autoscaler PM


--
dr.max Sent from my iPhone http://maximilien.org


Re: The CF App Autoscaler: An Update

Dr Nic Williams <drnicwilliams@...>
 

Congrats on GA announcement!


From: cf-dev@... <cf-dev@...> on behalf of Julz Friedman <julz.friedman@...>
Sent: Wednesday, March 14, 2018 7:38:08 AM
To: cf-dev@...
Subject: [cf-dev] The CF App Autoscaler: An Update
 

Hi cf-dev,


I wanted to give everyone a quick update on the progress of the application autoscaler project as we’ve recently hit some nice milestones.


WHAT IS THIS?


The app autoscaler is a cloud foundry extension (part of the Extensions PMC) that automatically scales cloud foundry applications based on load.


Why would I want it?


Do you like paying for more instances than you need? Do you like not having enough instances when load unexpectedly increases? If you, like me, want as many instances as you need and no more then you, like me, will like App Autoscaler.


Cool, who uses this already?


The big news - and the reason I'm writing this email - is that I'm excited to announce the first GA production deployment of the App Autoscaler: the app autoscaler is now used in production on the SAP Cloud. App Autoscaler’s design was initially based on the BlueMix App Autoscaler project which has been running in production and at scale for quite a while (on BlueMix, unsurprisingly). We’re hoping to be able to match all the features of the BlueMix autoscaler using the open-source autoscaler component in the next couple of months at which point we will deploy the open-source autoscaler to BlueMix also.


Is Autoscaler ready for me to use on my cloud?


I would say cautiously yes. The team feels that autoscaler is ready to go. We want to wait until App Autoscaler is running in two production environments before officially declaring a 1.0 version, but the current version is stable and is working well at scale.


Who is contributing to the App Autoscaler effort?


I'm proud to say that App Autoscaler has been a truly cross-foundation project, and we’ve had committers from IBM, SAP and Fujitsu. We work using a distributed commit model (and when we say distributed we mean distributed -- at various points we’ve had team members based in Australia, India, Seattle, China and the UK!).


LET ME TRY IT OUT!


Ok! You can get started with the bosh release at https://github.com/cloudfoundry-incubator/app-autoscaler-release. We're working on enabling deploying as a Cloud Foundry app (and removing the consul dependency) as an easier deployment model for some environments. Kick the tires, give us feedback (we're on slack in #autoscaler) and - it goes without saying - contributions welcome!


Anything else?


On behalf of the autoscaler team, thanks!


Julz

Garden & App Autoscaler PM



The CF App Autoscaler: An Update

Julz Friedman
 

Hi cf-dev,


I wanted to give everyone a quick update on the progress of the application autoscaler project as we’ve recently hit some nice milestones.


WHAT IS THIS?


The app autoscaler is a cloud foundry extension (part of the Extensions PMC) that automatically scales cloud foundry applications based on load.


Why would I want it?


Do you like paying for more instances than you need? Do you like not having enough instances when load unexpectedly increases? If you, like me, want as many instances as you need and no more then you, like me, will like App Autoscaler.


Cool, who uses this already?


The big news - and the reason I'm writing this email - is that I'm excited to announce the first GA production deployment of the App Autoscaler: the app autoscaler is now used in production on the SAP Cloud. App Autoscaler’s design was initially based on the BlueMix App Autoscaler project which has been running in production and at scale for quite a while (on BlueMix, unsurprisingly). We’re hoping to be able to match all the features of the BlueMix autoscaler using the open-source autoscaler component in the next couple of months at which point we will deploy the open-source autoscaler to BlueMix also.


Is Autoscaler ready for me to use on my cloud?


I would say cautiously yes. The team feels that autoscaler is ready to go. We want to wait until App Autoscaler is running in two production environments before officially declaring a 1.0 version, but the current version is stable and is working well at scale.


Who is contributing to the App Autoscaler effort?


I'm proud to say that App Autoscaler has been a truly cross-foundation project, and we’ve had committers from IBM, SAP and Fujitsu. We work using a distributed commit model (and when we say distributed we mean distributed -- at various points we’ve had team members based in Australia, India, Seattle, China and the UK!).


LET ME TRY IT OUT!


Ok! You can get started with the bosh release at https://github.com/cloudfoundry-incubator/app-autoscaler-release. We're working on enabling deploying as a Cloud Foundry app (and removing the consul dependency) as an easier deployment model for some environments. Kick the tires, give us feedback (we're on slack in #autoscaler) and - it goes without saying - contributions welcome!


Anything else?


On behalf of the autoscaler team, thanks!


Julz

Garden & App Autoscaler PM



Re: Feedback request: Disable logging Client IP’s in the Gorouter logs for compliance with the EU General Data Protection Regulation (GDPR)

Jouke Waleson <jt.waleson@...>
 

1: The custom log format would solve the problems, I think this should be the end-goal.
2: GDPR and security breach investigations are inherently at odds (which users or IP addresses have accessed my services vs. right-to-be-forgotten etc.). These should be conscious decisions on the part of the operators in any case and there should be proper documentation on the possibilities. Whether this is one config option or two is trivial IMHO.
3: Out of the box idea: loggregator holds some in-transit data and caches, but is this considered "capturing data"? The problem could be moved to the systems actually storing and archiving the data, which could implement a filter to remove any IP addresses.

At Mendix we use L7 load balancers btw.

-- 
Jouke


Re: Feedback request: Disable logging Client IP’s in the Gorouter logs for compliance with the EU General Data Protection Regulation (GDPR)

Shubha Anjur Tupil
 

Filip,

I agree that adding a configuration is not a very elegant solution. The Gorouter currently does not support a "Access Log Format" and while it is an elegant solution, the complexity of implementing and rolling it out (testing, maintenance, documentation) seems more than what we are willing to invest in right now. 

As we mention our main use case at this point is to enable our customers to be compliant with the EU General Data Protection Regulation (GDPR) (enforcement begins in May). Do you know if there are other component teams within CF that use a log format structure, we couldn't think of any? 

Regards, 
Shubha



On Tue, Mar 13, 2018 at 12:27 PM, Filip Hanik <fhanik@...> wrote:
having a property to disable one single header logging sounds like overkill. what if I want to disable other headers

Is the gorouter not using an "Access Log Format" string that effectively lets you create the format of the log that you need? That may be worth considering.
this would be fairly common practice around any type of http endpoint logging




On Mon, Mar 12, 2018 at 4:39 PM, Shubha Anjur Tupil <sanjurtupil@...> wrote:

Hello everyone, 

In lieu of The EU General Data Protection Regulation (GDPR), the routing team is investigating adding manifest properties to allow an operator to disable logging the client IP's in the X-Forwarded-For header in the access and error logs for Gorouter.

Enforcement of The EU General Data Protection Regulation (GDPR) (https://www.eugdpr.org/) begins May 28th and imposes steep fines. This law says that companies will be fined if they are capturing PII. The Gorouter currently captures Client IP addresses that are included in that definition. We are exploring manifest properties to allow operators to disable logging the originating client IP.

We need help weighing the options given the use-cases.

Use-cases:

#1. L3 Load Balancer e.g. Amazon NLB in front of the Gorouter 
The client IP is logged and the X-Forwarded-For header might have the originating client IP if an intermediary component or the originating client is adding the header.

#2. L7 Load Balancer e.g. Amazon NLB in front of the Gorotuer 
The client IP is the IP of the load balancer in front of the Gorouter, but the X-Forwarded-For header has the originating client IP.

Option1: We have one manifest property to disable logging both the X-Forwarded-For header and the Client IP in the Gorouter logs.

For use-case #1 this works, and there is only one manifest property to disable all PII being logged by the Gorouter.

For use-case #2 this results in the operator losing the LB IP which might be helpful for troubleshooting.

Option2: We have two separate manifest properties to disable logging the X-Forwarded-For header and/or the Client IP in the Gorouter logs. This is generally more flexible at the cost of user experience.

For use-case #1 this would mean that an operator would have to set two manifest properties instead of one. Both these properties would need to be exposed in Ops Man for PCF installations. It leads to a more cumbersome user experience, adding to the already long list of options.

For use-case #2 this results in a better outcome for an operator in that they still get the information on the LB the request came from, while the originating client IP will not be logged.

Based on the information we have, we are not sure which experience is better and which use-case to optimize for. Some questions we have

  1. What type of LB is more popular in CF environments; L3 or L7? This might help us optimize for Use-case #2 and go with Option #2.
  2. Is there a compelling reason from an operator perspective for installations with a L7 load balancer, making it important to have the Client IP (i.e. they would absolutely want to have Option #2 or maybe don’t care and Option #1 would be ok)?
  3. Does the user experience benefit by having just one option outweigh the benefit of having the client IP for an operator with an L7 Load Balancer? (Flexibility over Experience)

We would love to hear from you if you have thoughts on this. 

Thanks, 

Shannon Coen & Shubha Anjur Tupil
Product Managers, CF Routing





CF routing-release 0.173.0 Release Today

Shubha Anjur Tupil
 

Hi all,

We shipped routing-release 0.173.0 today. 

Release highlights include -   

- Previously if an operator sets `router.disable_http: true` in the Gorouter manifest, requests for a route bound to a route service running as an app on the platform would return a 502. This has now been fixed,  route services will work as expected when `router.disable_http: true` [details](https://www.pivotaltracker.com/story/show/154257824)
- Golang version updated from 1.9.1 to 1.9.4 [details](https://www.pivotaltracker.com/story/show/154885934)

Feel free to reach back to scoen@... and sanjurtupil@... with questions. 

Regards, 
Shannon Coen & Shubha Anjur Tupil 
Product Manager, CF Routing


Re: Feedback request: Disable logging Client IP’s in the Gorouter logs for compliance with the EU General Data Protection Regulation (GDPR)

Filip Hanik
 

having a property to disable one single header logging sounds like overkill. what if I want to disable other headers

Is the gorouter not using an "Access Log Format" string that effectively lets you create the format of the log that you need? That may be worth considering.
this would be fairly common practice around any type of http endpoint logging




On Mon, Mar 12, 2018 at 4:39 PM, Shubha Anjur Tupil <sanjurtupil@...> wrote:

Hello everyone, 

In lieu of The EU General Data Protection Regulation (GDPR), the routing team is investigating adding manifest properties to allow an operator to disable logging the client IP's in the X-Forwarded-For header in the access and error logs for Gorouter.

Enforcement of The EU General Data Protection Regulation (GDPR) (https://www.eugdpr.org/) begins May 28th and imposes steep fines. This law says that companies will be fined if they are capturing PII. The Gorouter currently captures Client IP addresses that are included in that definition. We are exploring manifest properties to allow operators to disable logging the originating client IP.

We need help weighing the options given the use-cases.

Use-cases:

#1. L3 Load Balancer e.g. Amazon NLB in front of the Gorouter 
The client IP is logged and the X-Forwarded-For header might have the originating client IP if an intermediary component or the originating client is adding the header.

#2. L7 Load Balancer e.g. Amazon NLB in front of the Gorotuer 
The client IP is the IP of the load balancer in front of the Gorouter, but the X-Forwarded-For header has the originating client IP.

Option1: We have one manifest property to disable logging both the X-Forwarded-For header and the Client IP in the Gorouter logs.

For use-case #1 this works, and there is only one manifest property to disable all PII being logged by the Gorouter.

For use-case #2 this results in the operator losing the LB IP which might be helpful for troubleshooting.

Option2: We have two separate manifest properties to disable logging the X-Forwarded-For header and/or the Client IP in the Gorouter logs. This is generally more flexible at the cost of user experience.

For use-case #1 this would mean that an operator would have to set two manifest properties instead of one. Both these properties would need to be exposed in Ops Man for PCF installations. It leads to a more cumbersome user experience, adding to the already long list of options.

For use-case #2 this results in a better outcome for an operator in that they still get the information on the LB the request came from, while the originating client IP will not be logged.

Based on the information we have, we are not sure which experience is better and which use-case to optimize for. Some questions we have

  1. What type of LB is more popular in CF environments; L3 or L7? This might help us optimize for Use-case #2 and go with Option #2.
  2. Is there a compelling reason from an operator perspective for installations with a L7 load balancer, making it important to have the Client IP (i.e. they would absolutely want to have Option #2 or maybe don’t care and Option #1 would be ok)?
  3. Does the user experience benefit by having just one option outweigh the benefit of having the client IP for an operator with an L7 Load Balancer? (Flexibility over Experience)

We would love to hear from you if you have thoughts on this. 

Thanks, 

Shannon Coen & Shubha Anjur Tupil
Product Managers, CF Routing




Feature Narrative - Isolated Loggregator Components

Adam Hevenor
 

One of the benefits we see of having re-engineered the persistance layer of Loggregator (see log-cache announcement) is a simpler architecture which can be deployed to multiple times within Cloud Foundry Application Runtime. This makes the concept of isolation segments one that works nicely with Loggregator, which is a feature we have received consistent feedback on over the last year. 

For more details about the problems this solved and work proposed please read the following Feature Narrative


Re: CF CLI v6.35.0 Release Today - service instance sharing; client credentials

Sam Gunaratne
 

Or would a broker only discover this ( if it cared) when the new binding API requests come in with different org/space GUIDs?
 
That is correct, the broker is only aware of service sharing when a bind request occurs with different org/space GUIDs. The act of sharing/unsharing a service instance is handled at the platform level. 

a service offering cannot be made shareable unless the vendor is aware of the concept of shareable services and ships a future version to modifies their /v2/catalog JSON
 
Yes, for a service to be shareable then the feature flag must be enabled in the platform AND the service broker's catalog must have the shareable flag defined. We wanted to be defensive here for a few reasons:
  • Sharing does not make sense for certain services (e.g. app scaling)
  • We wanted broker authors to opt-in to this change so they could understand the security considerations
  • Some hosted services may already assume single org/space service instance consumption for things like billing
  • We didn't want services to break if they received a binding to a shared service.
We can still see value in having cf-admins control certain sharing properties but decided to release the first version without it to get some feedback.

Sam

On Tue, Mar 13, 2018 at 1:21 AM, Dr Nic Williams <drnicwilliams@...> wrote:
I’m not sure what I’m asking is what you’re saying; I think you’re confirming the current implementation - a service offering cannot be made shareable unless the vendor is aware of the concept of shareable services and ships a future version to modifies their /v2/catalog JSON. I’m asking for admins to be able to make up their own decisions without requiring service brokers to be modified.

Question: when a service instance is shared with another space, does a broker API get invoked? Or would a broker only discover this ( if it cared) when the new binding API requests come in with different org/space GUIDs?


From: cf-dev@... <cf-dev@...> on behalf of Dr Nic Williams <drnicwilliams@...>
Sent: Monday, March 12, 2018 6:18:18 PM

To: cf-dev@...
Subject: Re: [cf-dev] CF CLI v6.35.0 Release Today - service instance sharing; client credentials
 
Jay, to confirm, an admin will be able to share a service broker/service offering even if it’s not explicitly supported by the /v2/catalog?


From: cf-dev@... <cf-dev@...> on behalf of Jay Badenhope <jbadenhope@...>
Sent: Monday, March 12, 2018 6:16:51 PM
To: cf-dev@...
Subject: Re: [cf-dev] CF CLI v6.35.0 Release Today - service instance sharing; client credentials
 
Hi Dr Nic,
Building on Denise's response, we also empower the admin to enable/restrict sharing. There are two settings that must be true in order to enable service instance sharing:
1. At the global level: "To enable service instance sharing, an administrator must enable the `service_instance_sharing` flag." https://docs.cloudfoundry.org/devguide/services/sharing-instances.html#enabling
2. At the service level, as you mentioned, "Service brokers must explicitly enable service instance sharing by setting a flag in their service-level metadata object." https://docs.cloudfoundry.org/services/enable-sharing.html#enabling

Matthias,
I'm going to connect with my UAA colleagues and make sure we have a good answer to your question.

Jay

On Mon, Mar 12, 2018 at 3:24 AM, <dyu@...> wrote:
The decision to have service authors opt in was to account for the fact that some services may not be shareable out-of-the-box, primarily due to security considerations. Some brokers may currently be designed to only issue global read+write permissions, but authors may want to change their service permissions model if shareability is now on the cards, for example, read+write for SpaceDevs in the original space, but read-only for spaces that received the instance via sharing.




--

Jay Badenhope

Product Manager
Pivotal Labs
+1.510.517.6973 
LinkedIn | Twitter



1521 - 1540 of 9389