1. cf-dev@lists.cloudfoundry.org
  2. Messages
   Date   
Re: Tomcat Internal Proxies with Load Balancer #cf

Matthias Winzeler
 

Hi Jon

We're facing a similar issue for which we have a PR created: https://github.com/cloudfoundry/java-buildpack/pull/546 

From some informal discussions with the the buildpack maintainers, it seems like this is not gonna be merged, because they don't want to support some specific tomcat conf parameters.
We were pointed to providing a custom Tomcat external configuration (as per https://github.com/cloudfoundry/java-buildpack/blob/master/docs/container-tomcat.md#external-tomcat-configuration) that could also be set as standard env group (and thus be operator-friendly), but it looks like we can not ship one external config that works for both Tomcat 7 and Tomcat 8.

Be aware that this would only solve the issue for standalone Tomcat. If your apps use it in the embedded form, they still have to write custom Spring config for it. 
This is probably also true for other app runtimes; I know at least of .NET core which also defines 'KnownProxies.

-Matthias

2018-02-15 17:15 GMT+01:00 Jon Martin <martindesignonline@...>:

We are in the process of standing up several PCF foundations in multiple locations.  Each foundation is fronted by a dedicated load balancer.  For Java application's we were noticing that httpRequest.getScheme() and httpRequest.getServerPort() where returning "http" and "80" respectively even though the load balancer forces the use of "https" and "443".

This led us to Tomcat's remoteIpValve documentation:
https://tomcat.apache.org/tomcat-8.5-doc/api/org/apache/catalina/valves/RemoteIpValve.html

In the Java Buildpack, adding an internalProxies attribute to Tomcat's RemoteIpValve with a regex that matched the ERT subnet of the foundation solved the problem.  A similar approach can be used for embedded Tomcat:
https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#howto-customize-tomcat-behind-a-proxy-server 

The problems we're trying to solve are:

  1. The ERT subnet regex will be different for each foundation
  2. We don't want to maintain multiple Java buildpacks for each foundation to support a single property value in Tomcat's server.xml
  3. The solution needs to solve for both standalone and embedded Tomcat's
We're trying to implement a solution with minimal maintenance and complexity for operators and developers.  That said, we're thinking we could configure an environment variable group with the ERT subnet regex for each foundation. 

  • For embedded Tomcat the environment variable group would be referenced in the application.  For example: server.tomcat.internal-proxies=${ERT_SUBNET}
  • For standalone Tomcat the environment variable group would be referenced in server.xml. e.g. internalProxies="${ERT_SUBNET}
The two questions I have are:
  1. Would we need to set an environment variable group or is the ERT subnet available in some form already at runtime.
  2. Would the Java Buildpack need to be updated to support standalone Tomcat.
  3. I'm assuming other companies have run into this, is there a standard supported means to solve this already?




--
Matthias Winzeler
Mattenenge 8
3011 Bern
mailto: matthias.winzeler@...
Tomcat Internal Proxies with Load Balancer #cf

Jon Martin
 
Edited

We are in the process of standing up several PCF foundations in multiple locations.  Each foundation is fronted by a dedicated load balancer.  For Java application's we were noticing that httpRequest.getScheme() and httpRequest.getServerPort() where returning "http" and "80" respectively even though the load balancer forces the use of "https" and "443".

This led us to Tomcat's remoteIpValve documentation:
https://tomcat.apache.org/tomcat-8.5-doc/api/org/apache/catalina/valves/RemoteIpValve.html

In the Java Buildpack, adding an internalProxies attribute to Tomcat's RemoteIpValve with a regex that matched the ERT subnet of the foundation solved the problem.  A similar approach can be used for embedded Tomcat:
https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#howto-customize-tomcat-behind-a-proxy-server 

The problems we're trying to solve are:

  1. The ERT subnet regex will be different for each foundation
  2. We don't want to maintain multiple Java buildpacks for each foundation to support a single property value in Tomcat's server.xml
  3. The solution needs to solve for both standalone and embedded Tomcat's
We're trying to implement a solution with minimal maintenance and complexity for operators and developers.  That said, we're thinking we could configure an environment variable group with the ERT subnet regex for each foundation. 

  • For embedded Tomcat the environment variable group would be referenced in the application.  For example: server.tomcat.internal-proxies=${ERT_SUBNET}
  • For standalone Tomcat the environment variable group would be referenced in server.xml. e.g. internalProxies="${ERT_SUBNET}
The questions I have are:
  1. Would we need to set an environment variable group or is the ERT subnet available in some form already at runtime.
  2. Would the Java Buildpack need to be updated to support standalone Tomcat.
  3. I'm assuming other companies have run into this, is there a standard supported means to solve this already?
How to rotate the CA certificate of the service_cf_internal_ca

Steffen Uhlig
 

Hi,

has anyone successfully rotated the CA certificate of the service_cf_internal_ca in a cf-deployment? Are there instructions more detailed than those for consul?

We are using cf-deployment at v1.14.0 and put an additional CA certificate (generated with bosh) on top of the existing CA cert. The deployment runs fine, but smoke tests fail with:

Stager is unavailable: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed

More details can be found in this bits-service story.

Any help or pointers are appreciated.

Regards

Steffen
Invoking a cloud foundry API having Basic authentication using Autosys HTTP job

shruthi s
 

Hi,
I have spring boot REST API having Basic authentication hosted in cloud foundry. I am using Autosys HTTP job to call the API periodically and it is not working. The Autosys job is sending in credentials but the logs in PCF app shows null. 
Has anybody tried anything like this ? Any suggestions or ideas will be helpful 
CAB call for February is Wednesday 02/21 @ 8a PST

Michael Maximilien
 

FYI...
 
First off, Happy Valentines Day! 
 
Second, reminder that the CAB call for February is scheduled for next Wednesday 02/21 @ 8a PST.
 
WIP agenda in [1] but summary:
 
  1. CFF updates.
  2. Highlights from different PMCs and teams.
  3. Community talks:
a. CF Documentation Process and How you can contribute? by Benjamin Tarnoff and team, Pivotal
b. CF Abacus Demo / Update by Hristo Lliev and team, SAP
 
I will send one more reminder next week on this list. Zoom soon.
 
Best,
 
PS: I will be at INDEXconf all week next week in San Francisco and want to personally invite you (if around) to CF Day @ INDEX on 02/20, the day before CAB call, at the Moscone Center. We have curated a great line up of CF community speakers on various topics (see agenda: https://goo.gl/p9tKmz) and register for FREE with code: CD1CFDAY here: https://goo.gl/KxuZcF
 
------
dr.max
ibm ☁ 
silicon valley, ca
maximilien.org
 
CVE-2018-1221: Gorouter websocket handling vulnerability

Molly Crowther
 

Hello all,


We have identified a critical-severity issue with Routing. This issue has been assigned CVE-2018-1221.


The public notice can be found here: https://www.cloudfoundry.org/blog/cve-2018-1221/


Description of the exploit

We have discovered a vulnerability with the websockets implementation in Gorouter.


The vulnerability is exposed when:

  • the LB recognizes HTTP requests (L7) and requests a websocket upgrade to Gorouter

  • the LB leverages HTTP keepalive connections to Gorouter


AWS Classic ELBs do no support websocket requests in HTTP mode and so do not expose the vulnerability, while AWS ALBs (Application Load Balancer) do support websockets in HTTP mode and so expose the vulnerability.


Developers with access to cf push are able to exploit this vulnerability in vulnerably installations to gather sensitive data, potentially including usernames and passwords, jwt/OAuth tokens, and UAA client ids and secrets.


How to tell if your CF installation is affected

Your CF installations are affected if ANY of the following are true:

  • Your load balancer recognizes HTTP requests (L7) AND will initiate a websocket handshake with Gorouter when the client initiates one AND the load balancer leverages HTTP keepalive connections to Gorouter

  • You are using AWS Application Load Balancers (ALB)


How to tell if your CF installation is NOT affected

Your CF installations are not affected if ANY of the following are true:

  • You are using LBs that are not HTTP-aware; they are passing through requests to Gorouter over TCP

  • Your load balancer does not use HTTP keepalive connections to Gorouter

  • You are using AWS Classic ELBs (these load balancers must be configured in TCP mode to support websockets; they do not support the websocket protocol in HTTP mode)


How to tell if your installation has been exploited

  • If requests for routes are intermittently routed to unexpected applications and unexpected responses are received. E.g. a request from cf CLI made to log into CF receives an unexpected or non-standard response.

  • If the number of HTTP requests the load balancer has a record of is far more than the number that all Gorouters know about, this could be an indication that Gorouter is sending these HTTP requests over what it considers to be an upgraded websocket connection.


Next Steps

  • Patch the gorouter using the latest routing-release or cf-deployment.

  • Rotate BOSH credentials once the patch is applied.


Please let the Routing team know if you have any questions or need further information.


Thank you,

Molly Crowther

Cloud Foundry Foundation Security Team
Open Service Broker API enhancement for generic actions / extensions needs reviews

Michael Rhodes <rhodie27@...>
 

All,

Any service broker author, service operator or openapi expert that has some free time to look at this proposal and PR please do. It would be greatly appreciated and super helpful. We're looking to move it to validating in the next few weeks.




Thanks,

-- 
-- Michael Rhodes
Do you configure "diego.bbs.convergence.expire_pending_task_duration_in_seconds" on the Diego BBS?

Eric Malm <emalm@...>
 

Hi, all,

Do any of you CF operators ever configure the "diego.bbs.convergence.expire_pending_task_duration_in_seconds" property on the Diego BBS? The Diego team is considering some changes to make task placement more resilient to transient lack of resources or compatible cells, and we may be interested in altering the current default value of 1800 seconds (30 minutes) for this property. Please let me know!

Thanks,
Eric 
Re: Looking for users with user cases for HTTP/2

Shannon Coen
 

I should clarify that we are looking at ingress use cases initially; HTTP/2 from external clients to apps on CF.

Shannon Coen
Product Manager, Cloud Foundry
Pivotal, Inc.

On Mon, Feb 12, 2018 at 5:41 PM, Shannon Coen <scoen@...> wrote:
The CF Routing team has been working on a productized integration with Istio+Envoy for Cloud Foundry. One of the first features we could deliver using this integration is support for HTTP/2. We’d like to hear from users with uses case for this feature, and/or who would be willing to test the feature in an experimental stage and give us feedback.

We’d like to learn:
  • Do you have workloads that currently leverage HTTP/2, or for which you would like to leverage H2?
  • What problem does HTTP/2 solve for you?
  • Do these apps require access to the certificate of the originating client (mutual auth), or is one-way TLS sufficient?
  • Is HTTP/2 without TLS viable for any use cases?
  • Are you using TCP routing to run apps on PAS that leverage H2?
  • If TCP Routing is not a viable solution, why?
Note: Current support for TCP Routing may enable running some workloads that require HTTP/2 or GRPC on Cloud Foundry. As the TCP Router is L4 and so agnostic to application protocols, this solution requires forfeiting HTTP features like context path routing. Also the TCP Router does not currently support TLS termination.

Thank you!

Shannon Coen
Product Manager, Cloud Foundry
Pivotal, Inc.


Looking for users with user cases for HTTP/2

Shannon Coen
 

The CF Routing team has been working on a productized integration with Istio+Envoy for Cloud Foundry. One of the first features we could deliver using this integration is support for HTTP/2. We’d like to hear from users with uses case for this feature, and/or who would be willing to test the feature in an experimental stage and give us feedback.

We’d like to learn:
  • Do you have workloads that currently leverage HTTP/2, or for which you would like to leverage H2?
  • What problem does HTTP/2 solve for you?
  • Do these apps require access to the certificate of the originating client (mutual auth), or is one-way TLS sufficient?
  • Is HTTP/2 without TLS viable for any use cases?
  • Are you using TCP routing to run apps on PAS that leverage H2?
  • If TCP Routing is not a viable solution, why?
Note: Current support for TCP Routing may enable running some workloads that require HTTP/2 or GRPC on Cloud Foundry. As the TCP Router is L4 and so agnostic to application protocols, this solution requires forfeiting HTTP features like context path routing. Also the TCP Router does not currently support TLS termination.

Thank you!

Shannon Coen
Product Manager, Cloud Foundry
Pivotal, Inc.

Re: cf deploy on openstack

Tim Downey
 

 Hi Russell,

It sounds like something went wrong with your Cloud Controller API as it tried to start up. Since it mentions that it failed while executing one of its pre-start scripts, you can "bosh ssh" on to the failing api vm (api/215a8746-2867-415f-892b-f80be09f779c) and inspect the following log files to find out what went wrong:

/var/vcap/sys/log/cloud_controller_ng/pre-start.stderr.log
/var/vcap/sys/log/cloud_controller_ng/pre-start.stdout.log

Hope that helps,
Tim
cf deploy on openstack

Russell Blue
 

Hi,

I deploy cloud foundry openstack Pike. At the time of construction VMs and get error:

Updating instance api: api/215a8746-2867-415f-892b-f80be09f779c (0) (canary) (00:13:04)
L Error: Action Failed get_task: Task 058419ac-7dad-4183-65f0-5dc8a2c1ffba result: 1 of 7 pre-start scripts failed. Failed Jobs: cloud_controller_ng. Successful Jobs: cc_uploader, file_server, policy-server-internal, route_registrar, routing-api, consul_agent


Kind Regards
Metric Drains - A New Way for App Developers to Monitor their Containers #loggregator

Adam Hevenor
 

In service to our mission of providing all users of Cloud Foundry observability of their systems the Loggregator team is excited to offer a new feature for App Developers to use external monitoring tools for monitoring their applications without the need for FIrehose scope. This is accomplished by the use of User Provided Syslog Drains and a new CLI Plugin and we have called the new feature Metric Drains.  

Leveraging the user provided abstraction as well as the tcp based syslog:// protocol is a powerful tool for streaming data off the platform. We have plans to support more data and logs for streaming to app developers this way as well potentially other streaming formats. We'd love your feedback and ideas on either of these. 

A special thanks to several community members that originally suggested and voted for this feature on our github repo!

Requirements: This feature still requires opt in at this time. To utilize this feature you'll need cf-deployment 1.9+ and to set the scalablesyslog.adapter.metrics_to_syslog_enabled feature flag on the cf-syslog-drain deployment. 
Scheduler for PCF #service

shruthi s
 

Hi,
I have a spring boot application implementing a commandline runner and terminates after the process. The main class looks like:
@SpringBootApplication
public class MyClass implements CommandLineRunner{
    @Override
    public void run(String... strings) throws Exception {
if(strings[i].equals("A"){
  do processA();
}
else if(strings[i].equals("B"){
  do processB();
}
System.exit(0);
}
 
public static void main(String[] args) throws Exception{
            SpringApplication.run(CTFRetryNotification.class, args);
    }
}

I run this command: java -jar executable.jar arg0 arg1 -Dspring.profiles.active=dev. It workd fine when i run the command in terminal. 

I am now trying to push this executable jar to PCF as an application, bind it to the scheduler and then schedule a job with command which executes the jar.
I am using the below manifest file to push the app:
---
applications:
- name: my-app
  memory: 1024M
  host: my-app
  domain: my.pcf.domain
  health-check-type: process
  instances: 1
  timeout: 60
  path: executable.jar
  env:
    SPRING_PROFILES_ACTIVE: dev 
    JBP_CONFIG_JAVA_MAIN: '{java_main_class: com.abc.MyClass}'
    JBP_CONFIG_JAVA_MAIN: '{ arguments: "arg0 arg1" }'

Command to push:  cf push -f manifest.yml --no-route

The app starts but suddenly crashes with the following message: 
 [API/0] [OUT] Process has crashed with type: "web"
 [API/0] [OUT] App instance exited with guid 7bd4c9a8-a67e-471d-ad39-43623b6f29cd payload: {"instance"=>"d296990c-ead6-48e0-5308-3657", "index"=>0, "reason"=>"CRASHED", "exit_description"=>"Codependent step exited", "crash_count"=>2, "crash_timestamp"=>1518194091169216190, "version"=>"ed0bbd60-9e1c-4d95-97aa-b750446332d5"}

i am trying to figure out the command i can use to run the jar by using run-task command in cf cli but i cannot get it to work.
cf run-task my-app '$PWD/.java-buildpack/open_jdk_jre/bin/java -Dspring.profiles.active=dev -cp $PWD/. org.springframework.boot.loader.JarLauncher arg0 arg1'

This command fails " Incorrect Usage: unknown flag `c'"
I tried using double quotes around the command but it fails "out of memory".

Has anybody done something like this in PCF? What would be the command i can use to run the jar? Thanks in advance
Re: Should route services really need a route-app mapping?

Krannich, Bernd <bernd.krannich@...>
 

Hi Shannon,

 

Thank you very much for getting back to us on the topic. Quite a few teams here at SAP will be happy to hear about the topic and I’ll make sure to pass your update on to them.

 

Thanks again,

Bernd

 

From: <cf-dev@...> on behalf of Shannon Coen <scoen@...>
Reply-To: "cf-dev@..." <cf-dev@...>
Date: Thursday, 8. February 2018 at 00:19
To: "cf-dev@..." <cf-dev@...>
Subject: Re: [cf-dev] Should route services really need a route-app mapping?

 

[Edited Message Follows]

Currently routes are only known to Gorouter when they are mapped to an app AND the app has running instances. If the app is stopped or crashing, Gorouter will return a 404 for the route. This is because routing configuration is passed through the Diego BBS. 

We've received requests for this several times in the past; see http://cf-dev.70369.x6.nabble.com/cf-dev-Brokered-route-services-only-receiving-traffic-for-routes-mapped-to-started-apps-td4699.html#a4742.

You may be happy to learn that we plan to support this use case in coming months. However, we will offer it through our integration with Istio Pilot and Envoy; we don't plan on offering it using Gorouter. One of the key differences between the current routing subsystem and our integration with Istio is that Istio will receive routes directly from Cloud Controller, while only backend IPs and ports will be obtained from Diego BBS. This will enable us to have the router (Envoy) be configured with a route that is associated with a route service but not an app. More on our initiative here: https://docs.google.com/document/d/1VldkvgWPUh13o5RCNjSvzoPFhbY9BtLqBDdk2k0z9fw/edit.

CF Summit Contributor Registration Time

Chip Childers <cchilders@...>
 

Hey all!

It's time again for contributors to get themselves registered for CF Summit North America in April! 

For those of you that have contributed to a CFF project (any repo in the cloudfoundry, cloudfoundry-incubator, cloudfoundry-attic or openservicebrokerapi GitHub orgs), we've got you covered with free registration. :)

CFNA18CONT

-chip
--
Chip Childers
CTO, Cloud Foundry Foundation
1.267.250.0815
Calling all CF Users!

Chip Childers <cchilders@...>
 

Hey all,

Are you a CF user? If so, we want to hear from you! The CFF is running it's latest user survey to help us get to know the community better. Please take a few minutes to share your story here > https://www.surveymonkey.com/r/CFUserSurvey2018 

Thanks!

-chip
--
Chip Childers
CTO, Cloud Foundry Foundation
1.267.250.0815
Re: Should route services really need a route-app mapping?

Shannon Coen
 
Edited

Currently routes are only known to Gorouter when they are mapped to an app AND the app has running instances. If the app is stopped or crashing, Gorouter will return a 404 for the route. This is because routing configuration is passed through the Diego BBS. 

We've received requests for this several times in the past; see http://cf-dev.70369.x6.nabble.com/cf-dev-Brokered-route-services-only-receiving-traffic-for-routes-mapped-to-started-apps-td4699.html#a4742.

You may be happy to learn that we plan to support this use case in coming months. However, we will offer it through our integration with Istio Pilot and Envoy; we don't plan on offering it using Gorouter. One of the key differences between the current routing subsystem and our integration with Istio is that Istio will receive routes directly from Cloud Controller, while only backend IPs and ports will be obtained from Diego BBS. This will enable us to have the router (Envoy) be configured with a route that is associated with a route service but not an app. More on our initiative here: https://docs.google.com/document/d/1VldkvgWPUh13o5RCNjSvzoPFhbY9BtLqBDdk2k0z9fw/edit.
Re: Change in CF Security RSS Feed

Matthew Kocher <mkocher@...>
 

Thanks Chip!

On Wed, Feb 7, 2018 at 11:03 AM, Chip Childers <cchilders@...> wrote:
Redirects should be in place now and the team is now aware of the importance. Thanks for pointing this out all!

On Wed, Feb 7, 2018 at 3:07 AM Lee Porte via Lists.Cloudfoundry.Org <lee.porte=digital.cabinet-office.gov.uk@lists.cloudfoundry.org> wrote:
I agree with Matthew.

A redirect should really have been put in place when the change was made. The changes broke our CVE alerting mechanism.

Lee

On 7 February 2018 at 00:51, Matthew Kocher <mkocher@...> wrote:
Can we put a redirect in place? Failing that, how about an update posted to the old feed before it goes dark?

Having the old feed go silent is bad form for something that people may be relying on for security updates.




On Tue, Feb 6, 2018 at 8:56 AM, Molly Crowther <mcrowther@...> wrote:
Hello all,

A few weeks ago, the foundation did some re-architecting of the CF blog to improve SEO and searchability. These updates changed the location of the Security RSS feed.

If you are using this feed, the new address is: https://www.cloudfoundry.org/foundryblog/security-advisory/feed/

Please let me know if you have any questions or concerns!

Thanks,
Molly Crowther
CFF Security Team



--
Chip Childers
CTO, Cloud Foundry Foundation
1.267.250.0815


Re: Change in CF Security RSS Feed

Chip Childers <cchilders@...>
 

Redirects should be in place now and the team is now aware of the importance. Thanks for pointing this out all!

On Wed, Feb 7, 2018 at 3:07 AM Lee Porte via Lists.Cloudfoundry.Org <lee.porte=digital.cabinet-office.gov.uk@...> wrote:

I agree with Matthew.

A redirect should really have been put in place when the change was made. The changes broke our CVE alerting mechanism.

Lee

On 7 February 2018 at 00:51, Matthew Kocher <mkocher@...> wrote:
Can we put a redirect in place? Failing that, how about an update posted to the old feed before it goes dark?

Having the old feed go silent is bad form for something that people may be relying on for security updates.




On Tue, Feb 6, 2018 at 8:56 AM, Molly Crowther <mcrowther@...> wrote:
Hello all,

A few weeks ago, the foundation did some re-architecting of the CF blog to improve SEO and searchability. These updates changed the location of the Security RSS feed.

If you are using this feed, the new address is: https://www.cloudfoundry.org/foundryblog/security-advisory/feed/

Please let me know if you have any questions or concerns!

Thanks,
Molly Crowther
CFF Security Team



--
Chip Childers
CTO, Cloud Foundry Foundation
1.267.250.0815
1701 - 1720 of 9429