sporadic connection resets between login and uaa
Sievers, Jan <jan.sievers@...>
Hi,
while running the CF 207 smoke and acceptance tests repeatedly, we noticed sporadic connection resets during 'cf login' (see log snippet from login log below). The connection reset is happening on the login machine when it's doing an HTTP POST to http://uaa.cf.<DOMAIN>/authenticate (via load balancer, and getting a connection reset from the load balancer). This is happening ~ 1 out of 5 times if we run the smoke tests every 5 minutes. We found that adding -Dhttp.keepAlive=false to JAVA_OPTS in /var/vcap/jobs/login/bin/login_ctl works around the problem. Otherwise, by default there is a pool of 5 connections being kept alive and reused. We use an F5 BigIP load balancer with 300 seconds socket idle timeout configured. Could this be a bug with stale connections being reused by the HTTP client on the login machine? Best Regards, Jan --- log snippet from login machine --- [2015-05-08 08:07:52.787] login - 9054 [http-bio-8080-exec-2] .... DEBUG --- DispatcherServlet: DispatcherServlet with name 'spring' processing POST request for [/error500] [2015-05-08 08:07:52.787] login - 9054 [http-bio-8080-exec-2] .... DEBUG --- RequestMappingHandlerMapping: Looking up handler method for path /error500 [2015-05-08 08:07:52.787] login - 9054 [http-bio-8080-exec-2] .... DEBUG --- RequestMappingHandlerMapping: Returning handler method [public java.lang.String org.cloudfoundry.identity.uaa.login.HomeController.error500(org.springframework.ui.Model,javax.servlet.http.HttpServletRequest)] [2015-05-08 08:07:52.787] login - 9054 [http-bio-8080-exec-2] .... ERROR --- HomeController: Internal error org.springframework.web.client.ResourceAccessException: I/O error on POST request for "http://uaa.cf.<DOMAIN>/authenticate":Connection reset; nested exception is java.net.SocketException: Connection reset at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:567) at org.springframework.security.oauth2.client.OAuth2RestTemplate.doExecute(OAuth2RestTemplate.java:128) at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:512) at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:454) at org.cloudfoundry.identity.uaa.login.RemoteUaaAuthenticationManager.authenticate(RemoteUaaAuthenticationManager.java:137) at org.cloudfoundry.identity.uaa.authentication.AuthzAuthenticationFilter.doFilter(AuthzAuthenticationFilter.java:138) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:50) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192) at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:344) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:261) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950) at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:683) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) [37/1995] at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1070) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:744) Caused by: java.net.SocketException: Connection reset at java.net.SocketInputStream.read(SocketInputStream.java:196) at java.net.SocketInputStream.read(SocketInputStream.java:122) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:136) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:152) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:270) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:260) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:161) at sun.reflect.GeneratedMethodAccessor121.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.http.impl.conn.CPoolProxy.invoke(CPoolProxy.java:138) at com.sun.proxy.$Proxy45.receiveResponseHeader(Unknown Source) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:271) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:254) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:195) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:86) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) at org.springframework.http.client.HttpComponentsClientHttpRequest.executeInternal(HttpComponentsClientHttpRequest.java:91) at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:48) at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:53) at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:551) ... 33 more |
|
Re: Cloudfoundry UAA / Questions
Daniel Jones
+1 for password expiry; that'd be really handy to have. On Sun, May 31, 2015 at 2:43 AM, Frans Thamura <frans(a)meruvian.org> wrote: fyi, we use UAA for our social login , take a look www.merv.id -- Regards, Daniel Jones EngineerBetter.com |
|
R: Re: Monitor all outbound connections from apps in warden
Michael Grifalconi <michael.grifalconi@...>
Hello, thank you for the hint!
toggle quoted message
Show quoted text
I'd prefer to do something at application level, like a shell script to run in parallel to the application that every X seconds prints the output of netstat, as the standard output is taken as a log on CF apps. Is it possible? (I'm really sorry and embarrassed about the spam after my email signature, this is due to my University and I can't avoid it :/ ) Thank you, Michael Il 29/05/15 20:06, Dieu Cao <dcao(a)pivotal.io> ha scritto:
|
|
be there for IPM
Filip Hanik
i may miss standup but I'll be there for IPM
|
|
Re: Cloudfoundry UAA / Questions
Frans Thamura
fyi, we use UAA for our social login , take a look www.merv.id
toggle quoted message
Show quoted text
F -- Frans Thamura (曽志胜) Java Champion Shadow Master and Lead Investor Meruvian. Integrated Hypermedia Java Solution Provider. Mobile: +628557888699 Blog: http://blogs.mervpolis.com/roller/flatburger (id) FB: http://www.facebook.com/meruvian TW: http://www.twitter.com/meruvian / @meruvian Website: http://www.meruvian.org "We grow because we share the same belief." On Sun, May 31, 2015 at 1:11 AM, Filip Hanik <fhanik(a)pivotal.io> wrote:
For users created in UAA database, are there any policies we could apply |
|
Re: Cloudfoundry UAA / Questions
Filip Hanik
1. For users created in UAA database, are there any policies we could
apply regarding password expiry/strength of the password/lockout on repeated retry failures etc..? Currently there is a password score calculator. There is a feature being implemented for a more clearly configurable password strength. Expect it to be in the next release. Lockout is implemented, and will also be configurable in the next release. 2. Is there any pluggable mechanism for user creation in UAA that we could use to create them say in AD – instead of in UAA user database? The UAA can integrate with LDAP (AD) or with SAML IDPs. When you use one of these authentication mechanism, a shadow account will be created in the UAA. These users will only be able to authenticate against their respective identity providers. 3. Is there any work/pocs done on UAA integration with Shibboleth Identity provider to have federated identity? I.e. Integration with identity providers behind firewalls? I believe Shibboleth is a SAML v2 provider, so it should be able to be configured like any other provider. 4. Is UAA HA/DR capable if the underlying user database is replicated? Basically does it boil down to underlying UAA database HA/DR and any tenants identity provider’s HA/DR capability? Yes, that is how we run our UAA in production. It's backed by a HA/DR database. 5. Other than notion of Zones/Multi-tenants are there any advantages of using UAA over plain Spring Security OAuth2/Spring Cloud Security? Yes, most of the work has already been done for you. On Sat, May 30, 2015 at 11:58 AM, Reddy, Satyapal <satyapal.reddy(a)emc.com> wrote: Looking into using UAA and have couple of questions: |
|
Cloudfoundry UAA / Questions
Satyapal Reddy
Looking into using UAA and have couple of questions:
1. For users created in UAA database, are there any policies we could apply regarding password expiry/strength of the password/lockout on repeated retry failures etc..? 2. Is there any pluggable mechanism for user creation in UAA that we could use to create them say in AD – instead of in UAA user database? 3. Is there any work/pocs done on UAA integration with Shibboleth Identity provider to have federated identity? I.e. Integration with identity providers behind firewalls? 4. Is UAA HA/DR capable if the underlying user database is replicated? Basically does it boil down to underlying UAA database HA/DR and any tenants identity provider’s HA/DR capability? 5. Other than notion of Zones/Multi-tenants are there any advantages of using UAA over plain Spring Security OAuth2/Spring Cloud Security? Thanks Satya |
|
Re: Cloud Foundry install documentation
Tom Sherrod <tom.sherrod@...>
Kim and James, thanks for responding and asking clarifying questions.
toggle quoted message
Show quoted text
Regarding latest, the end manifest ended up with lucid stemcell references. I changed them to trusty and then the deploy failed with cloud property object issues, like range missing. It went downhill from there. Kim, thanks for the pointer to github docs. I will start back from a clean slate and issue/update where possible. The first example issue was name missing from the resulting deploy manifest. I need to line up the questions and possible edits with the correct page, along with re-testing. Best, Tom On Fri, May 29, 2015 at 8:35 PM, Kim Hoffman <khoffman(a)pivotal.io> wrote:
Hi Tom, |
|
Re: Gorouter throughput
Simon Johansson <simon@...>
Great writeup Dieu, thanks!
toggle quoted message
Show quoted text
On Fri, May 29, 2015 at 10:50 PM, Dieu Cao <dcao(a)pivotal.io> wrote:
Yes, we recently ran some performance tests with the gorouter. |
|
Re: Cloud Foundry install documentation
Kim Hoffman <khoffman@...>
Hi Tom,
toggle quoted message
Show quoted text
The docs team was also not aware that this document isn't working. It looks like there have been various tweaks to this doc over time, including pretty recently. Do you have any more details you could share? If you do find that you know what's wrong with the topic, we welcome pull requests or git issues against our documentation. You can find this specific doc here: https://github.com/cloudfoundry/docs-deploying-cf/blob/master/openstack/install_cf_openstack.html.md Thanks! Kim On Fri, May 29, 2015 at 4:28 PM, James Bayer <jbayer(a)pivotal.io> wrote:
i was not aware that the documentation was not working for the latest |
|
Re: Cloud Foundry install documentation
James Bayer
i was not aware that the documentation was not working for the latest
toggle quoted message
Show quoted text
cf-release versions. did you find that the documentation was out of date? On Fri, May 29, 2015 at 7:30 AM, Tom Sherrod <tom.sherrod(a)gmail.com> wrote:
What version of Cloud Foundry works with --
Thank you, James Bayer |
|
Re: Gorouter throughput
Dieu Cao <dcao@...>
Yes, we recently ran some performance tests with the gorouter.
You can find the results in this doc [1] and raw results and additional graphs [2] Related stories in tracker [3][4] [image: Inline image 1] [image: Inline image 2] [1] Google doc: https://docs.google.com/document/d/18rrh0MNjCljd1Kt4L2mZuV2GPvO-SZZ2rk2eE769JZY/edit?usp=sharing [2] Excel sheet: https://docs.google.com/spreadsheets/d/1uulkoXtlV7haH0oroEKz7zQ5hxeeY8eYA3D7I7jAX9g/edit?usp=sharing [3] https://www.pivotaltracker.com/story/show/92895056 [4] https://www.pivotaltracker.com/story/show/93362206 On Fri, May 29, 2015 at 1:14 PM, john mcteague <john.mcteague(a)gmail.com> wrote: Is there any perf test data on the gorouter? Number of parallel |
|
Gorouter throughput
john mcteague <john.mcteague@...>
Is there any perf test data on the gorouter? Number of parallel connections
possible given a specific VM size would help us in our sizing efforts. Thanks |
|
Re: Setting Org Manager via API
Daniel Jones
Hi Dieu,
toggle quoted message
Show quoted text
Thanks for the response. Yep, that answers my question, and addresses some of the assumptions I'd been making. Good to hear the changes will be coming in v3. Have a good weekend! On Fri, May 29, 2015 at 6:37 PM, Dieu Cao <dcao(a)pivotal.io> wrote:
Yes, we'd recommend using only the role methods on the organization --
Regards, Daniel Jones EngineerBetter.com |
|
Re: Is CF etcd job using the latest etcd 2.0 release?
John Wong
Thank you Dieu.
toggle quoted message
Show quoted text
This is great to know. John On Fri, May 29, 2015 at 1:11 PM, Dieu Cao <dcao(a)pivotal.io> wrote:
In cf-release v210 [1] we upgraded etcd to v2.0.1. |
|
Re: Monitor all outbound connections from apps in warden
Dieu Cao <dcao@...>
You could set up a security group that logs all outbound connections.
These are logged on the DEAs. You would then need to correlate the warden handle with the application. I'm working with the docs team on getting this feature properly documented. Relevant stories where this feature was added. [1] https://www.pivotaltracker.com/story/show/73905126 [2] https://www.pivotaltracker.com/story/show/90078842 I don't know how you would do this via buildpacks. -Dieu CF Runtime PM On Fri, May 29, 2015 at 6:59 AM, Michael Grifalconi < michael.grifalconi(a)studenti.unimi.it> wrote: Hello all, |
|
Re: Setting Org Manager via API
Dieu Cao <dcao@...>
Yes, we'd recommend using only the role methods on the organization
resource as a general rule. Those roles are org specific so yes the association only exists to facillitate granting of org roles. I'm not sure if that answers your question? The /v2 end points have a certain amount of difficult to change meta magic. It's a known issue that deletes are returning 201's but it is actually surprisingly hard to fix. We'll address this as things move over to /v3 end points. -Dieu On Fri, May 29, 2015 at 1:59 AM, Daniel Jones < daniel.jones(a)engineerbetter.com> wrote: Thanks for the clarification. |
|
Re: Is CF etcd job using the latest etcd 2.0 release?
Dieu Cao <dcao@...>
In cf-release v210 [1] we upgraded etcd to v2.0.1.
toggle quoted message
Show quoted text
We're currently running it with 3 nodes in our CI and Prod environments. We'll be looking at upgrading to v2.0.11 in the next couple of weeks. -Dieu [1] https://github.com/cloudfoundry/cf-release/releases/tag/v210 On Fri, May 29, 2015 at 8:08 AM, John Wong <gokoproject(a)gmail.com> wrote:
I remember in the older version of CF release, CF was using an older |
|
Is CF etcd job using the latest etcd 2.0 release?
John Wong
I remember in the older version of CF release, CF was using an older
version of etcd (v.0,4.x), which has issue running etcd in cluster mode from time to time, and there was a discussion that we would move to v0.5.x which has been renamed to v2.X. Have we upgraded to the latest release? Previous discussion: https://groups.google.com/a/cloudfoundry.org/forum/#!searchin/vcap-dev/etcd$20Raft/vcap-dev/fQ0PD3oPh90/joW4fHBx5H0J Thanks. John |
|
Re: api and api_worker jobs fail to bosh update, but monit start OK
For similar further diagnostics, I've updated docs with a working way to be
toggle quoted message
Show quoted text
able to invoke steno-prettify to get cc_ng logs as timestamps, cf https://github.com/cloudfoundry/docs-running-cf/pull/7/files BTW, has anyone tried a log conversion for cc_ng logs collected into a syslog endpoint (such as a logstash filter that invokes the steno-prettify command or an equivalent ) ? Besides, I ran into the bin/console pry shell that seems to open interesting diagnostic/db interaction opportunities. If there are example of common cases using it, I'd be happy to test and report them in cf docs, to refine [2] [1] https://github.com/cloudfoundry/docs-running-cf/pull/7/files [2] https://github.com/Orange-OpenSource/docs-running-cf/commit/cc46acc26eff50988a652acc3582c7573d6c8c8e
|
|