Prior to the consolidation of uaa and the login server in uaa release 2.0, we were running our own login server to handle auth to our platform. We simply reduced the instance of the bundled CF login s

It looks like we were making this harder than it needed to be. Grepping through the configurations: https://github.com/cloudfoundry/cf-release/blob/fc3ea60ec1d9fefa42294ff64bcc81ba76e74481/jobs/cloud_

More than a year ago, there was some discussion and a proposal around adding placement pools so cloud foundry admins could better target how applications were placed on runners: https://docs.google.co

From the discussion on the story, it looks like this won't affect any billing? I want to be sure as we base our billing off event data, and missing an event could mean we'd continue to bill for applic

This is a pretty tricky one, as it takes a long time to manifest. After a while without a restart, cloud_controller_ng take a long time listing org users. For example, in an org with 350 users, before

Yes, we see similar behavior. The top slowest calls (with average times): * GET v2/organizations/([^/?#]+)/users (9s) * GET v2/organizations/([^/?#]+)/managers (7.48s) * GET v2/spaces/([^/?#]+)/manage

Memory doesn't look problematic: [image: Inline image 1]

Zack & Swetha, Thanks for the suggestion, will gather netstat info there next time. Amit, 1:20 delay is due to paging. The total call length for each page is closer to 10s. Just included those two cal

Looks like you're right and we're experiencing the same issue as you are Amit. We're suffering slow DNS lookups. The code is spending all of its time here: /var/vcap/packages/ruby-2.1.6/lib/ruby/2.1.0

Thanks for taking a look, fingers crossed you can see it happen as well. Our 217 install is on stemcell 3026 and our 212 install is on 2989. IaaS is CenturyLink Cloud. -Matt

As an admin, you can cf curl /v2/users https://apidocs.cloudfoundry.org/222/users/list_all_users.html

Amit, Here's a run with the problem manifesting: ... 00248 [200]: ruby 26ms | curl 33ms | nslookup 21ms 00249 [200]: ruby 20ms | curl 32ms | nslookup 14ms 00250 [200]: ruby 18ms | curl 30ms | nslookup

Amit, Here are the results: https://gist.github.com/cholick/b448df07e9e493369d9e The before and after pictures look pretty similar, nothing jumps out as interesting.

Here's the results: https://gist.github.com/cholick/1325fe0f592b1805eba5 The time all between opening connection and opened, with the corresponding ruby source in http.rb's connect method: D "opening

Amit, It looks like consul isn't configured as a recursive resolver. When running the above code, resolving fails on the first nameserver and the script fails. resolv-replace's TCPSocket.open is diffe

Here are the results of the ruby loop with strace: https://gist.github.com/cholick/e7e122e34b524cae5fa1 As expected, things eventually get slow. The bash version of the loop with a new vm each time di

Ah, I misunderstood. Consul isn't configured as a recursive resolver, so for a test with only the 127.0.0.1 in resolve.conf I changed the url in the ruby loop to "uaa.service.cf.internal", which is wh

Gotcha. Yeah, the rescue lets that test run; after 425k lookups, it never got slow. Here's a bit of the strace: https://gist.github.com/cholick/88c756760faca77208f8

So things got a little more interesting yesterday. We have some custom Golang jobs for auditing and some other purposes. A recent change added the consult_agent template to the box with these jobs and

Sure. Here's the Ruby: dns_open: 8 ms | dns_close: 0 ms dns_open: 9 ms | dns_close: 0 ms dns_open: 10 ms | dns_close: 0 ms dns_open: 3013 ms | dns_close: 0 ms dns_open: 5015 ms | dns_close: 0 ms dns_o