john mcteague <john.mcteague@...>
We are using cf v204 and all loggregators are the same size and config (other than zone).
The distribution of requests across app instances is fairly even as far as I can see.
John.
toggle quoted message
Show quoted text
On 26 May 2015 06:21, "Erik Jasiak" <ejasiak(a)pivotal.io> wrote: Hi John,
I'll be working on this with engineering in the morning; thanks for the details thus far.
This is puzzling: Metrons do not route traffic to dopplers outside their zone today. If all your app instances are spread evenly, and all are serving an equal amount of requests, then I would expect no major variability in Doppler load either.
For completeness, what version of CF are you running? I assume your configurations for all dopplers are roughly the same? All app instances per AZ are serving an equal number of requests?
Thanks, Erik Jasiak
On Monday, May 25, 2015, john mcteague <john.mcteague(a)gmail.com> wrote:
Correct, thanks.
On Mon, May 25, 2015 at 12:01 AM, James Bayer <jbayer(a)pivotal.io> wrote:
ok thanks for the extra detail.
to confirm, during the load test, the http traffic is being routed through zones 4 and 5 app instances on DEAs in a balanced way. however the dopplers associated with zone 4 / 5 are getting a very small amount of load sent their way. is that right?
On Sun, May 24, 2015 at 3:45 PM, john mcteague <john.mcteague(a)gmail.com> wrote:
I am seeing logs from zone 4 and 5 when tailing the logs (*cf logs hello-world | grep App | awk '{ print $2 }'*), I see a relatively even balance between all app instances, yet doppler on zones 1-3 consume far greater cpu resources (15x in some cases) than zones 4 and 5. Generally zones 4 and 5 barely get above 1% utilization.
Running *cf curl /v2/apps/guid/stats | grep host | sort *shows 30 instances, 6 in each zone, a perfect balance.
Each loggregator is running with 8GB RAM and 4vcpus.
John
On Sat, May 23, 2015 at 11:56 PM, James Bayer <jbayer(a)pivotal.io> wrote:
john,
can you say more about "receiving no load at all"? for example, if you restart one of the app instances in zone 4 or zone 5 do you see logs with "cf logs"? you can target a single app instance index to get restarted with using a "cf curl" command for terminating an app index [1]. you can find the details with json output from "cf stats" that should show you the private IPs for the DEAs hosting your app, which should help you figure out which zone each app index is in. http://apidocs.cloudfoundry.org/209/apps/terminate_the_running_app_instance_at_the_given_index.html
if you are seeing logs from zone 4 and zone 5, then what might be happening is that for some reason DEAs in zone 4 or zone 5 are not routable somewhere along the path. reasons for that could be: * DEAs in Zone 4 / Zone 5 not getting apps that are hosted there listed in the routing table * The routing table may be correct, but for some reason the routers cannot reach DEAs in zone 4 or zone 5 with outbound traffic and routers fails over to instances in DEAs 1-3 that it can reach * some other mystery
On Fri, May 22, 2015 at 2:06 PM, john mcteague < john.mcteague(a)gmail.com> wrote:
We map our dea's , dopplers and traffic controllers in 5 logical zones using the various zone properties of doppler, metron_agent and traffic_controller. This aligns to our physical failure domains in openstack.
During a recent load test we discovered that zones 4 and 5 were receiving no load at all, all traffic went to zones 1-3.
What would cause this unbalanced distribution? I have a single app running 30 instances and have verified it is evenly balanced across all 5 zones (6 instances in each). I have additionally verified that each logical zone in the bosh yml contains 1 dea, doppler server and traffic controller.
Thanks, John
_______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
-- Thank you,
James Bayer
-- Thank you,
James Bayer
|