Re: Doppler zoning query
john mcteague <john.mcteague@...>
I am seeing logs from zone 4 and 5 when tailing the logs (*cf logs
hello-world | grep App | awk '{ print $2 }'*), I see a relatively even
balance between all app instances, yet doppler on zones 1-3 consume far
greater cpu resources (15x in some cases) than zones 4 and 5. Generally
zones 4 and 5 barely get above 1% utilization.
Running *cf curl /v2/apps/guid/stats | grep host | sort *shows 30
instances, 6 in each zone, a perfect balance.
Each loggregator is running with 8GB RAM and 4vcpus.
John
toggle quoted message
Show quoted text
hello-world | grep App | awk '{ print $2 }'*), I see a relatively even
balance between all app instances, yet doppler on zones 1-3 consume far
greater cpu resources (15x in some cases) than zones 4 and 5. Generally
zones 4 and 5 barely get above 1% utilization.
Running *cf curl /v2/apps/guid/stats | grep host | sort *shows 30
instances, 6 in each zone, a perfect balance.
Each loggregator is running with 8GB RAM and 4vcpus.
John
On Sat, May 23, 2015 at 11:56 PM, James Bayer <jbayer(a)pivotal.io> wrote:
john,
can you say more about "receiving no load at all"? for example, if you
restart one of the app instances in zone 4 or zone 5 do you see logs with
"cf logs"? you can target a single app instance index to get restarted with
using a "cf curl" command for terminating an app index [1]. you can find
the details with json output from "cf stats" that should show you the
private IPs for the DEAs hosting your app, which should help you figure out
which zone each app index is in.
http://apidocs.cloudfoundry.org/209/apps/terminate_the_running_app_instance_at_the_given_index.html
if you are seeing logs from zone 4 and zone 5, then what might be
happening is that for some reason DEAs in zone 4 or zone 5 are not routable
somewhere along the path. reasons for that could be:
* DEAs in Zone 4 / Zone 5 not getting apps that are hosted there listed in
the routing table
* The routing table may be correct, but for some reason the routers cannot
reach DEAs in zone 4 or zone 5 with outbound traffic and routers fails over
to instances in DEAs 1-3 that it can reach
* some other mystery
On Fri, May 22, 2015 at 2:06 PM, john mcteague <john.mcteague(a)gmail.com>
wrote:We map our dea's , dopplers and traffic controllers in 5 logical zones
using the various zone properties of doppler, metron_agent and
traffic_controller. This aligns to our physical failure domains in
openstack.
During a recent load test we discovered that zones 4 and 5 were receiving
no load at all, all traffic went to zones 1-3.
What would cause this unbalanced distribution? I have a single app
running 30 instances and have verified it is evenly balanced across all 5
zones (6 instances in each). I have additionally verified that each logical
zone in the bosh yml contains 1 dea, doppler server and traffic controller.
Thanks,
John
_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
--
Thank you,
James Bayer