Re: cloud_controller_ng performance degrades slowly over time
|
Amit Kumar Gupta
Okay, interesting, hopefully we're narrowing in on something. There's a
toggle quoted message
Show quoted text
couple variables I'd like to eliminate, so I wonder if you could try the following. Also, feel free at any point to let me know if you are not interesting in digging further. Try all things as sudo, on one of the CCs. 1. It appears that the problem goes away when the CC process is restarted, so it feels as though there's some sort of resource that the ruby process is not able to GC, leading to this problem to show up eventually, and then go away when restarted. I want to confirm this by trying two different loops, one where the loop is in bash, spinning up a new ruby process each time, and one where the loop is in ruby. * bash loop: while true; do time /var/vcap/packages/ruby-VERSION/bin/ruby -r'net/protocol' -e 'TCPSocket.open("--UAA-DOMAIN--", 80).close'; done * ruby loop /var/vcap/packages/ruby-VERSION/bin/ruby -r'net/protocol' -e '1.step do |i|; t = Time.now; TCPSocket.open("--UAA-DOMAIN--", 80).close; puts "#{i}: #{(1000*(Time.now - t)).round}ms"; end' For each loop, it might also be useful to run `strace -f -p PID > SOME_FILE` to see what system calls are going on before and after. 2. Another variable is the interaction with the other nameservers. For this experiment, I would do `monit stop all` to take one of your CC's out of commission, so that the router doesn't load balance to it, because it will likely fail requests given the following changes: * monit stop all && watch monit summary # wait for all the processes to be stopped, then ctrl+c to stop the watch * monit start consul_agent && watch monit summary # wait for consul_agent to be running, then ctrl+c to stop the watch * Remove nameservers other than 127.0.0.1 from /etc/resolv.conf * Run the "ruby loop", and see if it still eventually gets slow * When it's all done, put the original nameservers back in /etc/resolv.conf, and `monit restart all` Again, strace-ing the ruby loop would be interesting here. 3. Finally, consul itself. Dmitriy (BOSH PM) has a little DNS resolver that can be run instead of consul, that will always SERVFAIL (same as what you see from consul when you nslookup something), so we can try that: * Modify `/var/vcap/bosh/etc/gemrc` to remove the `--local` flag * Run `gem install rubydns` * Dump the following into a file, say `/var/vcap/data/tmp/dns.rb`: #!/usr/bin/env ruby require "rubydns" RubyDNS.run_server(listen: [[:udp, "0.0.0.0", 53], [:tcp, "0.0.0.0", 53]]) do otherwise do |transaction| transaction.fail!(:ServFail) end end * monit stop all && watch monit summary # and again, wait for everything to be stopped * Run it with `ruby /var/vcap/data/tmp/dns.rb`. Note that this command, and the previous `gem install`, use the system gem/ruby, not the ruby package used by CC, so it maintains some separation. When running this, it will spit out logs to the terminal, so one can keep an eye on what it's doing, make sure it all looks reasonable * Make sure the original nameservers are back in the `/etc/resolv.conf` (i.e. ensure this experiment is independent of the previous experiment). * Run the "ruby loop" (in a separate shell session on the CC) * After it's all done, add back `--local` to `/var/vcap/bosh/etc/gemrc`, and `monit restart all` Again, run strace on the ruby process. What I hope we find out is that (1) only the ruby loop is affected, so it has something to do with long running ruby processes, (2) the problem is independent of the other nameservers listed in /etc/resolv.conf, and (3) the problem remains when running Dmitriy's DNS-FAILSERVer instead of consul on 127.0.0.1:53, to determine that the problem is not specific to consul. On Sun, Nov 1, 2015 at 5:18 PM, Matt Cholick <cholick(a)gmail.com> wrote:
Amit, |
|
|