Gotcha. Yeah, the rescue lets that test run; after 425k lookups, it never
got slow.

Here's a bit of the strace:
https://gist.github.com/cholick/88c756760faca77208f8


On Wed, Nov 4, 2015 at 11:59 AM, Amit Gupta <agupta(a)pivotal.io> wrote:

Hey Matt,

I wanted to keep using the uaa.SYSTEM_DOMAIN domain, not the internal
domain, for that experiment. I do expect the TCPSocket.open to fail when
talking to 127.0.0.1, what I wanted to know is, in the presence of no other
nameservers, does it eventually start to fail slow again, or does this
behaviour happen only when there are other nameservers. I imagine the
TCPSocket.open is blowing up on the first iteration in the loop and exiting
the script? My bad, can you replace:

TCPSocket.open("--UAA-DOMAIN--", 80).close

with

TCPSocket.open("--UAA-DOMAIN--", 80).close rescue nil

for the experiment with only 127.0.0.1 listed amongst the nameservers?

Yes, something about the move from the first to second nameserver seems
weird. I have seen strace of one case where it times out polling the FD of
the socket it opened to talk to 127.0.0.1, but in one of your straces it
looked like the poll timeout was on polling the FD for the socket for
8.8.8.8. The fact that the problem persists is interesting too, it seems
like it's not just a one-off race condition where someone messed up with FD
it was supposed to be polling.

Thanks,
Amit

On Wed, Nov 4, 2015 at 11:41 AM, Matt Cholick <cholick(a)gmail.com> wrote:

Ah, I misunderstood.

Consul isn't configured as a recursive resolver, so for a test with only
the 127.0.0.1 in resolve.conf I changed the url in the ruby loop to
"uaa.service.cf.internal", which is what uaa is registering for in consul.

I ran through 225k lookups and it never got slow. Here's a bit of the
strace:
https://gist.github.com/cholick/38e02ce3f351847d5fa3

Bother versions of that test definitely pointing to the move from the
first to the second nameserver in ruby, when the first nameserver doesn't
know the address.


On Tue, Nov 3, 2015 at 11:43 PM, Amit Gupta <agupta(a)pivotal.io> wrote:

I looked at the strace, I see you did indeed mean "loop without
resolver on localhost". If you try it with *only* a resolver on localhost,
do you get the eventually consistent DNS slowdown?

On Tue, Nov 3, 2015 at 8:33 PM, Amit Gupta <agupta(a)pivotal.io> wrote:

Thanks Matt!

When you say "the loop without the resolver on local host" did you
mean "the loop with only a resolver on local host"? Sorry if my setup
wasn't clear, but my intention was to only have 127.0.0.1 in
etc/resolv.conf.


On Tuesday, November 3, 2015, Matt Cholick <cholick(a)gmail.com> wrote:

Here are the results of the ruby loop with strace:
https://gist.github.com/cholick/e7e122e34b524cae5fa1

As expected, things eventually get slow. The bash version of the loop
with a new vm each time didn't get slow.

For the loop without a resolver on localhost, it never did get slow.
Though it's hard to prove with something so inconsistent, it hadn't
happened after 100k requests. Here's some of the strace:
https://gist.github.com/cholick/81e58f58e82bfe0a1489

On the final loop, with the SERVFAIL resolver, the issue did
manifest. Here's the trace of that run:
https://gist.github.com/cholick/bd2af46795911cb9f63c

Thanks for digging in on this.


On Mon, Nov 2, 2015 at 6:53 PM, Amit Gupta <agupta(a)pivotal.io> wrote:

Okay, interesting, hopefully we're narrowing in on something.
There's a couple variables I'd like to eliminate, so I wonder if you could
try the following. Also, feel free at any point to let me know if you are
not interesting in digging further.

Try all things as sudo, on one of the CCs.

1. It appears that the problem goes away when the CC process is
restarted, so it feels as though there's some sort of resource that the
ruby process is not able to GC, leading to this problem to show up
eventually, and then go away when restarted. I want to confirm this by
trying two different loops, one where the loop is in bash, spinning up a
new ruby process each time, and one where the loop is in ruby.

* bash loop:

while true; do time /var/vcap/packages/ruby-VERSION/bin/ruby
-r'net/protocol' -e 'TCPSocket.open("--UAA-DOMAIN--", 80).close'; done

* ruby loop

/var/vcap/packages/ruby-VERSION/bin/ruby -r'net/protocol' -e '1.step
do |i|; t = Time.now; TCPSocket.open("--UAA-DOMAIN--", 80).close; puts
"#{i}: #{(1000*(Time.now - t)).round}ms"; end'

For each loop, it might also be useful to run `strace -f -p PID >
SOME_FILE` to see what system calls are going on before and after.

2. Another variable is the interaction with the other nameservers.
For this experiment, I would do `monit stop all` to take one of
your CC's out of commission, so that the router doesn't load balance to it,
because it will likely fail requests given the following changes:

* monit stop all && watch monit summary # wait for all the
processes to be stopped, then ctrl+c to stop the watch
* monit start consul_agent && watch monit summary # wait for
consul_agent to be running, then ctrl+c to stop the watch
* Remove nameservers other than 127.0.0.1 from /etc/resolv.conf
* Run the "ruby loop", and see if it still eventually gets slow
* When it's all done, put the original nameservers back in
/etc/resolv.conf, and `monit restart all`

Again, strace-ing the ruby loop would be interesting here.

3. Finally, consul itself. Dmitriy (BOSH PM) has a little DNS
resolver that can be run instead of consul, that will always SERVFAIL (same
as what you see from consul when you nslookup something), so we can try
that:

* Modify `/var/vcap/bosh/etc/gemrc` to remove the `--local` flag
* Run `gem install rubydns`
* Dump the following into a file, say `/var/vcap/data/tmp/dns.rb`:

#!/usr/bin/env ruby

require "rubydns"

RubyDNS.run_server(listen: [[:udp, "0.0.0.0", 53], [:tcp, "0.0.0.0",
53]]) do
otherwise do |transaction|
transaction.fail!(:ServFail)
end
end

* monit stop all && watch monit summary # and again, wait for
everything to be stopped
* Run it with `ruby /var/vcap/data/tmp/dns.rb`. Note that this
command, and the previous `gem install`, use the system gem/ruby,
not the ruby package used by CC, so it maintains some separation. When
running this, it will spit out logs to the terminal, so one can keep an eye
on what it's doing, make sure it all looks reasonable
* Make sure the original nameservers are back in the
`/etc/resolv.conf` (i.e. ensure this experiment is independent of the
previous experiment).
* Run the "ruby loop" (in a separate shell session on the CC)
* After it's all done, add back `--local` to `
/var/vcap/bosh/etc/gemrc`, and `monit restart all`

Again, run strace on the ruby process.

What I hope we find out is that (1) only the ruby loop is affected,
so it has something to do with long running ruby processes, (2) the problem
is independent of the other nameservers listed in /etc/resolv.conf,
and (3) the problem remains when running Dmitriy's DNS-FAILSERVer instead
of consul on 127.0.0.1:53, to determine that the problem is not
specific to consul.

On Sun, Nov 1, 2015 at 5:18 PM, Matt Cholick <cholick(a)gmail.com>
wrote:

Amit,
It looks like consul isn't configured as a recursive resolver. When
running the above code, resolving fails on the first nameserver and the
script fails. resolv-replace's TCPSocket.open is different from the code
http.rb (and thus api) is using. http.rb is pulling in 'net/protocol'. I
changed the script, replacing the require for 'resolv-replace' to
'net/protocol' to match the cloud controller.

Results:

3286 -- ip_open: 1 ms | ip_close: 0 ms | dns_open: 4 ms |
dns_close: 0 ms
3287 -- ip_open: 1 ms | ip_close: 0 ms | dns_open: 5 ms |
dns_close: 0 ms
3288 -- ip_open: 1 ms | ip_close: 0 ms | dns_open: 6 ms |
dns_close: 0 ms
3289 -- ip_open: 1 ms | ip_close: 0 ms | dns_open: 5 ms |
dns_close: 0 ms
3290 -- ip_open: 1 ms | ip_close: 0 ms | dns_open: 5 ms |
dns_close: 0 ms
3291 -- ip_open: 1 ms | ip_close: 0 ms | dns_open: 5 ms |
dns_close: 0 ms
3292 -- ip_open: 1 ms | ip_close: 0 ms | dns_open: 5 ms |
dns_close: 0 ms
3293 -- ip_open: 1 ms | ip_close: 0 ms | dns_open: 5 ms |
dns_close: 0 ms
3294 -- ip_open: 1 ms | ip_close: 0 ms | dns_open: 2008 ms |
dns_close: 0 ms
3295 -- ip_open: 1 ms | ip_close: 0 ms | dns_open: 4010 ms |
dns_close: 0 ms
3296 -- ip_open: 1 ms | ip_close: 0 ms | dns_open: 4010 ms |
dns_close: 0 ms
3297 -- ip_open: 1 ms | ip_close: 0 ms | dns_open: 4006 ms |
dns_close: 0 ms
3298 -- ip_open: 2 ms | ip_close: 0 ms | dns_open: 4010 ms |
dns_close: 0 ms
3299 -- ip_open: 3 ms | ip_close: 0 ms | dns_open: 4011 ms |
dns_close: 0 ms
3300 -- ip_open: 1 ms | ip_close: 0 ms | dns_open: 4010 ms |
dns_close: 0 ms
3301 -- ip_open: 1 ms | ip_close: 0 ms | dns_open: 4011 ms |
dns_close: 0 ms
3302 -- ip_open: 1 ms | ip_close: 0 ms | dns_open: 4010 ms |
dns_close: 0 ms

And the consul logs, though there's nothing interesting there:
https://gist.github.com/cholick/03d74f7f012e54c50b56


On Fri, Oct 30, 2015 at 5:51 PM, Amit Gupta <agupta(a)pivotal.io>
wrote:

Yup, that's what I was suspecting. Can you try the following now:

1. Add something like the following to your cf manifest:

...
jobs:
...
- name: cloud_controller_z1
...
properties:
consul:
agent:
...
log_level: debug
...

This will set the debug level for the consul agents on your CC job
to debug, so we might be able to see more for its logs. It only sets it on
the job that matters, so when you redeploy, it won't have to roll the whole
deployment. It's okay if you can't/don't want to do this, I'm not sure how
much you want to play around with your environment, but it could be helpful.

2. Add the following line to the bottom of your /etc/resolv.conf

options timeout:4

Let's see if the slow DNS is on the order of 4000ms now, to pin
down where the 5s is exactly coming from.

3. Run the following script on your CC box:

require 'resolv-replace'

UAA_DOMAIN = '--CHANGE-ME--' # e.g. 'uaa.run.pivotal.io'
UAA_IP = '--CHANGE-ME-TOO--' # e.g. '52.21.135.158'

def dur(start_time, end_time)
"#{(1000*(end_time-start_time)).round} ms"
end

1.step do |i|
ip_start = Time.now
s = TCPSocket.open(UAA_IP, 80)
ip_open = Time.now
s.close
ip_close = Time.now

dns_start = Time.now
s = TCPSocket.open(UAA_DOMAIN, 80)
dns_open = Time.now
s.close
dns_close = Time.now

ip_open_dur = dur(ip_start, ip_open)
ip_close_dur = dur(ip_open, ip_close)
dns_open_dur = dur(dns_start, dns_open)
dns_close_dur = dur(dns_open, dns_close)

puts "#{"%04d" % i} -- ip_open: #{ip_open_dur} | ip_close:
#{ip_close_dur} | dns_open: #{dns_open_dur} | dns_close: #{dns_close_dur}"
end

You will need to first nslookup (or otherwise determine) the IP
that the UAA_DOMAIN resolves to (it will be some load balancer, possibly
the gorouter, ha_proxy, or your own upstream LB)

4. Grab the files in /var/vcap/sys/log/consul_agent/

Cheers,
Amit

On Fri, Oct 30, 2015 at 4:29 PM, Matt Cholick <cholick(a)gmail.com>
wrote:

Here's the results:

https://gist.github.com/cholick/1325fe0f592b1805eba5

The time all between opening connection and opened, with the
corresponding ruby source in http.rb's connect method:

D "opening connection to #{conn_address}:#{conn_port}..."

s = Timeout.timeout(@open_timeout, Net::OpenTimeout) {
TCPSocket.open(conn_address, conn_port, @local_host, @local_port)
}
s.setsockopt(Socket::IPPROTO_TCP, Socket::TCP_NODELAY, 1)
D "opened"

I don't know much ruby, so that's as far I drilled down.

-Matt

On Nov 16, 2015, at 04:39, Daniel Mikusa <dmikusa(a)pivotal.io> wrote:

I haven't tried this, but I think it should work.

1.) Make a directory. In that directory put your JAR file, your start script and anything else the app needs to run.
2.) From that directory, run `cf push <app-name> -b java_buildpack -c '$PWD/start-script.sh'`. This will upload your script, the JAR file and everything else in the current directory. It will also tell CF that you specifically want to use the Java build pack (which will install Java) and that you want to use your script to start your app.

What could be tricky about this is your start script. It's going to need to reference JAVA_HOME as `/home/vcap/app/.java-buildpack/open_jdk_jre`, and `java` as `$JAVA_HOME/bin/java` since `java` is not going to be on the $PATH.

You're also going to need to handle some of the things that the JBP would normally do like set -Xmx and other JVM memory settings to keep the JVM from exceeding the containers MEMORY_LIMIT. Note, *all* memory needs to fit under the limit, not just the JVM's heap. In other words, setting -Xmx == MEMORY_LIMIT is 100% wrong.

Beyond that, you'd need to make sure the app is listening on $PORT or if it's not taking web requests, disable that health check (`cf push --no-route` & `cf set-health-check none`).

Dan

On Fri, Nov 13, 2015 at 12:47 AM, dammina sahabandu <dammina(a)adroitlogic.com> wrote:
Hi All,
I have a java application which is a composition of several jars in the running environment. The main jar is invoked by a shell script. Is it possible to push such application into cloud foundry? And if it is possible can you please help me on how to achieve that. A simple guide will be really helpful.

Thank you in advance,
Dammina

Looks like most of your VMs are fine, but one of your etcd VM's agent is
unresponsive. Do "bosh cck" and resolve the unresponsive agent by
recreating from the last known apply spec.

On Mon, Oct 26, 2015 at 7:07 AM, Parthiban Annadurai <
senjiparthi(a)gmail.com> wrote:

Amit, I already tried that troubleshooting options, always same error. As
you mentioned, I have attached the BOSH VMs and BOSH CCK REPORT. Thanks.

On 26 October 2015 at 09:23, Amit Gupta <agupta(a)pivotal.io> wrote:

Try some of these diagnostics:
https://docs.cloudfoundry.org/running/troubleshooting.html#bound-timeout
Also, run "bosh vms --details" and "bosh cck --report" and share the
outputs.

On Mon, Oct 26, 2015 at 12:16 AM, Parthiban Annadurai <
senjiparthi(a)gmail.com> wrote:

Actually, sorry for the previous mails. Since, I have started with CF
v202 release and it thrown some error so am switched to CF v210. But, again
because of some rollback issues we went back with CF v202 itself.
Currently, bosh deploy throws the following, I have attached the Debug Logs
too. Thanks.

Director task 310
Started unknown
Started unknown > Binding deployment. Done (00:00:00)

Started preparing deployment
Started preparing deployment > Binding releases. Done (00:00:00)
Started preparing deployment > Binding existing deployment. Done
(00:00:01)
Started preparing deployment > Binding resource pools. Done (00:00:00)
Started preparing deployment > Binding stemcells. Done (00:00:00)
Started preparing deployment > Binding templates. Done (00:00:00)
Started preparing deployment > Binding properties. Done (00:00:00)
Started preparing deployment > Binding unallocated VMs. Done
(00:00:00)
Started preparing deployment > Binding instance networks. Done
(00:00:00)

Started preparing package compilation > Finding packages to compile.
Done (00:00:00)

Started preparing dns > Binding DNS. Done (00:00:00)

Started creating bound missing vms
Started creating bound missing vms > medium_z1/0
Started creating bound missing vms > medium_z1/1
Started creating bound missing vms > router_z2/0. Done (00:01:09)
Failed creating bound missing vms > medium_z1/1: Timed out sending
`get_task' to f4dfe4e9-c897-4b9b-b7c7-63a908c10190 after 45 seconds
(00:04:03)
Done creating bound missing vms > medium_z1/0 (00:05:14)
Failed creating bound missing vms (00:05:14)

Error 450002: Timed out sending `get_task' to
f4dfe4e9-c897-4b9b-b7c7-63a908c10190 after 45 seconds

On 24 October 2015 at 06:40, Parthiban Annadurai <senjiparthi(a)gmail.com

wrote:

Thanks Amit for your suggestions. Let you know after regenerating the
manifest again.

On 24 October 2015 at 11:47, Amit Gupta <agupta(a)pivotal.io> wrote:

Regenerate your manifest.

On Fri, Oct 23, 2015 at 10:49 PM, Parthiban Annadurai <
senjiparthi(a)gmail.com> wrote:

Okay Amit. Yaa, I changed my CF Version from v202 to v210. Could you
share that metron_agent.deployment property of the manifest which is
required in v210? Thanks.

On 24 October 2015 at 10:57, Amit Gupta <agupta(a)pivotal.io> wrote:

Parthiban,

Your log.txt shows that you're using cf-release version 210, but
your subject message says you're trying v202. Perhaps you've checked out
v202 of cf-release and used the spiff tooling to generate the manifests
from that version. v202 doesn't include the metron_agent.deployment
property in its manifest, which is required in v210.

On Fri, Oct 23, 2015 at 10:07 PM, Parthiban Annadurai <
senjiparthi(a)gmail.com> wrote:

I have created the manifest file using SPIFF tool. Any issues with
that?

On 23 October 2015 at 20:49, Amit Gupta <agupta(a)pivotal.io> wrote:

How did you create your manifest in the first place?

On Fri, Oct 23, 2015 at 8:17 AM, Parthiban Annadurai <
senjiparthi(a)gmail.com> wrote:

After trying the suggestions, now its throws the following error,

Started preparing configuration > Binding configuration. Failed:
Error filling in template `metron_agent.json.erb' for `ha_proxy_z1/0' (line
5: Can't find property `["metron_agent.deployment"]') (00:00:00)

Error 100: Error filling in template `metron_agent.json.erb' for
`ha_proxy_z1/0' (line 5: Can't find property `["metron_agent.deployment"]')

Could anyone on this??

On 22 October 2015 at 18:08, Amit Gupta <agupta(a)pivotal.io>
wrote:

Try running "bosh cck" and recreating VMs from last known apply
spec. You should also make sure that the IPs you're allocating to your
jobs are accessible from the BOSH director VM.

On Thu, Oct 22, 2015 at 5:27 AM, Parthiban Annadurai <
senjiparthi(a)gmail.com> wrote:

Yaa sure Amit. I have attached both the files with this mail.
Could you please? Thanks.



On 21 October 2015 at 19:49, Amit Gupta <agupta(a)pivotal.io>
wrote:

Can you share the output of "bosh vms" and "bosh task 51
--debug". It's preferable if you copy the terminal outputs and paste them
to Gists or Pastebins and share the links.

On Tue, Oct 20, 2015 at 6:18 AM, James Bayer <
jbayer(a)pivotal.io> wrote:

sometimes a message like that is due to networking issues.
does the bosh director and the VM it is creating have an available network
path to reach each other? sometimes ssh'ing in to the VM that is identified
can yield more debug clues.

On Tue, Oct 20, 2015 at 5:09 AM, Parthiban Annadurai <
senjiparthi(a)gmail.com> wrote:

Thanks Bharath and Amit for the helpful solutions. I have
surpassed that error. Now, bosh deploy strucks like in attached image.
Could you anyone please?

Regards

Parthiban A



On 20 October 2015 at 11:57, Amit Gupta <agupta(a)pivotal.io>
wrote:

Bharath, I think you mean to increase the *disk* size on
the compilation VMs, not the memory size.

Parthiban, the error message is happening during
compiling, saying "No space left on device". This means your compilation
VMs are running out of space on disk. This means you need to increase the
allocated disk for your compilation VMs. In the "compilation" section of
your deployment manifest, you can specify "cloud_properties". This is
where you will specify disk size. These "cloud_properties" look the same
as the could_properties specified for a resource pool. Depending on your
IaaS, the structure of the cloud_properties section differs. See here:
https://bosh.io/docs/deployment-manifest.html#resource-pools-cloud-properties

On Mon, Oct 19, 2015 at 11:13 PM, Bharath Posa <
bharathp(a)vedams.com> wrote:

hi parthiban

It seems you are running out of space in your vm in which
you are compiling . try to increase the size of memory in your compilation
vm .

regards
Bharath



On Mon, Oct 19, 2015 at 7:39 PM, Parthiban Annadurai <
senjiparthi(a)gmail.com> wrote:

Hello All,
Thanks All for the helpful suggestions.
Actually, now we r facing the following issue while kicking bosh deploy,

Done compiling packages >
nats/d3a1f853f4980682ed8b48e4706b7280e2b7ce0e (00:01:07)
Failed compiling packages >
buildpack_php/9c72be716ab8629d7e6feed43012d1d671720157: Action Failed
get_task: Task aba21e6a-2031-4a69-5b72-f238ecd07051 result: Compiling
package buildpack_php: Compressing compiled package: Shelling out to tar:
Running command: 'tar czf
/var/vcap/data/tmp/bosh-platform-disk-TarballCompressor-CompressFilesInDir762165297
-C
/var/vcap/data/packages/buildpack_php/9c72be716ab8629d7e6feed43012d1d671720157.1-
.', stdout: '', stderr: '
gzip: stdout: No space left on device
': signal: broken pipe (00:02:41)
Failed compiling packages (00:02:41)

Error 450001: Action Failed get_task: Task
aba21e6a-2031-4a69-5b72-f238ecd07051 result: Compiling package
buildpack_php: Compressing compiled package: Shelling out to tar: Running
command: 'tar czf
/var/vcap/data/tmp/bosh-platform-disk-TarballCompressor-CompressFilesInDir762165297
-C
/var/vcap/data/packages/buildpack_php/9c72be716ab8629d7e6feed43012d1d671720157.1-
.', stdout: '', stderr: '
gzip: stdout: No space left on device
': signal: broken pipe

Could Anyone on this issue?

Regards

Parthiban A

On 19 October 2015 at 14:30, Bharath Posa <
bharathp(a)vedams.com> wrote:

Hi partiban

can u do a checksum of the tar file .


it should come like this *sha1:
b6f596eaff4c7af21cc18a52ef97e19debb00403*

example:

*sha1sum {file}*

regards
Bharath

On Mon, Oct 19, 2015 at 1:12 PM, Eric Poelke <
epoelke(a)gmail.com> wrote:

You actually do not need to download it. if you just
run --

`bosh upload release
https://bosh.io/d/github.com/cloudfoundry/cf-release?v=202`
<https://bosh.io/d/github.com/cloudfoundry/cf-release?v=202>

The director will pull in the release directly from
bosh.io.

--
Thank you,

James Bayer

Sounds good. Thanks Sebastien. I'll watch the thread [1] for updates

[1] https://github.com/cloudfoundry-incubator/cf-abacus/issues/110