Error 450002: Timed out pinging to a87d5528-f5c5-4740-9255-a99a91d06e91 after 600 seconds


Daya Shetty <daya.shetty@...>
 

While deploying logsearch-boshrelease on vsphere I'm getting the timeout error while compiling packages. What is the best way to debug these kind of errors considering I cannot ssh to the job vm's as they are in an unknown state. I know it could be network related but hard to debug if I don't know how to debug the issue.

Thanks in advance for any pointers.

Here are the errors:
Started compiling packages
Started compiling packages > redis/cd861700c83a544b6d0bba3b620017d22cf0d726
Started compiling packages > kibana/3e5a1de92426c78c4b53abee09ae3f93d64b445f
Started compiling packages > nginx/07fa264094f0ebb55bcc9990578d9cc159748dc0
Started compiling packages > java8/0ae49bb225b4b74e6d4f9cab3681ebd2f725b9aa
. Failed: Timed out pinging to a87d5528-f5c5-4740-9255-a99a91d06e91 after 600 seconds (00:10:18)
Failed compiling packages > kibana/3e5a1de92426c78c4b53abee09ae3f93d64b445f: Timed out pinging to 375491f0-946a-43e8-9c01-c35b831bc0b6 after 600 seconds (00:10:20)
Failed compiling packages > redis/cd861700c83a544b6d0bba3b620017d22cf0d726: Timed out pinging to f5d4b821-5f09-40cf-8166-f5a315f671da after 600 seconds (00:10:21)
Failed compiling packages > nginx/07fa264094f0ebb55bcc9990578d9cc159748dc0: Timed out pinging to e6436a8b-d864-4bb9-b510-fead80f520fe after 600 seconds (00:10:22)

And here is the bosh vms output for logsearch

Deployment `logsearch'

Director task 296

Task 296 done

+-----------------+--------------------+---------------+-----+
| Job/index | State | Resource Pool | IPs |
+-----------------+--------------------+---------------+-----+
| unknown/unknown | unresponsive agent | | |
| unknown/unknown | unresponsive agent | | |
| unknown/unknown | unresponsive agent | | |
| unknown/unknown | unresponsive agent | | |
+-----------------+--------------------+---------------+-----+


Jim Park
 

At first blush, I'd check the subnet's security group to make sure the
compilation vms have access to the bosh director. As far as shell access, i
don't believe there should be anything preventing you from logging in with
the IAAS credentials.

On Mon, Nov 16, 2015 at 4:42 PM Daya Shetty <daya.shetty(a)bnymellon.com>
wrote:

While deploying logsearch-boshrelease on vsphere I'm getting the timeout
error while compiling packages. What is the best way to debug these kind of
errors considering I cannot ssh to the job vm's as they are in an unknown
state. I know it could be network related but hard to debug if I don't know
how to debug the issue.

Thanks in advance for any pointers.

Here are the errors:
Started compiling packages
Started compiling packages >
redis/cd861700c83a544b6d0bba3b620017d22cf0d726
Started compiling packages >
kibana/3e5a1de92426c78c4b53abee09ae3f93d64b445f
Started compiling packages >
nginx/07fa264094f0ebb55bcc9990578d9cc159748dc0
Started compiling packages >
java8/0ae49bb225b4b74e6d4f9cab3681ebd2f725b9aa
. Failed: Timed out pinging to a87d5528-f5c5-4740-9255-a99a91d06e91 after
600 seconds (00:10:18)
Failed compiling packages >
kibana/3e5a1de92426c78c4b53abee09ae3f93d64b445f: Timed out pinging to
375491f0-946a-43e8-9c01-c35b831bc0b6 after 600 seconds (00:10:20)
Failed compiling packages >
redis/cd861700c83a544b6d0bba3b620017d22cf0d726: Timed out pinging to
f5d4b821-5f09-40cf-8166-f5a315f671da after 600 seconds (00:10:21)
Failed compiling packages >
nginx/07fa264094f0ebb55bcc9990578d9cc159748dc0: Timed out pinging to
e6436a8b-d864-4bb9-b510-fead80f520fe after 600 seconds (00:10:22)

And here is the bosh vms output for logsearch

Deployment `logsearch'

Director task 296

Task 296 done

+-----------------+--------------------+---------------+-----+
| Job/index | State | Resource Pool | IPs |
+-----------------+--------------------+---------------+-----+
| unknown/unknown | unresponsive agent | | |
| unknown/unknown | unresponsive agent | | |
| unknown/unknown | unresponsive agent | | |
| unknown/unknown | unresponsive agent | | |
+-----------------+--------------------+---------------+-----+


Guruprakash Srinivasamurthy <guruprakashsrinivasamurthy@...>
 

It Could be that the Dynamic IP range used by the compilation VMs is being
utilized by another deployment which was also fired in parallel.



On Mon, Nov 16, 2015 at 4:42 PM, Daya Shetty <daya.shetty(a)bnymellon.com>
wrote:

While deploying logsearch-boshrelease on vsphere I'm getting the timeout
error while compiling packages. What is the best way to debug these kind of
errors considering I cannot ssh to the job vm's as they are in an unknown
state. I know it could be network related but hard to debug if I don't know
how to debug the issue.

Thanks in advance for any pointers.

Here are the errors:
Started compiling packages
Started compiling packages >
redis/cd861700c83a544b6d0bba3b620017d22cf0d726
Started compiling packages >
kibana/3e5a1de92426c78c4b53abee09ae3f93d64b445f
Started compiling packages >
nginx/07fa264094f0ebb55bcc9990578d9cc159748dc0
Started compiling packages >
java8/0ae49bb225b4b74e6d4f9cab3681ebd2f725b9aa
. Failed: Timed out pinging to a87d5528-f5c5-4740-9255-a99a91d06e91 after
600 seconds (00:10:18)
Failed compiling packages >
kibana/3e5a1de92426c78c4b53abee09ae3f93d64b445f: Timed out pinging to
375491f0-946a-43e8-9c01-c35b831bc0b6 after 600 seconds (00:10:20)
Failed compiling packages >
redis/cd861700c83a544b6d0bba3b620017d22cf0d726: Timed out pinging to
f5d4b821-5f09-40cf-8166-f5a315f671da after 600 seconds (00:10:21)
Failed compiling packages >
nginx/07fa264094f0ebb55bcc9990578d9cc159748dc0: Timed out pinging to
e6436a8b-d864-4bb9-b510-fead80f520fe after 600 seconds (00:10:22)

And here is the bosh vms output for logsearch

Deployment `logsearch'

Director task 296

Task 296 done

+-----------------+--------------------+---------------+-----+
| Job/index | State | Resource Pool | IPs |
+-----------------+--------------------+---------------+-----+
| unknown/unknown | unresponsive agent | | |
| unknown/unknown | unresponsive agent | | |
| unknown/unknown | unresponsive agent | | |
| unknown/unknown | unresponsive agent | | |
+-----------------+--------------------+---------------+-----+


Guruprakash Srinivasamurthy <guruprakashsrinivasamurthy@...>
 

One way to debug is to open another terminal session while the deployment
is running and do a *bosh task <task ID> --debug* and see whats going on
with your compilation machines. You may also check the vCenter to see if
the compilation VM consoles are accessible.

On Mon, Nov 16, 2015 at 9:04 PM, Guruprakash Srinivasamurthy <
guruprakashsrinivasamurthy(a)gmail.com> wrote:

It Could be that the Dynamic IP range used by the compilation VMs is being
utilized by another deployment which was also fired in parallel.



On Mon, Nov 16, 2015 at 4:42 PM, Daya Shetty <daya.shetty(a)bnymellon.com>
wrote:

While deploying logsearch-boshrelease on vsphere I'm getting the timeout
error while compiling packages. What is the best way to debug these kind of
errors considering I cannot ssh to the job vm's as they are in an unknown
state. I know it could be network related but hard to debug if I don't know
how to debug the issue.

Thanks in advance for any pointers.

Here are the errors:
Started compiling packages
Started compiling packages >
redis/cd861700c83a544b6d0bba3b620017d22cf0d726
Started compiling packages >
kibana/3e5a1de92426c78c4b53abee09ae3f93d64b445f
Started compiling packages >
nginx/07fa264094f0ebb55bcc9990578d9cc159748dc0
Started compiling packages >
java8/0ae49bb225b4b74e6d4f9cab3681ebd2f725b9aa
. Failed: Timed out pinging to a87d5528-f5c5-4740-9255-a99a91d06e91 after
600 seconds (00:10:18)
Failed compiling packages >
kibana/3e5a1de92426c78c4b53abee09ae3f93d64b445f: Timed out pinging to
375491f0-946a-43e8-9c01-c35b831bc0b6 after 600 seconds (00:10:20)
Failed compiling packages >
redis/cd861700c83a544b6d0bba3b620017d22cf0d726: Timed out pinging to
f5d4b821-5f09-40cf-8166-f5a315f671da after 600 seconds (00:10:21)
Failed compiling packages >
nginx/07fa264094f0ebb55bcc9990578d9cc159748dc0: Timed out pinging to
e6436a8b-d864-4bb9-b510-fead80f520fe after 600 seconds (00:10:22)

And here is the bosh vms output for logsearch

Deployment `logsearch'

Director task 296

Task 296 done

+-----------------+--------------------+---------------+-----+
| Job/index | State | Resource Pool | IPs |
+-----------------+--------------------+---------------+-----+
| unknown/unknown | unresponsive agent | | |
| unknown/unknown | unresponsive agent | | |
| unknown/unknown | unresponsive agent | | |
| unknown/unknown | unresponsive agent | | |
+-----------------+--------------------+---------------+-----+