Yeah. I can try. Before that can I know cause of failure for this one. Bcz
it is the current deployment we are having. We want to know what exactly
went wrong with it. Just curious in debugging it. :-)

Bharath
On 6 Apr 2016 23:58, "Jim CF Campbell" <jcampbell(a)pivotal.io> wrote:

Hi Barath,

That release is over a year old. Is there any chance you can upgrade to a

newer release?

Jim


On Wed, Apr 6, 2016 at 12:23 PM, Bharath Posa <bharathp(a)vedams.com> wrote:

Hi Jim

Its version 206

Bharath

On 6 Apr 2016 23:25, "Jim CF Campbell" <jcampbell(a)pivotal.io> wrote:

Hi Bharath,

This looks like it might be older code. What version of Cloud Foundry

are you running?

Thanks,
Jim


On Mon, Apr 4, 2016 at 11:09 PM, Bharath Posa <bharathp(a)vedams.com>

wrote:

Hi all,

we are recently running cloudfoundry in openstack. while recent saw

the metron agent in all the jobs failed . The metron.stderr.log provided
the below log

-------------
panic: 402: Standby Internal Error () [0]

goroutine 28 [running]:

github.com/cloudfoundry/loggregatorlib/servicediscovery.(*serverAddressList).Run(0xc208052120,
0x12a05f200)

/var/vcap/data/compile/metron_agent/loggregator/src/

github.com/cloudfoundry/loggregatorlib/servicediscovery/servicediscovery.go:57
+0x3c3

created by main.main

/var/vcap/data/compile/metron_agent/loggregator/src/metron/main.go:105
+0x1351

goroutine 1 [chan receive]:
main.forwardMessagesToDoppler(0xc20802c040, 0xc2080526c0, 0xc20801ece0)

/var/vcap/data/compile/metron_agent/loggregator/src/metron/main.go:187 +0x7a

main.main()

/var/vcap/data/compile/metron_agent/loggregator/src/metron/main.go:107
+0x137e

goroutine 5 [syscall]:
os/signal.loop()
/usr/local/go/src/os/signal/signal_unix.go:21 +0x1f
created by os/signal.init·1
/usr/local/go/src/os/signal/signal_unix.go:27 +0x35

goroutine 7 [select]:
github.com/cloudfoundry/gunk/workpool.worker(0xc208052600,

0xc20800bd40)

/var/vcap/data/compile/metron_agent/loggregator/src/

github.com/cloudfoundry/gunk/workpool/workpool.go:71 +0x151

created by github.com/cloudfoundry/gunk/workpool.New
/var/vcap/data/compile/metron_agent/loggregator/src/

github.com/cloudfoundry/gunk/workpool/workpool.go:44 +0x189

goroutine 8 [select]:
github.com/cloudfoundry/gunk/workpool.worker(0xc208052600,

0xc20800bd40)

/var/vcap/data/compile/metron_agent/loggregator/src/

github.com/cloudfoundry/gunk/workpool/workpool.go:71 +0x151

created by github.com/cloudfoundry/gunk/workpool.New
/var/vcap/data/compile/metron_agent/loggregator/src/

github.com/cloudfoundry/gunk/workpool/workpool.go:44 +0x189

goroutine 9 [select]:
github.com/cloudfoundry/gunk/workpool.worker(0xc208052600,

0xc20800bd40)

/var/vcap/data/compile/metron_agent/loggregator/src/

github.com/cloudfoundry/gunk/workpool/workpool.go:71 +0x151

created by github.com/cloudfoundry/gunk/workpool.New
/var/vcap/data/compile/metron_agent/loggregator/src/

github.com/cloudfoundry/gunk/workpool/workpool.go:44 +0x189

goroutine 10 [select]:
github.com/cloudfoundry/gunk/workpool.worker(0xc208052600,

0xc20800bd40)

/var/vcap/data/compile/metron_agent/loggregator/src/

github.com/cloudfoundry/gunk/workpool/workpool.go:71 +0x151

created by github.com/cloudfoundry/gunk/workpool.New
/var/vcap/data/compile/metron_agent/loggregator/src/

github.com/cloudfoundry/gunk/workpool/workpool.go:44 +0x189

goroutine 11 [select]:
github.com/cloudfoundry/gunk/workpool.worker(0xc208052600,

0xc20800bd40)

/var/vcap/data/compile/metron_agent/loggregator/src/

github.com/cloudfoundry/gunk/workpool/workpool.go:71 +0x151

created by github.com/cloudfoundry/gunk/workpool.New
/var/vcap/data/compile/metron_agent/loggregator/src/

github.com/cloudfoundry/gunk/workpool/workpool.go:44 +0x189

goroutine 12 [select]:
github.com/cloudfoundry/gunk/workpool.worker(0xc208052600,

0xc20800bd40)

/var/vcap/data/compile/metron_agent/loggregator/src/

github.com/cloudfoundry/gunk/workpool/workpool.go:71 +0x151

created by github.com/cloudfoundry/gunk/workpool.New
/var/vcap/data/compile/metron_agent/loggregator/src/

github.com/cloudfoundry/gunk/workpool/workpool.go:44 +0x189

goroutine 13 [select]:
github.com/cloudfoundry/gunk/workpool.worker(0xc208052600,

0xc20800bd40)

/var/vcap/data/compile/metron_agent/loggregator/src/

github.com/cloudfoundry/gunk/workpool/workpool.go:71 +0x151

created by github.com/cloudfoundry/gunk/workpool.New
/var/vcap/data/compile/metron_agent/loggregator/src/

github.com/cloudfoundry/gunk/workpool/workpool.go:44 +0x189

goroutine 14 [select]:
github.com/cloudfoundry/gunk/workpool.worker(0xc208052600,

0xc20800bd40)

/var/vcap/data/compile/metron_agent/loggregator/src/

github.com/cloudfoundry/gunk/workpool/workpool.go:71 +0x151

created by github.com/cloudfoundry/gunk/workpool.New
/var/vcap/data/compile/metron_agent/loggregator/src/

github.com/cloudfoundry/gunk/workpool/workpool.go:44 +0x189

goroutine 15 [select]:
github.com/cloudfoundry/gunk/workpool.worker(0xc208052600,

0xc20800bd40)

/var/vcap/data/compile/metron_agent/loggregator/src/

github.com/cloudfoundry/gunk/workpool/workpool.go:71 +0x151

created by github.com/cloudfoundry/gunk/workpool.New
/var/vcap/data/compile/metron_agent/loggregator/src/

github.com/cloudfoundry/gunk/workpool/workpool.go:44 +0x189

goroutine 16 [select]:
github.com/cloudfoundry/gunk/workpool.worker(0xc208052600,

0xc20800bd40)

/var/vcap/data/compile/metron_agent/loggregator/src/

github.com/cloudfoundry/gunk/workpool/workpool.go:71 +0x151

created by github.com/cloudfoundry/gunk/workpool.New
/var/vcap/data/compile/metron_agent/loggregator/src/

github.com/cloudfoundry/gunk/workpool/workpool.go:44 +0x189

goroutine 17 [select]:

github.com/cloudfoundry/loggregatorlib/cfcomponent/registrars/collectorregistrar.(*CollectorRegistrar).Run(0xc20800c0c0)

/var/vcap/data/compile/metron_agent/loggregator/src/

github.com/cloudfoundry/loggregatorlib/cfcomponent/registrars/collectorregistrar/collector_registrar.go:41
+0x201

created by main.main

/var/vcap/data/compile/metron_agent/loggregator/src/metron/main.go:78 +0xf7d

goroutine 18 [IO wait]:
net.(*pollDesc).Wait(0xc2080103e0, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc2080103e0, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).accept(0xc208010380, 0x0, 0x7ff2322ffbe0, 0xc20802aba8)
/usr/local/go/src/net/fd_unix.go:419 +0x40b
net.(*TCPListener).AcceptTCP(0xc20802e060, 0x596f54, 0x0, 0x0)
/usr/local/go/src/net/tcpsock_posix.go:234 +0x4e
net/http.tcpKeepAliveListener.Accept(0xc20802e060, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:1976 +0x4c
net/http.(*Server).Serve(0xc2080527e0, 0x7ff232301bb8, 0xc20802e060,

0x0, 0x0)

/usr/local/go/src/net/http/server.go:1728 +0x92
net/http.(*Server).ListenAndServe(0xc2080527e0, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:1718 +0x154
net/http.ListenAndServe(0xc20802abb0, 0xf, 0x7ff232301b90,

0xc20800bc50, 0x0, 0x0)

/usr/local/go/src/net/http/server.go:1808 +0xba

github.com/cloudfoundry/loggregatorlib/cfcomponent.Component.StartMonitoringEndpoints(0xc20801ece0,
0xc20802aac0, 0x9, 0x7ff232301940, 0xa393a8, 0x834ff0, 0xb, 0x0, 0x0, 0x0,
...)

/var/vcap/data/compile/metron_agent/loggregator/src/

github.com/cloudfoundry/loggregatorlib/cfcomponent/component.go:76 +0x719

main.startMonitoringEndpoints(0xc20801ece0, 0xc20802aac0, 0x9,

0x7ff232301940, 0xa393a8, 0x834ff0, 0xb, 0x0, 0x0, 0x0, ...)

/var/vcap/data/compile/metron_agent/loggregator/src/metron/main.go:118 +0x3e

created by main.main

/var/vcap/data/compile/metron_agent/loggregator/src/metron/main.go:80 +0xfb8

goroutine 19 [IO wait]:
net.(*pollDesc).Wait(0xc208010450, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc208010450, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).readFrom(0xc2080103f0, 0xc2080c8000, 0xffff, 0xffff, 0x0,

0x0, 0x0, 0x7ff2322ffbe0, 0xc20802acf8)

/usr/local/go/src/net/fd_unix.go:269 +0x4a1
net.(*UDPConn).ReadFromUDP(0xc20802e068, 0xc2080c8000, 0xffff, 0xffff,

0x20, 0x0, 0x0, 0x0)

/usr/local/go/src/net/udpsock_posix.go:67 +0x124
net.(*UDPConn).ReadFrom(0xc20802e068, 0xc2080c8000, 0xffff, 0xffff,

0xffff, 0x0, 0x0, 0x0, 0x0)

/usr/local/go/src/net/udpsock_posix.go:82 +0x12e

github.com/cloudfoundry/loggregatorlib/agentlistener.(*agentListener).Start(0xc208010fc0)

/var/vcap/data/compile/metron_agent/loggregator/src/

github.com/cloudfoundry/loggregatorlib/agentlistener/agent_listener.go:46
+0x3a5

created by main.main

/var/vcap/data/compile/metron_agent/loggregator/src/metron/main.go:83
+0x101a

goroutine 20 [chan receive]:

metron/legacy_message/legacy_unmarshaller.(*legacyUnmarshaller).Run(0xc20802a9d0,
0xc208052540, 0xc2080523c0)

/var/vcap/data/compile/metron_agent/loggregator/src/metron/legacy_message/legacy_unmarshaller/legacy_unmarshaller.go:30
+0x71

created by main.main

/var/vcap/data/compile/metron_agent/loggregator/src/metron/main.go:86
+0x1096

goroutine 21 [chan receive]:

metron/legacy_message/legacy_message_converter.(*legacyMessageConverter).Run(0xc20802e020,
0xc2080523c0, 0xc208052300)

/var/vcap/data/compile/metron_agent/loggregator/src/metron/legacy_message/legacy_message_converter/legacy_message_converter.go:28
+0x5c

created by main.main

/var/vcap/data/compile/metron_agent/loggregator/src/metron/main.go:87
+0x10dc

goroutine 22 [IO wait]:
net.(*pollDesc).Wait(0xc208010530, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc208010530, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).readFrom(0xc2080104d0, 0xc2080e0000, 0xffff, 0xffff, 0x0,

0x0, 0x0, 0x7ff2322ffbe0, 0xc20802add0)

/usr/local/go/src/net/fd_unix.go:269 +0x4a1
net.(*UDPConn).ReadFromUDP(0xc20802e098, 0xc2080e0000, 0xffff, 0xffff,

0x20, 0x0, 0x0, 0x0)

/usr/local/go/src/net/udpsock_posix.go:67 +0x124
net.(*UDPConn).ReadFrom(0xc20802e098, 0xc2080e0000, 0xffff, 0xffff,

0xffff, 0x0, 0x0, 0x0, 0x0)

/usr/local/go/src/net/udpsock_posix.go:82 +0x12e
metron/eventlistener.(*eventListener).Start(0xc208060400)

/var/vcap/data/compile/metron_agent/loggregator/src/metron/eventlistener/event_listener.go:54
+0x3a0

created by main.main

/var/vcap/data/compile/metron_agent/loggregator/src/metron/main.go:89
+0x1108

goroutine 23 [chan receive]:

github.com/cloudfoundry/dropsonde/dropsonde_unmarshaller.(*dropsondeUnmarshaller).Run(0xc20802c080,
0xc2080525a0, 0xc208052300)

/var/vcap/data/compile/metron_agent/loggregator/src/

github.com/cloudfoundry/dropsonde/dropsonde_unmarshaller/dropsonde_unmarshaller.go:65
+0x71

created by main.main

/var/vcap/data/compile/metron_agent/loggregator/src/metron/main.go:91
+0x114e

goroutine 24 [chan receive]:
metron/message_aggregator.(*messageAggregator).Run(0xc208052180,

0xc208052300, 0xc208052420)

/var/vcap/data/compile/metron_agent/loggregator/src/metron/message_aggregator/message_aggregator.go:57
+0x5c

created by main.main

/var/vcap/data/compile/metron_agent/loggregator/src/metron/main.go:94
+0x11ca

goroutine 25 [chan receive]:
metron/varz_forwarder.(*VarzForwarder).Run(0xc20802c500, 0xc208052420,

0xc208052480)

/var/vcap/data/compile/metron_agent/loggregator/src/metron/varz_forwarder/varz_forwarder.go:31
+0x51

created by main.main

/var/vcap/data/compile/metron_agent/loggregator/src/metron/main.go:97
+0x1237

goroutine 26 [chan receive]:

github.com/cloudfoundry/dropsonde/dropsonde_marshaller.(*dropsondeMarshaller).Run(0xc20801e0a0,
0xc208052480, 0xc2080524e0)

/var/vcap/data/compile/metron_agent/loggregator/src/

github.com/cloudfoundry/dropsonde/dropsonde_marshaller/dropsonde_marshaller.go:59
+0x5f

created by main.main

/var/vcap/data/compile/metron_agent/loggregator/src/metron/main.go:100
+0x12a3

goroutine 27 [chan receive]:
main.signMessages(0xc20802b4b8, 0x8, 0xc2080524e0, 0xc2080526c0)

/var/vcap/data/compile/metron_agent/loggregator/src/metron/main.go:111 +0x7a

created by main.main

/var/vcap/data/compile/metron_agent/loggregator/src/metron/main.go:103
+0x130e

goroutine 33 [syscall, locked to thread]:
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:2232 +0x1

goroutine 31 [IO wait]:
net.(*pollDesc).Wait(0xc208010a00, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc208010a00, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).Read(0xc2080109a0, 0xc208059000, 0x1000, 0x1000, 0x0,

0x7ff2322ffbe0, 0xc2080daa00)

/usr/local/go/src/net/fd_unix.go:242 +0x40f
net.(*conn).Read(0xc20802e150, 0xc208059000, 0x1000, 0x1000, 0x0, 0x0,

0x0)

/usr/local/go/src/net/net.go:121 +0xdc
net/http.noteEOFReader.Read(0x7ff232301d08, 0xc20802e150,

0xc20805af78, 0xc208059000, 0x1000, 0x1000, 0x75b940, 0x0, 0x0)

/usr/local/go/src/net/http/transport.go:1270 +0x6e
net/http.(*noteEOFReader).Read(0xc20801f2e0, 0xc208059000, 0x1000,

0x1000, 0xc208012000, 0x0, 0x0)

<autogenerated>:125 +0xd4
bufio.(*Reader).fill(0xc208053200)
/usr/local/go/src/bufio/bufio.go:97 +0x1ce
bufio.(*Reader).Peek(0xc208053200, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:132 +0xf0
net/http.(*persistConn).readLoop(0xc20805af20)
/usr/local/go/src/net/http/transport.go:842 +0xa4
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:660 +0xc9f

goroutine 32 [select]:
net/http.(*persistConn).writeLoop(0xc20805af20)
/usr/local/go/src/net/http/transport.go:945 +0x41d
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:661 +0xcbc

goroutine 36 [IO wait]:
net.(*pollDesc).Wait(0xc20810a3e0, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc20810a3e0, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).Read(0xc20810a380, 0xc2080fb000, 0x1000, 0x1000, 0x0,

0x7ff2322ffbe0, 0xc2080dad28)

/usr/local/go/src/net/fd_unix.go:242 +0x40f
net.(*conn).Read(0xc20802e318, 0xc2080fb000, 0x1000, 0x1000, 0x0, 0x0,

0x0)

/usr/local/go/src/net/net.go:121 +0xdc
net/http.noteEOFReader.Read(0x7ff232301d08, 0xc20802e318,

0xc20805ad68, 0xc2080fb000, 0x1000, 0x1000, 0x75b940, 0x0, 0x0)

/usr/local/go/src/net/http/transport.go:1270 +0x6e
net/http.(*noteEOFReader).Read(0xc2080dd000, 0xc2080fb000, 0x1000,

0x1000, 0xc208012000, 0x0, 0x0)

<autogenerated>:125 +0xd4
bufio.(*Reader).fill(0xc2081086c0)
/usr/local/go/src/bufio/bufio.go:97 +0x1ce
bufio.(*Reader).Peek(0xc2081086c0, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:132 +0xf0
net/http.(*persistConn).readLoop(0xc20805ad10)
/usr/local/go/src/net/http/transport.go:842 +0xa4
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:660 +0xc9f

goroutine 37 [select]:
net/http.(*persistConn).writeLoop(0xc20805ad10)
/usr/local/go/src/net/http/transport.go:945 +0x41d
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:661 +0xcbc
----------------------

I tried to restart it using monit restart still it is failing .
any body got idea on this error


regards
Bharath

--
Jim Campbell | Product Manager | Cloud Foundry | Pivotal.io |

303.618.0963

--
Jim Campbell | Product Manager | Cloud Foundry | Pivotal.io | 303.618.0963