Hi Bharath,

This looks like it might be older code. What version of Cloud Foundry are
you running?

Thanks,
Jim


On Mon, Apr 4, 2016 at 11:09 PM, Bharath Posa <bharathp(a)vedams.com> wrote:

Hi all,

we are recently running cloudfoundry in openstack. while recent saw the
metron agent in all the jobs failed . The metron.stderr.log provided the
below log

-------------
panic: 402: Standby Internal Error () [0]

goroutine 28 [running]:

github.com/cloudfoundry/loggregatorlib/servicediscovery.(*serverAddressList).Run(0xc208052120,
0x12a05f200)
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/loggregatorlib/servicediscovery/servicediscovery.go:57
+0x3c3
created by main.main

/var/vcap/data/compile/metron_agent/loggregator/src/metron/main.go:105
+0x1351

goroutine 1 [chan receive]:
main.forwardMessagesToDoppler(0xc20802c040, 0xc2080526c0, 0xc20801ece0)

/var/vcap/data/compile/metron_agent/loggregator/src/metron/main.go:187 +0x7a
main.main()

/var/vcap/data/compile/metron_agent/loggregator/src/metron/main.go:107
+0x137e

goroutine 5 [syscall]:
os/signal.loop()
/usr/local/go/src/os/signal/signal_unix.go:21 +0x1f
created by os/signal.init·1
/usr/local/go/src/os/signal/signal_unix.go:27 +0x35

goroutine 7 [select]:
github.com/cloudfoundry/gunk/workpool.worker(0xc208052600, 0xc20800bd40)
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/gunk/workpool/workpool.go:71 +0x151
created by github.com/cloudfoundry/gunk/workpool.New
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/gunk/workpool/workpool.go:44 +0x189

goroutine 8 [select]:
github.com/cloudfoundry/gunk/workpool.worker(0xc208052600, 0xc20800bd40)
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/gunk/workpool/workpool.go:71 +0x151
created by github.com/cloudfoundry/gunk/workpool.New
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/gunk/workpool/workpool.go:44 +0x189

goroutine 9 [select]:
github.com/cloudfoundry/gunk/workpool.worker(0xc208052600, 0xc20800bd40)
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/gunk/workpool/workpool.go:71 +0x151
created by github.com/cloudfoundry/gunk/workpool.New
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/gunk/workpool/workpool.go:44 +0x189

goroutine 10 [select]:
github.com/cloudfoundry/gunk/workpool.worker(0xc208052600, 0xc20800bd40)
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/gunk/workpool/workpool.go:71 +0x151
created by github.com/cloudfoundry/gunk/workpool.New
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/gunk/workpool/workpool.go:44 +0x189

goroutine 11 [select]:
github.com/cloudfoundry/gunk/workpool.worker(0xc208052600, 0xc20800bd40)
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/gunk/workpool/workpool.go:71 +0x151
created by github.com/cloudfoundry/gunk/workpool.New
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/gunk/workpool/workpool.go:44 +0x189

goroutine 12 [select]:
github.com/cloudfoundry/gunk/workpool.worker(0xc208052600, 0xc20800bd40)
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/gunk/workpool/workpool.go:71 +0x151
created by github.com/cloudfoundry/gunk/workpool.New
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/gunk/workpool/workpool.go:44 +0x189

goroutine 13 [select]:
github.com/cloudfoundry/gunk/workpool.worker(0xc208052600, 0xc20800bd40)
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/gunk/workpool/workpool.go:71 +0x151
created by github.com/cloudfoundry/gunk/workpool.New
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/gunk/workpool/workpool.go:44 +0x189

goroutine 14 [select]:
github.com/cloudfoundry/gunk/workpool.worker(0xc208052600, 0xc20800bd40)
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/gunk/workpool/workpool.go:71 +0x151
created by github.com/cloudfoundry/gunk/workpool.New
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/gunk/workpool/workpool.go:44 +0x189

goroutine 15 [select]:
github.com/cloudfoundry/gunk/workpool.worker(0xc208052600, 0xc20800bd40)
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/gunk/workpool/workpool.go:71 +0x151
created by github.com/cloudfoundry/gunk/workpool.New
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/gunk/workpool/workpool.go:44 +0x189

goroutine 16 [select]:
github.com/cloudfoundry/gunk/workpool.worker(0xc208052600, 0xc20800bd40)
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/gunk/workpool/workpool.go:71 +0x151
created by github.com/cloudfoundry/gunk/workpool.New
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/gunk/workpool/workpool.go:44 +0x189

goroutine 17 [select]:

github.com/cloudfoundry/loggregatorlib/cfcomponent/registrars/collectorregistrar.(*CollectorRegistrar).Run(0xc20800c0c0)
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/loggregatorlib/cfcomponent/registrars/collectorregistrar/collector_registrar.go:41
+0x201
created by main.main

/var/vcap/data/compile/metron_agent/loggregator/src/metron/main.go:78 +0xf7d

goroutine 18 [IO wait]:
net.(*pollDesc).Wait(0xc2080103e0, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc2080103e0, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).accept(0xc208010380, 0x0, 0x7ff2322ffbe0, 0xc20802aba8)
/usr/local/go/src/net/fd_unix.go:419 +0x40b
net.(*TCPListener).AcceptTCP(0xc20802e060, 0x596f54, 0x0, 0x0)
/usr/local/go/src/net/tcpsock_posix.go:234 +0x4e
net/http.tcpKeepAliveListener.Accept(0xc20802e060, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:1976 +0x4c
net/http.(*Server).Serve(0xc2080527e0, 0x7ff232301bb8, 0xc20802e060, 0x0,
0x0)
/usr/local/go/src/net/http/server.go:1728 +0x92
net/http.(*Server).ListenAndServe(0xc2080527e0, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:1718 +0x154
net/http.ListenAndServe(0xc20802abb0, 0xf, 0x7ff232301b90, 0xc20800bc50,
0x0, 0x0)
/usr/local/go/src/net/http/server.go:1808 +0xba

github.com/cloudfoundry/loggregatorlib/cfcomponent.Component.StartMonitoringEndpoints(0xc20801ece0,
0xc20802aac0, 0x9, 0x7ff232301940, 0xa393a8, 0x834ff0, 0xb, 0x0, 0x0, 0x0,
...)
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/loggregatorlib/cfcomponent/component.go:76 +0x719
main.startMonitoringEndpoints(0xc20801ece0, 0xc20802aac0, 0x9,
0x7ff232301940, 0xa393a8, 0x834ff0, 0xb, 0x0, 0x0, 0x0, ...)

/var/vcap/data/compile/metron_agent/loggregator/src/metron/main.go:118 +0x3e
created by main.main

/var/vcap/data/compile/metron_agent/loggregator/src/metron/main.go:80 +0xfb8

goroutine 19 [IO wait]:
net.(*pollDesc).Wait(0xc208010450, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc208010450, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).readFrom(0xc2080103f0, 0xc2080c8000, 0xffff, 0xffff, 0x0,
0x0, 0x0, 0x7ff2322ffbe0, 0xc20802acf8)
/usr/local/go/src/net/fd_unix.go:269 +0x4a1
net.(*UDPConn).ReadFromUDP(0xc20802e068, 0xc2080c8000, 0xffff, 0xffff,
0x20, 0x0, 0x0, 0x0)
/usr/local/go/src/net/udpsock_posix.go:67 +0x124
net.(*UDPConn).ReadFrom(0xc20802e068, 0xc2080c8000, 0xffff, 0xffff,
0xffff, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/udpsock_posix.go:82 +0x12e

github.com/cloudfoundry/loggregatorlib/agentlistener.(*agentListener).Start(0xc208010fc0)
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/loggregatorlib/agentlistener/agent_listener.go:46
+0x3a5
created by main.main

/var/vcap/data/compile/metron_agent/loggregator/src/metron/main.go:83
+0x101a

goroutine 20 [chan receive]:
metron/legacy_message/legacy_unmarshaller.(*legacyUnmarshaller).Run(0xc20802a9d0,
0xc208052540, 0xc2080523c0)

/var/vcap/data/compile/metron_agent/loggregator/src/metron/legacy_message/legacy_unmarshaller/legacy_unmarshaller.go:30
+0x71
created by main.main

/var/vcap/data/compile/metron_agent/loggregator/src/metron/main.go:86
+0x1096

goroutine 21 [chan receive]:
metron/legacy_message/legacy_message_converter.(*legacyMessageConverter).Run(0xc20802e020,
0xc2080523c0, 0xc208052300)

/var/vcap/data/compile/metron_agent/loggregator/src/metron/legacy_message/legacy_message_converter/legacy_message_converter.go:28
+0x5c
created by main.main

/var/vcap/data/compile/metron_agent/loggregator/src/metron/main.go:87
+0x10dc

goroutine 22 [IO wait]:
net.(*pollDesc).Wait(0xc208010530, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc208010530, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).readFrom(0xc2080104d0, 0xc2080e0000, 0xffff, 0xffff, 0x0,
0x0, 0x0, 0x7ff2322ffbe0, 0xc20802add0)
/usr/local/go/src/net/fd_unix.go:269 +0x4a1
net.(*UDPConn).ReadFromUDP(0xc20802e098, 0xc2080e0000, 0xffff, 0xffff,
0x20, 0x0, 0x0, 0x0)
/usr/local/go/src/net/udpsock_posix.go:67 +0x124
net.(*UDPConn).ReadFrom(0xc20802e098, 0xc2080e0000, 0xffff, 0xffff,
0xffff, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/udpsock_posix.go:82 +0x12e
metron/eventlistener.(*eventListener).Start(0xc208060400)

/var/vcap/data/compile/metron_agent/loggregator/src/metron/eventlistener/event_listener.go:54
+0x3a0
created by main.main

/var/vcap/data/compile/metron_agent/loggregator/src/metron/main.go:89
+0x1108

goroutine 23 [chan receive]:

github.com/cloudfoundry/dropsonde/dropsonde_unmarshaller.(*dropsondeUnmarshaller).Run(0xc20802c080,
0xc2080525a0, 0xc208052300)
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/dropsonde/dropsonde_unmarshaller/dropsonde_unmarshaller.go:65
+0x71
created by main.main

/var/vcap/data/compile/metron_agent/loggregator/src/metron/main.go:91
+0x114e

goroutine 24 [chan receive]:
metron/message_aggregator.(*messageAggregator).Run(0xc208052180,
0xc208052300, 0xc208052420)

/var/vcap/data/compile/metron_agent/loggregator/src/metron/message_aggregator/message_aggregator.go:57
+0x5c
created by main.main

/var/vcap/data/compile/metron_agent/loggregator/src/metron/main.go:94
+0x11ca

goroutine 25 [chan receive]:
metron/varz_forwarder.(*VarzForwarder).Run(0xc20802c500, 0xc208052420,
0xc208052480)

/var/vcap/data/compile/metron_agent/loggregator/src/metron/varz_forwarder/varz_forwarder.go:31
+0x51
created by main.main

/var/vcap/data/compile/metron_agent/loggregator/src/metron/main.go:97
+0x1237

goroutine 26 [chan receive]:

github.com/cloudfoundry/dropsonde/dropsonde_marshaller.(*dropsondeMarshaller).Run(0xc20801e0a0,
0xc208052480, 0xc2080524e0)
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/dropsonde/dropsonde_marshaller/dropsonde_marshaller.go:59
+0x5f
created by main.main

/var/vcap/data/compile/metron_agent/loggregator/src/metron/main.go:100
+0x12a3

goroutine 27 [chan receive]:
main.signMessages(0xc20802b4b8, 0x8, 0xc2080524e0, 0xc2080526c0)

/var/vcap/data/compile/metron_agent/loggregator/src/metron/main.go:111 +0x7a
created by main.main

/var/vcap/data/compile/metron_agent/loggregator/src/metron/main.go:103
+0x130e

goroutine 33 [syscall, locked to thread]:
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:2232 +0x1

goroutine 31 [IO wait]:
net.(*pollDesc).Wait(0xc208010a00, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc208010a00, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).Read(0xc2080109a0, 0xc208059000, 0x1000, 0x1000, 0x0,
0x7ff2322ffbe0, 0xc2080daa00)
/usr/local/go/src/net/fd_unix.go:242 +0x40f
net.(*conn).Read(0xc20802e150, 0xc208059000, 0x1000, 0x1000, 0x0, 0x0,
0x0)
/usr/local/go/src/net/net.go:121 +0xdc
net/http.noteEOFReader.Read(0x7ff232301d08, 0xc20802e150, 0xc20805af78,
0xc208059000, 0x1000, 0x1000, 0x75b940, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:1270 +0x6e
net/http.(*noteEOFReader).Read(0xc20801f2e0, 0xc208059000, 0x1000,
0x1000, 0xc208012000, 0x0, 0x0)
<autogenerated>:125 +0xd4
bufio.(*Reader).fill(0xc208053200)
/usr/local/go/src/bufio/bufio.go:97 +0x1ce
bufio.(*Reader).Peek(0xc208053200, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:132 +0xf0
net/http.(*persistConn).readLoop(0xc20805af20)
/usr/local/go/src/net/http/transport.go:842 +0xa4
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:660 +0xc9f

goroutine 32 [select]:
net/http.(*persistConn).writeLoop(0xc20805af20)
/usr/local/go/src/net/http/transport.go:945 +0x41d
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:661 +0xcbc

goroutine 36 [IO wait]:
net.(*pollDesc).Wait(0xc20810a3e0, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc20810a3e0, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).Read(0xc20810a380, 0xc2080fb000, 0x1000, 0x1000, 0x0,
0x7ff2322ffbe0, 0xc2080dad28)
/usr/local/go/src/net/fd_unix.go:242 +0x40f
net.(*conn).Read(0xc20802e318, 0xc2080fb000, 0x1000, 0x1000, 0x0, 0x0,
0x0)
/usr/local/go/src/net/net.go:121 +0xdc
net/http.noteEOFReader.Read(0x7ff232301d08, 0xc20802e318, 0xc20805ad68,
0xc2080fb000, 0x1000, 0x1000, 0x75b940, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:1270 +0x6e
net/http.(*noteEOFReader).Read(0xc2080dd000, 0xc2080fb000, 0x1000,
0x1000, 0xc208012000, 0x0, 0x0)
<autogenerated>:125 +0xd4
bufio.(*Reader).fill(0xc2081086c0)
/usr/local/go/src/bufio/bufio.go:97 +0x1ce
bufio.(*Reader).Peek(0xc2081086c0, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:132 +0xf0
net/http.(*persistConn).readLoop(0xc20805ad10)
/usr/local/go/src/net/http/transport.go:842 +0xa4
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:660 +0xc9f

goroutine 37 [select]:
net/http.(*persistConn).writeLoop(0xc20805ad10)
/usr/local/go/src/net/http/transport.go:945 +0x41d
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:661 +0xcbc
----------------------

I tried to restart it using monit restart still it is failing .
any body got idea on this error


regards
Bharath

--
Jim Campbell | Product Manager | Cloud Foundry | Pivotal.io | 303.618.0963