Re: Failure of metron agent in cloudfoundry


Jim CF Campbell
 

Hi Bharath,

This looks like it might be older code. What version of Cloud Foundry are
you running?

Thanks,
Jim

On Mon, Apr 4, 2016 at 11:09 PM, Bharath Posa <bharathp(a)vedams.com> wrote:

Hi all,

we are recently running cloudfoundry in openstack. while recent saw the
metron agent in all the jobs failed . The metron.stderr.log provided the
below log

-------------
panic: 402: Standby Internal Error () [0]

goroutine 28 [running]:

github.com/cloudfoundry/loggregatorlib/servicediscovery.(*serverAddressList).Run(0xc208052120,
0x12a05f200)
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/loggregatorlib/servicediscovery/servicediscovery.go:57
+0x3c3
created by main.main

/var/vcap/data/compile/metron_agent/loggregator/src/metron/main.go:105
+0x1351

goroutine 1 [chan receive]:
main.forwardMessagesToDoppler(0xc20802c040, 0xc2080526c0, 0xc20801ece0)

/var/vcap/data/compile/metron_agent/loggregator/src/metron/main.go:187 +0x7a
main.main()

/var/vcap/data/compile/metron_agent/loggregator/src/metron/main.go:107
+0x137e

goroutine 5 [syscall]:
os/signal.loop()
/usr/local/go/src/os/signal/signal_unix.go:21 +0x1f
created by os/signal.init·1
/usr/local/go/src/os/signal/signal_unix.go:27 +0x35

goroutine 7 [select]:
github.com/cloudfoundry/gunk/workpool.worker(0xc208052600, 0xc20800bd40)
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/gunk/workpool/workpool.go:71 +0x151
created by github.com/cloudfoundry/gunk/workpool.New
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/gunk/workpool/workpool.go:44 +0x189

goroutine 8 [select]:
github.com/cloudfoundry/gunk/workpool.worker(0xc208052600, 0xc20800bd40)
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/gunk/workpool/workpool.go:71 +0x151
created by github.com/cloudfoundry/gunk/workpool.New
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/gunk/workpool/workpool.go:44 +0x189

goroutine 9 [select]:
github.com/cloudfoundry/gunk/workpool.worker(0xc208052600, 0xc20800bd40)
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/gunk/workpool/workpool.go:71 +0x151
created by github.com/cloudfoundry/gunk/workpool.New
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/gunk/workpool/workpool.go:44 +0x189

goroutine 10 [select]:
github.com/cloudfoundry/gunk/workpool.worker(0xc208052600, 0xc20800bd40)
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/gunk/workpool/workpool.go:71 +0x151
created by github.com/cloudfoundry/gunk/workpool.New
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/gunk/workpool/workpool.go:44 +0x189

goroutine 11 [select]:
github.com/cloudfoundry/gunk/workpool.worker(0xc208052600, 0xc20800bd40)
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/gunk/workpool/workpool.go:71 +0x151
created by github.com/cloudfoundry/gunk/workpool.New
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/gunk/workpool/workpool.go:44 +0x189

goroutine 12 [select]:
github.com/cloudfoundry/gunk/workpool.worker(0xc208052600, 0xc20800bd40)
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/gunk/workpool/workpool.go:71 +0x151
created by github.com/cloudfoundry/gunk/workpool.New
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/gunk/workpool/workpool.go:44 +0x189

goroutine 13 [select]:
github.com/cloudfoundry/gunk/workpool.worker(0xc208052600, 0xc20800bd40)
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/gunk/workpool/workpool.go:71 +0x151
created by github.com/cloudfoundry/gunk/workpool.New
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/gunk/workpool/workpool.go:44 +0x189

goroutine 14 [select]:
github.com/cloudfoundry/gunk/workpool.worker(0xc208052600, 0xc20800bd40)
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/gunk/workpool/workpool.go:71 +0x151
created by github.com/cloudfoundry/gunk/workpool.New
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/gunk/workpool/workpool.go:44 +0x189

goroutine 15 [select]:
github.com/cloudfoundry/gunk/workpool.worker(0xc208052600, 0xc20800bd40)
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/gunk/workpool/workpool.go:71 +0x151
created by github.com/cloudfoundry/gunk/workpool.New
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/gunk/workpool/workpool.go:44 +0x189

goroutine 16 [select]:
github.com/cloudfoundry/gunk/workpool.worker(0xc208052600, 0xc20800bd40)
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/gunk/workpool/workpool.go:71 +0x151
created by github.com/cloudfoundry/gunk/workpool.New
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/gunk/workpool/workpool.go:44 +0x189

goroutine 17 [select]:

github.com/cloudfoundry/loggregatorlib/cfcomponent/registrars/collectorregistrar.(*CollectorRegistrar).Run(0xc20800c0c0)
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/loggregatorlib/cfcomponent/registrars/collectorregistrar/collector_registrar.go:41
+0x201
created by main.main

/var/vcap/data/compile/metron_agent/loggregator/src/metron/main.go:78 +0xf7d

goroutine 18 [IO wait]:
net.(*pollDesc).Wait(0xc2080103e0, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc2080103e0, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).accept(0xc208010380, 0x0, 0x7ff2322ffbe0, 0xc20802aba8)
/usr/local/go/src/net/fd_unix.go:419 +0x40b
net.(*TCPListener).AcceptTCP(0xc20802e060, 0x596f54, 0x0, 0x0)
/usr/local/go/src/net/tcpsock_posix.go:234 +0x4e
net/http.tcpKeepAliveListener.Accept(0xc20802e060, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:1976 +0x4c
net/http.(*Server).Serve(0xc2080527e0, 0x7ff232301bb8, 0xc20802e060, 0x0,
0x0)
/usr/local/go/src/net/http/server.go:1728 +0x92
net/http.(*Server).ListenAndServe(0xc2080527e0, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:1718 +0x154
net/http.ListenAndServe(0xc20802abb0, 0xf, 0x7ff232301b90, 0xc20800bc50,
0x0, 0x0)
/usr/local/go/src/net/http/server.go:1808 +0xba

github.com/cloudfoundry/loggregatorlib/cfcomponent.Component.StartMonitoringEndpoints(0xc20801ece0,
0xc20802aac0, 0x9, 0x7ff232301940, 0xa393a8, 0x834ff0, 0xb, 0x0, 0x0, 0x0,
...)
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/loggregatorlib/cfcomponent/component.go:76 +0x719
main.startMonitoringEndpoints(0xc20801ece0, 0xc20802aac0, 0x9,
0x7ff232301940, 0xa393a8, 0x834ff0, 0xb, 0x0, 0x0, 0x0, ...)

/var/vcap/data/compile/metron_agent/loggregator/src/metron/main.go:118 +0x3e
created by main.main

/var/vcap/data/compile/metron_agent/loggregator/src/metron/main.go:80 +0xfb8

goroutine 19 [IO wait]:
net.(*pollDesc).Wait(0xc208010450, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc208010450, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).readFrom(0xc2080103f0, 0xc2080c8000, 0xffff, 0xffff, 0x0,
0x0, 0x0, 0x7ff2322ffbe0, 0xc20802acf8)
/usr/local/go/src/net/fd_unix.go:269 +0x4a1
net.(*UDPConn).ReadFromUDP(0xc20802e068, 0xc2080c8000, 0xffff, 0xffff,
0x20, 0x0, 0x0, 0x0)
/usr/local/go/src/net/udpsock_posix.go:67 +0x124
net.(*UDPConn).ReadFrom(0xc20802e068, 0xc2080c8000, 0xffff, 0xffff,
0xffff, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/udpsock_posix.go:82 +0x12e

github.com/cloudfoundry/loggregatorlib/agentlistener.(*agentListener).Start(0xc208010fc0)
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/loggregatorlib/agentlistener/agent_listener.go:46
+0x3a5
created by main.main

/var/vcap/data/compile/metron_agent/loggregator/src/metron/main.go:83
+0x101a

goroutine 20 [chan receive]:
metron/legacy_message/legacy_unmarshaller.(*legacyUnmarshaller).Run(0xc20802a9d0,
0xc208052540, 0xc2080523c0)

/var/vcap/data/compile/metron_agent/loggregator/src/metron/legacy_message/legacy_unmarshaller/legacy_unmarshaller.go:30
+0x71
created by main.main

/var/vcap/data/compile/metron_agent/loggregator/src/metron/main.go:86
+0x1096

goroutine 21 [chan receive]:
metron/legacy_message/legacy_message_converter.(*legacyMessageConverter).Run(0xc20802e020,
0xc2080523c0, 0xc208052300)

/var/vcap/data/compile/metron_agent/loggregator/src/metron/legacy_message/legacy_message_converter/legacy_message_converter.go:28
+0x5c
created by main.main

/var/vcap/data/compile/metron_agent/loggregator/src/metron/main.go:87
+0x10dc

goroutine 22 [IO wait]:
net.(*pollDesc).Wait(0xc208010530, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc208010530, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).readFrom(0xc2080104d0, 0xc2080e0000, 0xffff, 0xffff, 0x0,
0x0, 0x0, 0x7ff2322ffbe0, 0xc20802add0)
/usr/local/go/src/net/fd_unix.go:269 +0x4a1
net.(*UDPConn).ReadFromUDP(0xc20802e098, 0xc2080e0000, 0xffff, 0xffff,
0x20, 0x0, 0x0, 0x0)
/usr/local/go/src/net/udpsock_posix.go:67 +0x124
net.(*UDPConn).ReadFrom(0xc20802e098, 0xc2080e0000, 0xffff, 0xffff,
0xffff, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/udpsock_posix.go:82 +0x12e
metron/eventlistener.(*eventListener).Start(0xc208060400)

/var/vcap/data/compile/metron_agent/loggregator/src/metron/eventlistener/event_listener.go:54
+0x3a0
created by main.main

/var/vcap/data/compile/metron_agent/loggregator/src/metron/main.go:89
+0x1108

goroutine 23 [chan receive]:

github.com/cloudfoundry/dropsonde/dropsonde_unmarshaller.(*dropsondeUnmarshaller).Run(0xc20802c080,
0xc2080525a0, 0xc208052300)
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/dropsonde/dropsonde_unmarshaller/dropsonde_unmarshaller.go:65
+0x71
created by main.main

/var/vcap/data/compile/metron_agent/loggregator/src/metron/main.go:91
+0x114e

goroutine 24 [chan receive]:
metron/message_aggregator.(*messageAggregator).Run(0xc208052180,
0xc208052300, 0xc208052420)

/var/vcap/data/compile/metron_agent/loggregator/src/metron/message_aggregator/message_aggregator.go:57
+0x5c
created by main.main

/var/vcap/data/compile/metron_agent/loggregator/src/metron/main.go:94
+0x11ca

goroutine 25 [chan receive]:
metron/varz_forwarder.(*VarzForwarder).Run(0xc20802c500, 0xc208052420,
0xc208052480)

/var/vcap/data/compile/metron_agent/loggregator/src/metron/varz_forwarder/varz_forwarder.go:31
+0x51
created by main.main

/var/vcap/data/compile/metron_agent/loggregator/src/metron/main.go:97
+0x1237

goroutine 26 [chan receive]:

github.com/cloudfoundry/dropsonde/dropsonde_marshaller.(*dropsondeMarshaller).Run(0xc20801e0a0,
0xc208052480, 0xc2080524e0)
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/dropsonde/dropsonde_marshaller/dropsonde_marshaller.go:59
+0x5f
created by main.main

/var/vcap/data/compile/metron_agent/loggregator/src/metron/main.go:100
+0x12a3

goroutine 27 [chan receive]:
main.signMessages(0xc20802b4b8, 0x8, 0xc2080524e0, 0xc2080526c0)

/var/vcap/data/compile/metron_agent/loggregator/src/metron/main.go:111 +0x7a
created by main.main

/var/vcap/data/compile/metron_agent/loggregator/src/metron/main.go:103
+0x130e

goroutine 33 [syscall, locked to thread]:
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:2232 +0x1

goroutine 31 [IO wait]:
net.(*pollDesc).Wait(0xc208010a00, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc208010a00, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).Read(0xc2080109a0, 0xc208059000, 0x1000, 0x1000, 0x0,
0x7ff2322ffbe0, 0xc2080daa00)
/usr/local/go/src/net/fd_unix.go:242 +0x40f
net.(*conn).Read(0xc20802e150, 0xc208059000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:121 +0xdc
net/http.noteEOFReader.Read(0x7ff232301d08, 0xc20802e150, 0xc20805af78,
0xc208059000, 0x1000, 0x1000, 0x75b940, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:1270 +0x6e
net/http.(*noteEOFReader).Read(0xc20801f2e0, 0xc208059000, 0x1000, 0x1000,
0xc208012000, 0x0, 0x0)
<autogenerated>:125 +0xd4
bufio.(*Reader).fill(0xc208053200)
/usr/local/go/src/bufio/bufio.go:97 +0x1ce
bufio.(*Reader).Peek(0xc208053200, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:132 +0xf0
net/http.(*persistConn).readLoop(0xc20805af20)
/usr/local/go/src/net/http/transport.go:842 +0xa4
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:660 +0xc9f

goroutine 32 [select]:
net/http.(*persistConn).writeLoop(0xc20805af20)
/usr/local/go/src/net/http/transport.go:945 +0x41d
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:661 +0xcbc

goroutine 36 [IO wait]:
net.(*pollDesc).Wait(0xc20810a3e0, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc20810a3e0, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).Read(0xc20810a380, 0xc2080fb000, 0x1000, 0x1000, 0x0,
0x7ff2322ffbe0, 0xc2080dad28)
/usr/local/go/src/net/fd_unix.go:242 +0x40f
net.(*conn).Read(0xc20802e318, 0xc2080fb000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:121 +0xdc
net/http.noteEOFReader.Read(0x7ff232301d08, 0xc20802e318, 0xc20805ad68,
0xc2080fb000, 0x1000, 0x1000, 0x75b940, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:1270 +0x6e
net/http.(*noteEOFReader).Read(0xc2080dd000, 0xc2080fb000, 0x1000, 0x1000,
0xc208012000, 0x0, 0x0)
<autogenerated>:125 +0xd4
bufio.(*Reader).fill(0xc2081086c0)
/usr/local/go/src/bufio/bufio.go:97 +0x1ce
bufio.(*Reader).Peek(0xc2081086c0, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:132 +0xf0
net/http.(*persistConn).readLoop(0xc20805ad10)
/usr/local/go/src/net/http/transport.go:842 +0xa4
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:660 +0xc9f

goroutine 37 [select]:
net/http.(*persistConn).writeLoop(0xc20805ad10)
/usr/local/go/src/net/http/transport.go:945 +0x41d
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:661 +0xcbc
----------------------

I tried to restart it using monit restart still it is failing .
any body got idea on this error


regards
Bharath
--
Jim Campbell | Product Manager | Cloud Foundry | Pivotal.io | 303.618.0963

Join cf-dev@lists.cloudfoundry.org to automatically receive all group messages.