Date   

Re: cf v233 api_z1/api_z2 failing

Ranga Rajagopalan
 

Thanks, Omar.


Re: Spotting performance regressions

Mark St.Godard
 

Hi Michal

(sorry for late response.. I caught you on Slack as well, but responding to
dev list)

Yes we have recently added a stage in our continuous delivery pipeline to
catch performance regressions.
This stage right now is not using routing-perf-release, it is currently
using a command-line (golang-based) http load generator and we are
currently measuring and emitting requests / sec per run. The stage also
will fail if the results of the run are below a specific threshold.

We also have a datadog dashboard with all the routing related metrics we
are getting from the firehose.

Important thing to note is that this is just the beginning of our
performance analysis related work on gorouter,
Shannon (Routing PM) is heading up our next initiative which focuses on
gorouter performance and metrics.
Part of this work will likely include enhancing our performance automation
test suite and more info on metrics.

I'll let Shannon chime in if he had additional info to add.

Cheers


On Wed, Apr 6, 2016 at 5:47 AM, Michal Tekel <
michal.tekel(a)digital.cabinet-office.gov.uk> wrote:

Thanks!

That helps a lot. Looks like (from slack chat and pivotal tracker) that
the routingteam already has a job set-up which runs perf. tests each hour
and posts results to datadog, which is something very similar to what we
were looking to do. I got in touch with them on slack...


Thanks again...

On 5 April 2016 at 17:39, Guillaume Berche <bercheg(a)gmail.com> wrote:

Hi Michal,

I understand the routing team is working on automated performance tests
that aims at detecting performance regressions on the gorouter. We have not
yet set up execution of those at Orange yet. More in the pointers below.
The routing team was considering to open sourcing the routing ci repo, I'm
not sure of the current status now.

https://github.com/cloudfoundry-incubator/routing-perf-release
https://cf-routing.ci.cf-app.com/pipelines/routing/jobs/run-cf-load-test
https://cloudfoundry.slack.com/archives/routing/p1455232667000627
https://www.pivotaltracker.com/n/projects/1358110 with search:
label:"gorouter performance" includedone:true includes current perf
regression and effort for automated perf testing and determining max
request rate and concurrent requests

Hope this helps,

Guillaume.



Guillaume.

On Mon, Apr 4, 2016 at 4:08 PM, Michal Tekel <
michal.tekel(a)digital.cabinet-office.gov.uk> wrote:

Hi,

I wanted to check if you know about a good way to spot performance
regressions on CF and if anyone is doing it (at least partially) already.
We are mainly interested in discovering user-facing degradation. That is,
app operations (deploy, scale, delete, manage routes) and routing layer
operations (latency per request, total throughput, SSL termination
capacity) - esp. in connection with (heavier) routing services usage.

We did a brief investigation about the best way to automate this kind of
regression testing, but didn't find any complete solution. We had a look at
[CF PATs](1) tool and made a [container](2) it can be run from. But that
only gives us metrics on app operations and no way of spotting regressions
or automating the tests.

We were wondering if perhaps someone already does any kind of
performance measurements as a part of their CF build/deploy
pipelines/process and what their experience with tracking the results is...

[1] https://github.com/cloudfoundry-incubator/pat
[2] https://hub.docker.com/r/keymon/governmentpaas-cf-pats/


Re: Spotting performance regressions

Michal Tekel
 

Thanks!

That helps a lot. Looks like (from slack chat and pivotal tracker) that the
routingteam already has a job set-up which runs perf. tests each hour and
posts results to datadog, which is something very similar to what we were
looking to do. I got in touch with them on slack...


Thanks again...

On 5 April 2016 at 17:39, Guillaume Berche <bercheg(a)gmail.com> wrote:

Hi Michal,

I understand the routing team is working on automated performance tests
that aims at detecting performance regressions on the gorouter. We have not
yet set up execution of those at Orange yet. More in the pointers below.
The routing team was considering to open sourcing the routing ci repo, I'm
not sure of the current status now.

https://github.com/cloudfoundry-incubator/routing-perf-release
https://cf-routing.ci.cf-app.com/pipelines/routing/jobs/run-cf-load-test
https://cloudfoundry.slack.com/archives/routing/p1455232667000627
https://www.pivotaltracker.com/n/projects/1358110 with search:
label:"gorouter performance" includedone:true includes current perf
regression and effort for automated perf testing and determining max
request rate and concurrent requests

Hope this helps,

Guillaume.



Guillaume.

On Mon, Apr 4, 2016 at 4:08 PM, Michal Tekel <
michal.tekel(a)digital.cabinet-office.gov.uk> wrote:

Hi,

I wanted to check if you know about a good way to spot performance
regressions on CF and if anyone is doing it (at least partially) already.
We are mainly interested in discovering user-facing degradation. That is,
app operations (deploy, scale, delete, manage routes) and routing layer
operations (latency per request, total throughput, SSL termination
capacity) - esp. in connection with (heavier) routing services usage.

We did a brief investigation about the best way to automate this kind of
regression testing, but didn't find any complete solution. We had a look at
[CF PATs](1) tool and made a [container](2) it can be run from. But that
only gives us metrics on app operations and no way of spotting regressions
or automating the tests.

We were wondering if perhaps someone already does any kind of performance
measurements as a part of their CF build/deploy pipelines/process and what
their experience with tracking the results is...

[1] https://github.com/cloudfoundry-incubator/pat
[2] https://hub.docker.com/r/keymon/governmentpaas-cf-pats/


Re: cf v233 api_z1/api_z2 failing

Elazhary, Omar <omar.elazhary@...>
 

Hi Ranga,

Apparently this is related to the following issue: https://github.com/cloudfoundry/cloud_controller_ng/issues/570

I hope this gets fixed in the new release.

Regards,
Omar

On 4/6/16, 5:57 AM, "Ranga Rajagopalan" <ranga.rajagopalan(a)gmail.com> wrote:

I've tried with version 3215 stemcells - still same issue.

| bosh-vsphere-esxi-ubuntu-trusty-go_agent | ubuntu-trusty | 3215*


Re: cf v233 api_z1/api_z2 failing

Ranga Rajagopalan
 

I've tried with version 3215 stemcells - still same issue.

| bosh-vsphere-esxi-ubuntu-trusty-go_agent | ubuntu-trusty | 3215*


Re: Persistent Volumes on Cloud Foundry

Dr Nic Williams <drnicwilliams@...>
 

Is a goal of this work to run pure data services like PostgreSQL or Redis inside "application instances"?

On Tue, Apr 5, 2016 at 5:01 PM -0700, "Ted Young" <tyoung(a)pivotal.io> wrote:










This proposal describes the changes necessary to the Service Broker API to utilize the new Volume Management features of the Diego runtime. It contains examples of how services can provide a variety of persistent data access to CF applications, where each service maintains control of the volume lifecycle. This is to allow some services that provide blank storage space to applications, and other services that provide access to complex or externally managed data (such as IBM's Watson).
http://bit.ly/cf-volume-proposal

We are moving fast on delivering a beta of this feature, so please have a look and give feedback now if this is of interest to you. More detail will be added to the proposal as necessary.
Cheers,
Ted YoungSenior Engineer / Product ManagerPivotal Cloud Foundry


Cf login doesnt work

Nithiyasri Gnanasekaran -X (ngnanase - TECH MAHINDRA LIM@Cisco) <ngnanase at cisco.com...>
 

Hi

I am using cf-231 . After deploying, I can set cf endpoint. But I could not login
While logging, it gives me the following
404 Not Found: Requested route ('uaa.cisco.com<http://uaa.cisco.com>') does not exist.
Server error, status code: 404, error code: , message:
Related properties : uaa.require_htttps:false in yml

Below are the trace:
root(a)dev-inception-vm1:/opt/cisco/vms-installer/tenant-vikramdevtes t1/cf-deploy# cf login
API endpoint: https://api.vikramdevtest1.io

Email> admin

Password>
Authenticating...
Server error, status code: 404, error code: , message:

Password> root(a)dev-inception-vm1:/opt/cisco/vms-installer/tenant-vi t1/cf-deploy# ls
cf-231-final-V.yml cf-template-231.yml service.yml
cf-settings.rb cf-vikramdevtest1.yml
root(a)dev-inception-vm1:/opt/cisco/vms-installer/tenant-vikramdevtes t1/cf-deploy# vimdiff cf-template-231.yml cf-231-final-V.yml
2 files to edit
root(a)dev-inception-vm1:/opt/cisco/vms-installer/tenant-vikramdevtes t1/cf-deploy# CF_TRACE_true
CF_TRACE_true: command not found
root(a)dev-inception-vm1:/opt/cisco/vms-installer/tenant-vikramdevtes t1/cf-deploy# CF_TRACE=true cf login
API endpoint: https://api.vikramdevtest1.io

REQUEST: [2016-04-05T17:38:18Z]
GET /v2/info HTTP/1.1
Host: api.vikramdevtest1.io<http://api.vikramdevtest1.io>
Accept: application/json
Content-Type: application/json
User-Agent: go-cli 6.12.2-24abed3 / linux



RESPONSE: [2016-04-05T17:38:18Z]
HTTP/1.1 200 OK
Content-Length: 586
Content-Type: application/json;charset=utf-8
Date: Tue, 05 Apr 2016 17:38:18 GMT
Server: nginx
X-Content-Type-Options: nosniff
X-Vcap-Request-Id: 54184ed0-310b-4a2f-5d5f-a1c21a397d49
X-Vcap-Request-Id: 876ca517-01fd-4f73-7a85-955955f3de41::86dcd763-5 f8e-42b3-b657-7af57ec9ea21

{"name":"","build":"","support":"http://support.cloudfoundry.com"," version":0,"description":"","authorization_endpoint":"http://uaa.vi kramdevtest1.io<http://kramdevtest1.io>","token_endpoint":"http://uaa.vikramdevtest1.io","m in_cli_version":null,"min_recommended_cli_version":null,"api_versio n":"2.51.0","app_ssh_endpoint":"ssh.vikramdevtest1.io:2222<http://ssh.vikramdevtest1.io:2222>","app_ss h_host_key_fingerprint":null,"app_ssh_oauth_client":"ssh-proxy","ro uting_endpoint":"https://api.vikramdevtest1.io/routing","logging_en dpoint":"wss://loggregator.vikramdevtest1.io:4443<http://loggregator.vikramdevtest1.io:4443>","doppler_logging _endpoint":"wss://doppler.vikramdevtest1.io:4443<http://doppler.vikramdevtest1.io:4443>"}

REQUEST: [2016-04-05T17:38:18Z]
GET /login HTTP/1.1
Host: uaa.vikramdevtest1.io<http://uaa.vikramdevtest1.io>
Accept: application/json
Content-Type: application/json
User-Agent: go-cli 6.12.2-24abed3 / linux



RESPONSE: [2016-04-05T17:38:18Z]
HTTP/1.1 200 OK
Content-Length: 447
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Cache-Control: no-store
Content-Language: en-US
Content-Type: application/json;charset=UTF-8
Date: Tue, 05 Apr 2016 17:38:18 GMT
Expires: 0
Pragma: no-cache
Server: Apache-Coyote/1.1
Strict-Transport-Security: max-age=31536000 ; includeSubDomains
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-Vcap-Request-Id: d35da14d-3367-4032-6eef-d2050839147f
X-Xss-Protection: 1; mode=block

{"app":{"version":"3.1.0"},"links":{"uaa":"http://uaa.vikramdevtest 1.io<http://1.io>","passwd":"https://console.vikramdevtest1.io/password_resets/n ew","login":"http://login.vikramdevtest1.io","register":"https://co nsole.vikramdevtest1.io/register<http://nsole.vikramdevtest1.io/register>"},"zone_name":"uaa","entityID":"lo gin.vikramdevtest1.io<http://gin.vikramdevtest1.io>","commit_id":"9b5c13d","idpDefinitions":{},"p rompts":{"username":["text","Email"],"password":["password","Passwo rd"]},"timestamp":"2016-02-05T14:27:13+0000"}

Email> admin

Password>
Authenticating...

REQUEST: [2016-04-05T17:38:29Z]
POST /oauth/token HTTP/1.1
Host: uaa.vikramdevtest1.io<http://uaa.vikramdevtest1.io>
Accept: application/json
Authorization: [PRIVATE DATA HIDDEN]
Content-Type: application/x-www-form-urlencoded
User-Agent: go-cli 6.12.2-24abed3 / linux

grant_type=password&password=[PRIVATE DATA HIDDEN]&scope=&username=admin

RESPONSE: [2016-04-05T17:38:29Z]
HTTP/1.1 404 Not Found
Content-Length: 65
Content-Type: text/plain; charset=utf-8
Date: Tue, 05 Apr 2016 17:38:29 GMT
X-Cf-Routererror: unknown_route
X-Content-Type-Options: nosniff
X-Vcap-Request-Id: dab37d6c-3fea-428c-516a-ec7906ff6d16

404 Not Found: Requested route ('uaa.cisco.com<http://uaa.cisco.com>') does not exist.

Server error, status code: 404, error code: , message:

Password> root(a)dev-inception-vm1:/opt/cisco/vms-installer/tenant-vikramdevtest1/cf-deploy#

Regards
Nithiyasri


Persistent Volumes on Cloud Foundry

Ted Young
 

This proposal describes the changes necessary to the Service Broker API to
utilize the new Volume Management features of the Diego runtime. It
contains examples of how services can provide a variety of persistent data
access to CF applications, where each service maintains control of the
volume lifecycle. This is to allow some services that provide blank storage
space to applications, and other services that provide access to complex or
externally managed data (such as IBM's Watson).

http://bit.ly/cf-volume-proposal

We are moving fast on delivering a beta of this feature, so please have a
look and give feedback now if this is of interest to you. More detail will
be added to the proposal as necessary.

Cheers,

Ted Young
Senior Engineer / Product Manager
Pivotal Cloud Foundry


Re: cf v233 api_z1/api_z2 failing

Ranga Rajagopalan
 

Logs from /var/vcap/data/sys/log/nginx_ctl.err.log:

[2016-04-05 21:40:34+0000] ------------ STARTING nginx_ctl at Tue Apr 5 21:40:34 UTC 2016 --------------
[2016-04-05 21:41:30+0000] /var/vcap/packages/capi_utils/utils.sh: line 70: #{timeout}: syntax error: operand expected (error token is #{timeout})
[2016-04-05 21:41:39+0000] ------------ STARTING nginx_ctl at Tue Apr 5 21:41:39 UTC 2016 --------------
[2016-04-05 21:41:52+0000] /var/vcap/packages/capi_utils/utils.sh: line 70: #{timeout}: syntax error: operand expected (error token is #{timeout})


cf v233 api_z1/api_z2 failing

Ranga Rajagopalan
 

Hi,
I just deployed v233 in a vSphere environment. Both api_z1 and api_z2 are failing. Any help would be appreciated.

Stemcell version:

| bosh-vsphere-esxi-ubuntu-trusty-go_agent | bosh-vsphere-esxi-ubuntu-trusty-go_agent | 2950* |

bosh vm status:

...
| api_z1/0 (03390add-275d-4927-8d5a-8ad3f60729fe) | starting | n/a | large_z1 | 10.90.27.129 |
| api_z2/0 (fc8fb93f-f5be-4389-bdb5-398427f23cef) | failing | n/a | large_z2 | 10.90.28.137 |
...

Looks like cloud_controller_ng, cloud_controller_worker_local_1, cloud_controller_worker_local_2, nginx_cc are failing/initializing.

monit status from api_z1.

Process 'cloud_controller_ng'
status Connection failed
monitoring status monitored
...
Process 'cloud_controller_worker_local_1'
status initializing
monitoring status initializing
data collected Tue Apr 5 23:08:30 2016

Process 'cloud_controller_worker_local_2'
status initializing
monitoring status initializing
data collected Tue Apr 5 23:08:30 2016

Process 'nginx_cc'
status initializing
monitoring status initializing
data collected Tue Apr 5 23:08:30 2016

Logs look similar from both nodes z1/z2.

Logs from /var/vcap/data/sys/log/cloud_controller_ng_ctl.err.log:

[2016-04-05 21:36:18+0000] ------------ STARTING cloud_controller_ng_ctl at Tue Apr 5 21:36:18 UTC 2016 --------------
[2016-04-05 21:36:30+0000] /var/vcap/data/packages/cloud_controller_ng/0f8b96021b770958c5e092c2c985a4aeb7977f80.1-489b907bc3437165bc52a390fdc42d37d5c13383/cloud_controller_ng/app/controllers/v3/mixins/app_subresource.rb:5: warning: already initialized constant AppSubresource::ROLES_FOR_READING
[2016-04-05 21:36:30+0000] /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/app/controllers/v3/mixins/app_subresource.rb:5: warning: previous definition of ROLES_FOR_READING was here
[2016-04-05 21:36:31+0000] /var/vcap/data/packages/cloud_controller_ng/0f8b96021b770958c5e092c2c985a4aeb7977f80.1-489b907bc3437165bc52a390fdc42d37d5c13383/cloud_controller_ng/app/controllers/base/base_controller.rb:11: warning: already initialized constant VCAP::CloudController::RestController::BaseController::V2_ROUTE_PREFIX
[2016-04-05 21:36:31+0000] /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/app/controllers/base/base_controller.rb:11: warning: previous definition of V2_ROUTE_PREFIX was here
[2016-04-05 21:36:31+0000] /var/vcap/data/packages/cloud_controller_ng/0f8b96021b770958c5e092c2c985a4aeb7977f80.1-489b907bc3437165bc52a390fdc42d37d5c13383/cloud_controller_ng/app/controllers/base/front_controller.rb:7: warning: already initialized constant VCAP::CloudController::Errors
[2016-04-05 21:36:31+0000] /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/app/controllers/base/front_controller.rb:7: warning: previous definition of Errors was here
[2016-04-05 21:36:31+0000] /var/vcap/data/packages/cloud_controller_ng/0f8b96021b770958c5e092c2c985a4aeb7977f80.1-489b907bc3437165bc52a390fdc42d37d5c13383/cloud_controller_ng/app/controllers/base/model_controller.rb:8: warning: already initialized constant VCAP::CloudController::RestController::ModelController::CENSORED_MESSAGE
[2016-04-05 21:36:31+0000] /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/app/controllers/base/model_controller.rb:8: warning: previous definition of CENSORED_MESSAGE was here
[2016-04-05 21:37:31+0000] ------------ STARTING cloud_controller_ng_ctl at Tue Apr 5 21:37:31 UTC 2016 --------------
[2016-04-05 21:37:35+0000] /var/vcap/data/packages/cloud_controller_ng/0f8b96021b770958c5e092c2c985a4aeb7977f80.1-489b907bc3437165bc52a390fdc42d37d5c13383/cloud_controller_ng/app/controllers/v3/mixins/app_subresource.rb:5: warning: already initialized constant AppSubresource::ROLES_FOR_READING
[2016-04-05 21:37:35+0000] /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/app/controllers/v3/mixins/app_subresource.rb:5: warning: previous definition of ROLES_FOR_READING was here
[2016-04-05 21:37:35+0000] /var/vcap/data/packages/cloud_controller_ng/0f8b96021b770958c5e092c2c985a4aeb7977f80.1-489b907bc3437165bc52a390fdc42d37d5c13383/cloud_controller_ng/app/controllers/base/base_controller.rb:11: warning: already initialized constant VCAP::CloudController::RestController::BaseController::V2_ROUTE_PREFIX
[2016-04-05 21:37:35+0000] /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/app/controllers/base/base_controller.rb:11: warning: previous definition of V2_ROUTE_PREFIX was here
[2016-04-05 21:37:35+0000] /var/vcap/data/packages/cloud_controller_ng/0f8b96021b770958c5e092c2c985a4aeb7977f80.1-489b907bc3437165bc52a390fdc42d37d5c13383/cloud_controller_ng/app/controllers/base/front_controller.rb:7: warning: already initialized constant VCAP::CloudController::Errors
[2016-04-05 21:37:35+0000] /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/app/controllers/base/front_controller.rb:7: warning: previous definition of Errors was here
[2016-04-05 21:37:35+0000] /var/vcap/data/packages/cloud_controller_ng/0f8b96021b770958c5e092c2c985a4aeb7977f80.1-489b907bc3437165bc52a390fdc42d37d5c13383/cloud_controller_ng/app/controllers/base/model_controller.rb:8: warning: already initialized constant VCAP::CloudController::RestController::ModelController::CENSORED_MESSAGE
[2016-04-05 21:37:35+0000] /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/app/controllers/base/model_controller.rb:8: warning: previous definition of CENSORED_MESSAGE was here
[2016-04-05 21:37:53+0000] ------------ STARTING cloud_controller_ng_ctl at Tue Apr 5 21:37:53 UTC 2016 --------------

Logs from /var/vcap/data/sys/log/cloud_controller_worker_ctl.err.log:

[2016-04-05 21:36:32+0000] ------------ STARTING cloud_controller_worker_ctl at Tue Apr 5 21:36:32 UTC 2016 --------------
[2016-04-05 21:36:33+0000] ------------ STARTING cloud_controller_worker_ctl at Tue Apr 5 21:36:33 UTC 2016 --------------
[2016-04-05 21:36:51+0000] rake aborted!
[2016-04-05 21:36:51+0000] rake aborted!
[2016-04-05 21:36:51+0000] Sequel::Migrator::NotCurrentError: migrator is not current
[2016-04-05 21:36:51+0000] Sequel::Migrator::NotCurrentError: migrator is not current
[2016-04-05 21:36:51+0000] Tasks: TOP => buildpacks:install
[2016-04-05 21:36:51+0000] Tasks: TOP => jobs:local
[2016-04-05 21:36:51+0000] (See full trace by running task with --trace)
[2016-04-05 21:36:51+0000] (See full trace by running task with --trace)
[2016-04-05 21:36:53+0000] ------------ STARTING cloud_controller_worker_ctl at Tue Apr 5 21:36:53 UTC 2016 --------------
[2016-04-05 21:36:54+0000] ------------ STARTING cloud_controller_worker_ctl at Tue Apr 5 21:36:54 UTC 2016 --------------
[2016-04-05 21:37:06+0000] rake aborted!
[2016-04-05 21:37:06+0000] Sequel::Migrator::NotCurrentError: migrator is not current
[2016-04-05 21:37:06+0000] Tasks: TOP => buildpacks:install
[2016-04-05 21:37:06+0000] (See full trace by running task with --trace)
[2016-04-05 21:37:06+0000] rake aborted!
[2016-04-05 21:37:06+0000] Sequel::Migrator::NotCurrentError: migrator is not current
[2016-04-05 21:37:06+0000] Tasks: TOP => jobs:local
[2016-04-05 21:37:06+0000] (See full trace by running task with --trace)
[2016-04-05 21:37:15+0000] ------------ STARTING cloud_controller_worker_ctl at Tue Apr 5 21:37:15 UTC 2016 --------------
[2016-04-05 21:37:16+0000] ------------ STARTING cloud_controller_worker_ctl at Tue Apr 5 21:37:16 UTC 2016 --------------
[2016-04-05 21:37:29+0000] rake aborted!
[2016-04-05 21:37:29+0000] SignalException: SIGTERM


Re: Reg the upgrade from cf-205 to cf-231

Amit Kumar Gupta
 

Sounds like a new issue. Can you start a new mailing list thread?

Thanks,
Amit

On Tue, Apr 5, 2016 at 10:44 AM, Nithiyasri Gnanasekaran -X (ngnanase -
TECH MAHINDRA LIM at Cisco) <ngnanase(a)cisco.com> wrote:

Hi Amit



I tried cf login after setting the endpoint. But it fails with the below
msg. Related property (uaa.require_https= false) Last time when I got the
error, we changed this property.

But this time, instead of this setting, it fails. Cf logs below



root(a)dev-inception-vm1:/opt/cisco/vms-installer/tenant-vikramdevtes
t1/cf-deploy# cf login

API endpoint: https://api.vikramdevtest1.io



Email> admin



Password>

Authenticating...

Server error, status code: 404, error code: , message:



Password> root(a)dev-inception-vm1:/opt/cisco/vms-installer/tenant-vi
t1/cf-deploy#
ls

cf-231-final-V.yml cf-template-231.yml service.yml

cf-settings.rb cf-vikramdevtest1.yml

root(a)dev-inception-vm1:/opt/cisco/vms-installer/tenant-vikramdevtes
t1/cf-deploy# vimdiff cf-template-231.yml cf-231-final-V.yml

2 files to edit

root(a)dev-inception-vm1:/opt/cisco/vms-installer/tenant-vikramdevtes
t1/cf-deploy# CF_TRACE_true

CF_TRACE_true: command not found

root(a)dev-inception-vm1:/opt/cisco/vms-installer/tenant-vikramdevtes
t1/cf-deploy# CF_TRACE=true cf login

API endpoint: https://api.vikramdevtest1.io



REQUEST: [2016-04-05T17:38:18Z]

GET /v2/info HTTP/1.1

Host: api.vikramdevtest1.io

Accept: application/json

Content-Type: application/json

User-Agent: go-cli 6.12.2-24abed3 / linux







RESPONSE: [2016-04-05T17:38:18Z]

HTTP/1.1 200 OK

Content-Length: 586

Content-Type: application/json;charset=utf-8

Date: Tue, 05 Apr 2016 17:38:18 GMT

Server: nginx

X-Content-Type-Options: nosniff

X-Vcap-Request-Id: 54184ed0-310b-4a2f-5d5f-a1c21a397d49

X-Vcap-Request-Id: 876ca517-01fd-4f73-7a85-955955f3de41::86dcd763-5
f8e-42b3-b657-7af57ec9ea21



{"name":"","build":"","support":"http://support.cloudfoundry.com","
version":0,"description":"","authorization_endpoint":"
http://uaa.vi
kramdevtest1.io","token_endpoint":"
http://uaa.vikramdevtest1.io","m
in_cli_version":null,"min_recommended_cli_version":null,"api_versio
n":"2.51.0","app_ssh_endpoint":"ssh.vikramdevtest1.io:2222","app_ss
h_host_key_fingerprint":null,"app_ssh_oauth_client":"ssh-proxy","ro
uting_endpoint":"https://api.vikramdevtest1.io/routing","logging_en
dpoint":"wss://loggregator.vikramdevtest1.io:4443","doppler_logging
_endpoint":"wss://doppler.vikramdevtest1.io:4443"}



REQUEST: [2016-04-05T17:38:18Z]

GET /login HTTP/1.1

Host: uaa.vikramdevtest1.io

Accept: application/json

Content-Type: application/json

User-Agent: go-cli 6.12.2-24abed3 / linux







RESPONSE: [2016-04-05T17:38:18Z]

HTTP/1.1 200 OK

Content-Length: 447

Cache-Control: no-cache, no-store, max-age=0, must-revalidate

Cache-Control: no-store

Content-Language: en-US

Content-Type: application/json;charset=UTF-8

Date: Tue, 05 Apr 2016 17:38:18 GMT

Expires: 0

Pragma: no-cache

Server: Apache-Coyote/1.1

Strict-Transport-Security: max-age=31536000 ; includeSubDomains

X-Content-Type-Options: nosniff

X-Frame-Options: DENY

X-Vcap-Request-Id: d35da14d-3367-4032-6eef-d2050839147f

X-Xss-Protection: 1; mode=block



{"app":{"version":"3.1.0"},"links":{"uaa":"http://uaa.vikramdevtest
1.io
","passwd":"https://console.vikramdevtest1.io/password_resets/n
ew","login":"http://login.vikramdevtest1.io","register":"https://co
nsole.vikramdevtest1.io/register"},"zone_name":"uaa","entityID":"lo
gin.vikramdevtest1.io","commit_id":"9b5c13d","idpDefinitions":{},"p
rompts":{"username":["text","Email"],"password":["password","Passwo
rd"]},"timestamp":"2016-02-05T14:27:13+0000"}



Email> admin



Password>

Authenticating...



REQUEST: [2016-04-05T17:38:29Z]

POST /oauth/token HTTP/1.1

Host: uaa.vikramdevtest1.io

Accept: application/json

Authorization: [PRIVATE DATA HIDDEN]

Content-Type: application/x-www-form-urlencoded

User-Agent: go-cli 6.12.2-24abed3 / linux



grant_type=password&password=[PRIVATE DATA HIDDEN]&scope=&username=admin



RESPONSE: [2016-04-05T17:38:29Z]

HTTP/1.1 404 Not Found

Content-Length: 65

Content-Type: text/plain; charset=utf-8

Date: Tue, 05 Apr 2016 17:38:29 GMT

X-Cf-Routererror: unknown_route

X-Content-Type-Options: nosniff

X-Vcap-Request-Id: dab37d6c-3fea-428c-516a-ec7906ff6d16



404 Not Found: Requested route ('uaa.cisco.com') does not exist.



Server error, status code: 404, error code: , message:



Password> root(a)dev-inception-vm1
:/opt/cisco/vms-installer/tenant-vikramdevtest1/cf-deploy#



Regards

Nithiyasri


Re: Binary Service Broker Feature Narrative

Mike Youngstrom
 

An interesting idea. I see the licensing of agent binaries and upgrading
of agent binaries as a real problem that needs to be solved. I like the
idea of the brokers providing binary agent downloads for supported
platforms.

However, I'm less comfortable asking the broker to be responsible for
scripting the installation of this agent for every possible buildpack. I'd
feel better about keeping the agent configuration logic in the buildpack.
Simply having a script run at staging or startup that sets some environment
variables or something may be enough for some platforms but the integration
may be tighter and more involved for other platforms. I'm inclined to
think that how the agent is integrated into the buildpack should remain in
the buildpack.

Thoughts?

Mike

On Tue, Apr 5, 2016 at 12:15 PM, Danny Rosen <drosen(a)pivotal.io> wrote:

Hi there,
This feature narrative [1] looks to propose a new method for delivering
service-agents. This new and exciting feature would enable an ecosystem of
third-party developers to more easily create and maintain service-agents
for usage in Cloud Foundry deployments.

[1] -
https://docs.google.com/document/d/145aOpNoq7BpuB3VOzUIDh-HBx0l3v4NHLYfW8xt2zK0/edit#


Re: Cloud Controller System Domain vs App Domains

Nicholas Calugar
 

Hi John,

The point of seeding is so that there aren't two decisions (code paths) for
requests to create a route. Seeding the database requires no changes to
route creation logic.

Would anyone else like to comment on either of the two proposals before we
make a decision and start implementing a fix?


Thanks,

Nick

On Thu, Mar 31, 2016 at 10:52 AM john mcteague <john.mcteague(a)gmail.com>
wrote:

For option 2 would it not be simpler to have a single property such as
cc.blacklisted_system_domain_routes that contains the desired list and have
the CC deny route requests for those routes?

I don't see what storing them in the DB or creating real routes actually
buys us here. Everything would be available via config.

I'm in favour of some form of option 2 for those of us who have existing
deployments but would rather not make a change in either app or system
domains but are exposed to this issue.

John.
On 31 Mar 2016 5:47 a.m., "Nicholas Calugar" <ncalugar(a)pivotal.io> wrote:

Hi Cloud Foundry,

We've had a recurring issue brought to our attention regarding a Cloud
Foundry deployment using a system_domain that is in the list of
app_domains. When the system domain is in the list of app domains, a Shared
Domain is created for the system domain. This is problematic because it
allows users to create routes on the system domain, see this [1] recent
issue as an example.

[1] https://github.com/cloudfoundry/cloud_controller_ng/issues/568

I'd like to propose two solutions and get some feedback regarding which
the community would prefer. Please respond with your preferred solution and
a brief reason why.

*Solution 1 - Require a Unique System Domain*

Instead of recommending a unique system domain, we would enforce this in
the Cloud Controller. The proposed change is as follows:

1. REQUIRE system_domain to NOT be in the list of app_domains
2. REQUIRE a system_domain_organization

This will create a Private Domain for the system domain. Failure to
configure correctly would not allow the Cloud Controller to start.

If we decide to implement this, an operator should ensure their
deployment uses a unique system domain and a system_domain_organization and
correct DNS entries before upgrading.

Example for BOSH-lite

- app_domains: [bosh-lite.com]
- system_domain: system.bosh-lite.com
- system_domain_organization: system


- api endpoint: api.system.bosh-lite.com
- sample app endpoint: dora.bosh-lite.com

Example for a PaaS:

- app_domains: [yuge-paas-apps.io]
- system_domain: yuge-pass.com
- system_domain_organization: yuge-system-org


- api endpoint: api.yuge-paas.com
- sample app endpoint: dora.yuge-paas-apps.io

*Pro: *Cloud Controller now enforces what was previously the recommended
configuration for separate system and apps domains.
Con: Second SSL cert for the system domain and possibly a second DNS
record if system domain is not covered by the current wildcard record.

*Solution 2 - Cloud Controller Seeds System Routes*

To prevent a non-system app from requesting a hostname on a shared system
domain, the Cloud Controller will take a list of hostnames and seed the
database with routes. As routes are associated with a space, we will
require a system_organization and system_space. An operator could choose to
omit hostnames as desired.

cc.system_hostnames:
description: List of hostnames for which routes will be created on the
system domain.
default: [api,uaa,login,doppler,loggregator,hm9000]
cc.system_space:
description: Space where system routes will be created.
default: system

*Pro:* Significantly less change for operators running Cloud Foundry
with matching system and apps domains.
*Con: *Cloud Controller has knowledge of unrelated system components and
the list of defaults needs to be maintained as we add and remove components.


Thanks,

-Nick
--
Nicholas Calugar
CAPI Product Manager
Pivotal Software, Inc.
--
Nicholas Calugar
CAPI Product Manager
Pivotal Software, Inc.


Binary Service Broker Feature Narrative

Danny Rosen
 

Hi there,
This feature narrative [1] looks to propose a new method for delivering
service-agents. This new and exciting feature would enable an ecosystem of
third-party developers to more easily create and maintain service-agents
for usage in Cloud Foundry deployments.

[1] -
https://docs.google.com/document/d/145aOpNoq7BpuB3VOzUIDh-HBx0l3v4NHLYfW8xt2zK0/edit#


Re: Reg the upgrade from cf-205 to cf-231

Nithiyasri Gnanasekaran -X (ngnanase - TECH MAHINDRA LIM@Cisco) <ngnanase at cisco.com...>
 

Hi Amit

I tried cf login after setting the endpoint. But it fails with the below msg. Related property (uaa.require_https= false) Last time when I got the error, we changed this property.
But this time, instead of this setting, it fails. Cf logs below

root(a)dev-inception-vm1:/opt/cisco/vms-installer/tenant-vikramdevtes t1/cf-deploy# cf login
API endpoint: https://api.vikramdevtest1.io

Email> admin

Password>
Authenticating...
Server error, status code: 404, error code: , message:

Password> root(a)dev-inception-vm1:/opt/cisco/vms-installer/tenant-vi t1/cf-deploy# ls
cf-231-final-V.yml cf-template-231.yml service.yml
cf-settings.rb cf-vikramdevtest1.yml
root(a)dev-inception-vm1:/opt/cisco/vms-installer/tenant-vikramdevtes t1/cf-deploy# vimdiff cf-template-231.yml cf-231-final-V.yml
2 files to edit
root(a)dev-inception-vm1:/opt/cisco/vms-installer/tenant-vikramdevtes t1/cf-deploy# CF_TRACE_true
CF_TRACE_true: command not found
root(a)dev-inception-vm1:/opt/cisco/vms-installer/tenant-vikramdevtes t1/cf-deploy# CF_TRACE=true cf login
API endpoint: https://api.vikramdevtest1.io

REQUEST: [2016-04-05T17:38:18Z]
GET /v2/info HTTP/1.1
Host: api.vikramdevtest1.io
Accept: application/json
Content-Type: application/json
User-Agent: go-cli 6.12.2-24abed3 / linux



RESPONSE: [2016-04-05T17:38:18Z]
HTTP/1.1 200 OK
Content-Length: 586
Content-Type: application/json;charset=utf-8
Date: Tue, 05 Apr 2016 17:38:18 GMT
Server: nginx
X-Content-Type-Options: nosniff
X-Vcap-Request-Id: 54184ed0-310b-4a2f-5d5f-a1c21a397d49
X-Vcap-Request-Id: 876ca517-01fd-4f73-7a85-955955f3de41::86dcd763-5 f8e-42b3-b657-7af57ec9ea21

{"name":"","build":"","support":"http://support.cloudfoundry.com"," version":0,"description":"","authorization_endpoint":"http://uaa.vi kramdevtest1.io","token_endpoint":"http://uaa.vikramdevtest1.io","m in_cli_version":null,"min_recommended_cli_version":null,"api_versio n":"2.51.0","app_ssh_endpoint":"ssh.vikramdevtest1.io:2222","app_ss h_host_key_fingerprint":null,"app_ssh_oauth_client":"ssh-proxy","ro uting_endpoint":"https://api.vikramdevtest1.io/routing","logging_en dpoint":"wss://loggregator.vikramdevtest1.io:4443","doppler_logging _endpoint":"wss://doppler.vikramdevtest1.io:4443"}

REQUEST: [2016-04-05T17:38:18Z]
GET /login HTTP/1.1
Host: uaa.vikramdevtest1.io
Accept: application/json
Content-Type: application/json
User-Agent: go-cli 6.12.2-24abed3 / linux



RESPONSE: [2016-04-05T17:38:18Z]
HTTP/1.1 200 OK
Content-Length: 447
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Cache-Control: no-store
Content-Language: en-US
Content-Type: application/json;charset=UTF-8
Date: Tue, 05 Apr 2016 17:38:18 GMT
Expires: 0
Pragma: no-cache
Server: Apache-Coyote/1.1
Strict-Transport-Security: max-age=31536000 ; includeSubDomains
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-Vcap-Request-Id: d35da14d-3367-4032-6eef-d2050839147f
X-Xss-Protection: 1; mode=block

{"app":{"version":"3.1.0"},"links":{"uaa":"http://uaa.vikramdevtest 1.io","passwd":"https://console.vikramdevtest1.io/password_resets/n ew","login":"http://login.vikramdevtest1.io","register":"https://co nsole.vikramdevtest1.io/register"},"zone_name":"uaa","entityID":"lo gin.vikramdevtest1.io","commit_id":"9b5c13d","idpDefinitions":{},"p rompts":{"username":["text","Email"],"password":["password","Passwo rd"]},"timestamp":"2016-02-05T14:27:13+0000"}

Email> admin

Password>
Authenticating...

REQUEST: [2016-04-05T17:38:29Z]
POST /oauth/token HTTP/1.1
Host: uaa.vikramdevtest1.io
Accept: application/json
Authorization: [PRIVATE DATA HIDDEN]
Content-Type: application/x-www-form-urlencoded
User-Agent: go-cli 6.12.2-24abed3 / linux

grant_type=password&password=[PRIVATE DATA HIDDEN]&scope=&username=admin

RESPONSE: [2016-04-05T17:38:29Z]
HTTP/1.1 404 Not Found
Content-Length: 65
Content-Type: text/plain; charset=utf-8
Date: Tue, 05 Apr 2016 17:38:29 GMT
X-Cf-Routererror: unknown_route
X-Content-Type-Options: nosniff
X-Vcap-Request-Id: dab37d6c-3fea-428c-516a-ec7906ff6d16

404 Not Found: Requested route ('uaa.cisco.com') does not exist.

Server error, status code: 404, error code: , message:

Password> root(a)dev-inception-vm1:/opt/cisco/vms-installer/tenant-vikramdevtest1/cf-deploy#

Regards
Nithiyasri


Re: Spotting performance regressions

Guillaume Berche
 

Hi Michal,

I understand the routing team is working on automated performance tests
that aims at detecting performance regressions on the gorouter. We have not
yet set up execution of those at Orange yet. More in the pointers below.
The routing team was considering to open sourcing the routing ci repo, I'm
not sure of the current status now.

https://github.com/cloudfoundry-incubator/routing-perf-release
https://cf-routing.ci.cf-app.com/pipelines/routing/jobs/run-cf-load-test
https://cloudfoundry.slack.com/archives/routing/p1455232667000627
https://www.pivotaltracker.com/n/projects/1358110 with search:
label:"gorouter performance" includedone:true includes current perf
regression and effort for automated perf testing and determining max
request rate and concurrent requests

Hope this helps,

Guillaume.



Guillaume.

On Mon, Apr 4, 2016 at 4:08 PM, Michal Tekel <
michal.tekel(a)digital.cabinet-office.gov.uk> wrote:

Hi,

I wanted to check if you know about a good way to spot performance
regressions on CF and if anyone is doing it (at least partially) already.
We are mainly interested in discovering user-facing degradation. That is,
app operations (deploy, scale, delete, manage routes) and routing layer
operations (latency per request, total throughput, SSL termination
capacity) - esp. in connection with (heavier) routing services usage.

We did a brief investigation about the best way to automate this kind of
regression testing, but didn't find any complete solution. We had a look at
[CF PATs](1) tool and made a [container](2) it can be run from. But that
only gives us metrics on app operations and no way of spotting regressions
or automating the tests.

We were wondering if perhaps someone already does any kind of performance
measurements as a part of their CF build/deploy pipelines/process and what
their experience with tracking the results is...

[1] https://github.com/cloudfoundry-incubator/pat
[2] https://hub.docker.com/r/keymon/governmentpaas-cf-pats/


Re: cf create-buildpack fails in v233 with error 403 Forbidden

Wayne Ha <wayne.h.ha@...>
 

I found this in v230:

bosh-lite-v230.yml: fog_connection:
bosh-lite-v230.yml- local_root: /var/vcap/store
bosh-lite-v230.yml- provider: Local

I found it was removed in v231:

bosh-lite-v231.yml: fog_connection: null

I don't know if it is a good change but if I put the above setting back to v233 then it fixed my create buildpack problem.


OpenStack Leads

David <David@...>
 

Hi,



I hope this email finds you well.



As your organization is a part of " OpenStack Summit 2016 " I'm writing to
check if you would be interested in a leads of OpenStack users for marketing
campaign and lead generation initiative.



You may choose job titles such as: IT Architects, CTOs, IT Directors, Data
Centre Managers, Developers and more.



Each lead will be delivered to you with multi-channel data fields such as
Company name, Web address, Contact Name, Job Title, Phone Number, Direct
Business Emails, Postal address, Industry, employee size, Revenue size, SIC
code, Fax number, Application/Technology usage etc.



Appreciate your time and look forward to hearing from you.



Regards,



David Reid| Lead Generation Specialist
Lead2Tech LLC| Technology Division

If you do not wish to receive any more emails please reply to this email
with Leave out in the subject line.






<http://1816396.sigclick.mailinfo.com/sigclick/01090007/010B044E/06030500/0A
4E0020111222.jpg>


Failed to upgrade cf from 212 to 233 (etcd_z1/1 failed)

王小锋 <zzuwxf at gmail.com...>
 

Hi, there

I am trying to upgrade cf from version 212 to latest version 233 using
bosh, I am using spiff to generate the new manifest file. During upgrade,
it failed when updating job etcd_z1/1.

I login to etcd_z1/1 VM and found the following error message in etcd error
log.
Anyone meet the same issue? your help is greatly appreciated !!

2016/04/05 06:36:16 filePurge: successfully removed file
/var/vcap/store/etcd/member/snap/00000000000000d6-0000000003712a07.snap
2016/04/05 06:36:16 filePurge: successfully removed file
/var/vcap/store/etcd/member/wal/000000000000168e-0000000003712a08.wal
2016/04/05 06:46:09 rafthttp: server streaming to d7734b019d58ad5c at term
214 has been stopped
2016/04/05 06:46:09 etcdserver: removed member d7734b019d58ad5c from
cluster f61148e36337417a
2016/04/05 06:46:10 etcdserver: reject message from removed member
d7734b019d58ad5c
2016/04/05 06:46:18 etcdserver: added member e3682036054bdae0 [
http://10.10.16.20:7001] to cluster f61148e36337417a
2016/04/05 06:46:18 sender: error posting to e3682036054bdae0: dial tcp
10.10.16.20:7001: connection refused
2016/04/05 06:46:18 sender: the connection with e3682036054bdae0 became
inactive
2016/04/05 06:46:20 sender: the connection with e3682036054bdae0 became
active
2016/04/05 06:46:20 rafthttp: path msgapp/e3682036054bdae0 cannot be parsed
2016/04/05 06:46:20 rafthttp: path message/e3682036054bdae0 cannot be parsed
2016/04/05 06:46:20 raft: ddea47ef60bbcdca received msgApp
rejection(lastindex: 0) from e3682036054bdae0 for index 57802518
2016/04/05 06:46:20 raft: ddea47ef60bbcdca decreased progress of
e3682036054bdae0 to [next = 1, match = 0, wait = 0]
2016/04/05 06:46:20 raft: ddea47ef60bbcdca [firstindex: 57797981, commit:
57802541] sent snapshot[index: 57797980, term: 214] to e3682036054bdae0
[next = 1, match = 0, wait = 0]
2016/04/05 06:46:20 rafthttp: server streaming to e3682036054bdae0 at term
0 has been stopped
2016/04/05 06:46:20 rafthttp: path message/e3682036054bdae0 cannot be parsed
2016/04/05 06:46:20 rafthttp: path msgapp/e3682036054bdae0 cannot be parsed
2016/04/05 06:46:20 rafthttp: path message/e3682036054bdae0 cannot be parsed


Failure of metron agent in cloudfoundry

Bharath
 

Hi all,

we are recently running cloudfoundry in openstack. while recent saw the
metron agent in all the jobs failed . The metron.stderr.log provided the
below log

-------------
panic: 402: Standby Internal Error () [0]

goroutine 28 [running]:
github.com/cloudfoundry/loggregatorlib/servicediscovery.(*serverAddressList).Run(0xc208052120,
0x12a05f200)
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/loggregatorlib/servicediscovery/servicediscovery.go:57
+0x3c3
created by main.main

/var/vcap/data/compile/metron_agent/loggregator/src/metron/main.go:105
+0x1351

goroutine 1 [chan receive]:
main.forwardMessagesToDoppler(0xc20802c040, 0xc2080526c0, 0xc20801ece0)

/var/vcap/data/compile/metron_agent/loggregator/src/metron/main.go:187 +0x7a
main.main()

/var/vcap/data/compile/metron_agent/loggregator/src/metron/main.go:107
+0x137e

goroutine 5 [syscall]:
os/signal.loop()
/usr/local/go/src/os/signal/signal_unix.go:21 +0x1f
created by os/signal.init·1
/usr/local/go/src/os/signal/signal_unix.go:27 +0x35

goroutine 7 [select]:
github.com/cloudfoundry/gunk/workpool.worker(0xc208052600, 0xc20800bd40)
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/gunk/workpool/workpool.go:71 +0x151
created by github.com/cloudfoundry/gunk/workpool.New
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/gunk/workpool/workpool.go:44 +0x189

goroutine 8 [select]:
github.com/cloudfoundry/gunk/workpool.worker(0xc208052600, 0xc20800bd40)
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/gunk/workpool/workpool.go:71 +0x151
created by github.com/cloudfoundry/gunk/workpool.New
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/gunk/workpool/workpool.go:44 +0x189

goroutine 9 [select]:
github.com/cloudfoundry/gunk/workpool.worker(0xc208052600, 0xc20800bd40)
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/gunk/workpool/workpool.go:71 +0x151
created by github.com/cloudfoundry/gunk/workpool.New
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/gunk/workpool/workpool.go:44 +0x189

goroutine 10 [select]:
github.com/cloudfoundry/gunk/workpool.worker(0xc208052600, 0xc20800bd40)
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/gunk/workpool/workpool.go:71 +0x151
created by github.com/cloudfoundry/gunk/workpool.New
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/gunk/workpool/workpool.go:44 +0x189

goroutine 11 [select]:
github.com/cloudfoundry/gunk/workpool.worker(0xc208052600, 0xc20800bd40)
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/gunk/workpool/workpool.go:71 +0x151
created by github.com/cloudfoundry/gunk/workpool.New
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/gunk/workpool/workpool.go:44 +0x189

goroutine 12 [select]:
github.com/cloudfoundry/gunk/workpool.worker(0xc208052600, 0xc20800bd40)
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/gunk/workpool/workpool.go:71 +0x151
created by github.com/cloudfoundry/gunk/workpool.New
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/gunk/workpool/workpool.go:44 +0x189

goroutine 13 [select]:
github.com/cloudfoundry/gunk/workpool.worker(0xc208052600, 0xc20800bd40)
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/gunk/workpool/workpool.go:71 +0x151
created by github.com/cloudfoundry/gunk/workpool.New
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/gunk/workpool/workpool.go:44 +0x189

goroutine 14 [select]:
github.com/cloudfoundry/gunk/workpool.worker(0xc208052600, 0xc20800bd40)
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/gunk/workpool/workpool.go:71 +0x151
created by github.com/cloudfoundry/gunk/workpool.New
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/gunk/workpool/workpool.go:44 +0x189

goroutine 15 [select]:
github.com/cloudfoundry/gunk/workpool.worker(0xc208052600, 0xc20800bd40)
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/gunk/workpool/workpool.go:71 +0x151
created by github.com/cloudfoundry/gunk/workpool.New
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/gunk/workpool/workpool.go:44 +0x189

goroutine 16 [select]:
github.com/cloudfoundry/gunk/workpool.worker(0xc208052600, 0xc20800bd40)
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/gunk/workpool/workpool.go:71 +0x151
created by github.com/cloudfoundry/gunk/workpool.New
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/gunk/workpool/workpool.go:44 +0x189

goroutine 17 [select]:
github.com/cloudfoundry/loggregatorlib/cfcomponent/registrars/collectorregistrar.(*CollectorRegistrar).Run(0xc20800c0c0)
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/loggregatorlib/cfcomponent/registrars/collectorregistrar/collector_registrar.go:41
+0x201
created by main.main

/var/vcap/data/compile/metron_agent/loggregator/src/metron/main.go:78 +0xf7d

goroutine 18 [IO wait]:
net.(*pollDesc).Wait(0xc2080103e0, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc2080103e0, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).accept(0xc208010380, 0x0, 0x7ff2322ffbe0, 0xc20802aba8)
/usr/local/go/src/net/fd_unix.go:419 +0x40b
net.(*TCPListener).AcceptTCP(0xc20802e060, 0x596f54, 0x0, 0x0)
/usr/local/go/src/net/tcpsock_posix.go:234 +0x4e
net/http.tcpKeepAliveListener.Accept(0xc20802e060, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:1976 +0x4c
net/http.(*Server).Serve(0xc2080527e0, 0x7ff232301bb8, 0xc20802e060, 0x0,
0x0)
/usr/local/go/src/net/http/server.go:1728 +0x92
net/http.(*Server).ListenAndServe(0xc2080527e0, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:1718 +0x154
net/http.ListenAndServe(0xc20802abb0, 0xf, 0x7ff232301b90, 0xc20800bc50,
0x0, 0x0)
/usr/local/go/src/net/http/server.go:1808 +0xba
github.com/cloudfoundry/loggregatorlib/cfcomponent.Component.StartMonitoringEndpoints(0xc20801ece0,
0xc20802aac0, 0x9, 0x7ff232301940, 0xa393a8, 0x834ff0, 0xb, 0x0, 0x0, 0x0,
...)
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/loggregatorlib/cfcomponent/component.go:76 +0x719
main.startMonitoringEndpoints(0xc20801ece0, 0xc20802aac0, 0x9,
0x7ff232301940, 0xa393a8, 0x834ff0, 0xb, 0x0, 0x0, 0x0, ...)

/var/vcap/data/compile/metron_agent/loggregator/src/metron/main.go:118 +0x3e
created by main.main

/var/vcap/data/compile/metron_agent/loggregator/src/metron/main.go:80 +0xfb8

goroutine 19 [IO wait]:
net.(*pollDesc).Wait(0xc208010450, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc208010450, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).readFrom(0xc2080103f0, 0xc2080c8000, 0xffff, 0xffff, 0x0, 0x0,
0x0, 0x7ff2322ffbe0, 0xc20802acf8)
/usr/local/go/src/net/fd_unix.go:269 +0x4a1
net.(*UDPConn).ReadFromUDP(0xc20802e068, 0xc2080c8000, 0xffff, 0xffff,
0x20, 0x0, 0x0, 0x0)
/usr/local/go/src/net/udpsock_posix.go:67 +0x124
net.(*UDPConn).ReadFrom(0xc20802e068, 0xc2080c8000, 0xffff, 0xffff, 0xffff,
0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/udpsock_posix.go:82 +0x12e
github.com/cloudfoundry/loggregatorlib/agentlistener.(*agentListener).Start(0xc208010fc0)
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/loggregatorlib/agentlistener/agent_listener.go:46
+0x3a5
created by main.main

/var/vcap/data/compile/metron_agent/loggregator/src/metron/main.go:83
+0x101a

goroutine 20 [chan receive]:
metron/legacy_message/legacy_unmarshaller.(*legacyUnmarshaller).Run(0xc20802a9d0,
0xc208052540, 0xc2080523c0)

/var/vcap/data/compile/metron_agent/loggregator/src/metron/legacy_message/legacy_unmarshaller/legacy_unmarshaller.go:30
+0x71
created by main.main

/var/vcap/data/compile/metron_agent/loggregator/src/metron/main.go:86
+0x1096

goroutine 21 [chan receive]:
metron/legacy_message/legacy_message_converter.(*legacyMessageConverter).Run(0xc20802e020,
0xc2080523c0, 0xc208052300)

/var/vcap/data/compile/metron_agent/loggregator/src/metron/legacy_message/legacy_message_converter/legacy_message_converter.go:28
+0x5c
created by main.main

/var/vcap/data/compile/metron_agent/loggregator/src/metron/main.go:87
+0x10dc

goroutine 22 [IO wait]:
net.(*pollDesc).Wait(0xc208010530, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc208010530, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).readFrom(0xc2080104d0, 0xc2080e0000, 0xffff, 0xffff, 0x0, 0x0,
0x0, 0x7ff2322ffbe0, 0xc20802add0)
/usr/local/go/src/net/fd_unix.go:269 +0x4a1
net.(*UDPConn).ReadFromUDP(0xc20802e098, 0xc2080e0000, 0xffff, 0xffff,
0x20, 0x0, 0x0, 0x0)
/usr/local/go/src/net/udpsock_posix.go:67 +0x124
net.(*UDPConn).ReadFrom(0xc20802e098, 0xc2080e0000, 0xffff, 0xffff, 0xffff,
0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/udpsock_posix.go:82 +0x12e
metron/eventlistener.(*eventListener).Start(0xc208060400)

/var/vcap/data/compile/metron_agent/loggregator/src/metron/eventlistener/event_listener.go:54
+0x3a0
created by main.main

/var/vcap/data/compile/metron_agent/loggregator/src/metron/main.go:89
+0x1108

goroutine 23 [chan receive]:
github.com/cloudfoundry/dropsonde/dropsonde_unmarshaller.(*dropsondeUnmarshaller).Run(0xc20802c080,
0xc2080525a0, 0xc208052300)
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/dropsonde/dropsonde_unmarshaller/dropsonde_unmarshaller.go:65
+0x71
created by main.main

/var/vcap/data/compile/metron_agent/loggregator/src/metron/main.go:91
+0x114e

goroutine 24 [chan receive]:
metron/message_aggregator.(*messageAggregator).Run(0xc208052180,
0xc208052300, 0xc208052420)

/var/vcap/data/compile/metron_agent/loggregator/src/metron/message_aggregator/message_aggregator.go:57
+0x5c
created by main.main

/var/vcap/data/compile/metron_agent/loggregator/src/metron/main.go:94
+0x11ca

goroutine 25 [chan receive]:
metron/varz_forwarder.(*VarzForwarder).Run(0xc20802c500, 0xc208052420,
0xc208052480)

/var/vcap/data/compile/metron_agent/loggregator/src/metron/varz_forwarder/varz_forwarder.go:31
+0x51
created by main.main

/var/vcap/data/compile/metron_agent/loggregator/src/metron/main.go:97
+0x1237

goroutine 26 [chan receive]:
github.com/cloudfoundry/dropsonde/dropsonde_marshaller.(*dropsondeMarshaller).Run(0xc20801e0a0,
0xc208052480, 0xc2080524e0)
/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/dropsonde/dropsonde_marshaller/dropsonde_marshaller.go:59
+0x5f
created by main.main

/var/vcap/data/compile/metron_agent/loggregator/src/metron/main.go:100
+0x12a3

goroutine 27 [chan receive]:
main.signMessages(0xc20802b4b8, 0x8, 0xc2080524e0, 0xc2080526c0)

/var/vcap/data/compile/metron_agent/loggregator/src/metron/main.go:111 +0x7a
created by main.main

/var/vcap/data/compile/metron_agent/loggregator/src/metron/main.go:103
+0x130e

goroutine 33 [syscall, locked to thread]:
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:2232 +0x1

goroutine 31 [IO wait]:
net.(*pollDesc).Wait(0xc208010a00, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc208010a00, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).Read(0xc2080109a0, 0xc208059000, 0x1000, 0x1000, 0x0,
0x7ff2322ffbe0, 0xc2080daa00)
/usr/local/go/src/net/fd_unix.go:242 +0x40f
net.(*conn).Read(0xc20802e150, 0xc208059000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:121 +0xdc
net/http.noteEOFReader.Read(0x7ff232301d08, 0xc20802e150, 0xc20805af78,
0xc208059000, 0x1000, 0x1000, 0x75b940, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:1270 +0x6e
net/http.(*noteEOFReader).Read(0xc20801f2e0, 0xc208059000, 0x1000, 0x1000,
0xc208012000, 0x0, 0x0)
<autogenerated>:125 +0xd4
bufio.(*Reader).fill(0xc208053200)
/usr/local/go/src/bufio/bufio.go:97 +0x1ce
bufio.(*Reader).Peek(0xc208053200, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:132 +0xf0
net/http.(*persistConn).readLoop(0xc20805af20)
/usr/local/go/src/net/http/transport.go:842 +0xa4
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:660 +0xc9f

goroutine 32 [select]:
net/http.(*persistConn).writeLoop(0xc20805af20)
/usr/local/go/src/net/http/transport.go:945 +0x41d
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:661 +0xcbc

goroutine 36 [IO wait]:
net.(*pollDesc).Wait(0xc20810a3e0, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc20810a3e0, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).Read(0xc20810a380, 0xc2080fb000, 0x1000, 0x1000, 0x0,
0x7ff2322ffbe0, 0xc2080dad28)
/usr/local/go/src/net/fd_unix.go:242 +0x40f
net.(*conn).Read(0xc20802e318, 0xc2080fb000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:121 +0xdc
net/http.noteEOFReader.Read(0x7ff232301d08, 0xc20802e318, 0xc20805ad68,
0xc2080fb000, 0x1000, 0x1000, 0x75b940, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:1270 +0x6e
net/http.(*noteEOFReader).Read(0xc2080dd000, 0xc2080fb000, 0x1000, 0x1000,
0xc208012000, 0x0, 0x0)
<autogenerated>:125 +0xd4
bufio.(*Reader).fill(0xc2081086c0)
/usr/local/go/src/bufio/bufio.go:97 +0x1ce
bufio.(*Reader).Peek(0xc2081086c0, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:132 +0xf0
net/http.(*persistConn).readLoop(0xc20805ad10)
/usr/local/go/src/net/http/transport.go:842 +0xa4
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:660 +0xc9f

goroutine 37 [select]:
net/http.(*persistConn).writeLoop(0xc20805ad10)
/usr/local/go/src/net/http/transport.go:945 +0x41d
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:661 +0xcbc
----------------------

I tried to restart it using monit restart still it is failing .
any body got idea on this error


regards
Bharath

4961 - 4980 of 9426