Re: Diego IAAS settings for vsphere


Eric Malm <emalm@...>
 

Hi, Daya,

Thanks very much for the additional logs. Those BBS errors look normal
(requests often bounce off nonexistent records), and from the auctioneer
logs the auction process is retrieving state from your one cell correctly.
From the adjacent 'auctioneer.auction.scheduling' and
'auctioneer.auction.scheduled' log lines at the end, though, it does look
like the auction is failing to place the desired task on the only available
cell. Since those log lines are less than a millisecond apart, it's
extremely unlikely that the auctioneer is telling the cell to take on the
task and the cell is refusing (although we could also look for that
interaction in the cell rep logs). Unfortunately, we intentionally don't
have a lot of logging in the code that runs the placement algorithm, so it
may be impossible in this version of the codebase to introspect the
specific decisions it's making about task placement.

One other pathway to try would be to move an existing app from the DEAs to
the Diego backend via the Diego-Enabler CLI plugin (
https://github.com/cloudfoundry-incubator/Diego-Enabler), and see if its
corresponding LRP is placed correctly. The placement decision should be
based entirely on matching the LRP's rootfs to the cell and then validating
the cell has enough capacity, just as it is for Tasks. It may be easier to
compare (and adjust through CC) the resource requirements of the app's
DesiredLRP that it would be on the more ephemeral Task submitted for
staging. In any case, if you haven't already resolved the placement
problems, please try out moving a test app over and let us know how that
turns out.

Best,
Eric

On Wed, Jan 27, 2016 at 3:37 PM, Shetty, Daya <Daya.Shetty(a)bnymellon.com>
wrote:

Hi, Eric,

As per your suggestion I changed the logLevel=debug and did a tail on the
auctioneer & bbs logs while doing a cf push of buildpack based app to
diego. Unfortunately I still saw only LogLevel: 1 messsages on the
Auctioneer logs but did see LogLevel 0,1 & 2 messages on the bbs logs.

I have attached the bbs.log. Here are some errors I see in the bbs log.

{"timestamp":"1453934586.220547676","source":"bbs","message":"bbs.actuallrp-handler.actual-lrp-groups.etcd-error.resource-not-found","log_level":0,"data":{"error":{"errorCode":100,"message":"Key
not found","cause":"/v1","index":5},"session":"3.19.1"}}

{"timestamp":"1453934586.224381685","source":"bbs","message":"bbs.task-handler.tasks.etcd-error.resource-not-found","log_level":0,"data":{"error":{"errorCode":100,"message":"Key
not found","cause":"/v1","index":5},"session":"7.25.1"}}

"timestamp":"1453934595.548539162","source":"bbs","message":"bbs.desiredlrp-handler.remove-desired-lrp.remove-desired-lrp.failed","log_level":2,"data":{"error":"100:
Key not found (/v1/desired_lrp)
[8]","process-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-e42855a5-0122-4e1a-a003-a4eac1fe3deb","session":"6.19.1"}}

{"timestamp":"1453934595.548731089","source":"bbs","message":"bbs.desiredlrp-handler.remove-desired-lrp.remove-desired-lrp.etcd-error.resource-not-found","log_level":0,"data":{"error":{"errorCode":100,"message":"Key
not
found","cause":"/v1/desired_lrp","index":8},"process-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-e42855a5-0122-4e1a-a003-a4eac1fe3deb","session":"6.19.1.1"}}

{"timestamp":"1453934595.735231638","source":"bbs","message":"bbs.actuallrp-handler.actual-lrp-groups.etcd-error.resource-not-found","log_level":0,"data":{"error":{"errorCode":100,"message":"Key
not found","cause":"/v1/actual","index":9},"session":"3.20.1"}}

{"timestamp":"1453934595.738165140","source":"bbs","message":"bbs.desiredlrp-handler.desired-lrps.desired-lrp-scheduling-infos.etcd-error.resource-not-found","log_level":0,"data":{"error":{"errorCode":100,"message":"Key
not
found","cause":"/v1/desired_lrp","index":9},"filter":{"Domain":""},"session":"6.20.1.1"}}

{"timestamp":"1453934600.773372650","source":"bbs","message":"bbs.desiredlrp-handler.desired-lrps.desired-lrp-scheduling-infos.etcd-error.resource-not-found","log_level":0,"data":{"error":{"errorCode":100,"message":"Key
not
found","cause":"/v1/desired_lrp","index":9},"filter":{"Domain":"cf-apps"},"session":"6.21.1.1"}}

{"timestamp":"1453934610.201195955","source":"bbs","message":"bbs.converge-lrps.etcd-error.resource-not-found","log_level":0,"data":{"error":{"errorCode":100,"message":"Key
not found","cause":"/v1/actual","index":9},"session":"111.1"}}

{"timestamp":"1453934610.205472708","source":"bbs","message":"bbs.converge-lrps.etcd-error.resource-not-found","log_level":0,"data":{"error":{"errorCode":100,"message":"Key
not found","cause":"/v1/desired_lrp","index":9},"session":"111.2"}}

{"timestamp":"1453934616.234182835","source":"bbs","message":"bbs.actuallrp-handler.actual-lrp-groups.etcd-error.resource-not-found","log_level":0,"data":{"error":{"errorCode":100,"message":"Key
not found","cause":"/v1/actual","index":9},"session":"3.21.1"}}

Regards
Daya


From: Daya Shetty <Daya.Shetty(a)bnymellon.com>
Reply-To: "Discussions about Cloud Foundry projects and the system
overall." <cf-dev(a)lists.cloudfoundry.org>
Date: Wednesday, January 27, 2016 at 2:00 PM
To: "Discussions about Cloud Foundry projects and the system overall." <
cf-dev(a)lists.cloudfoundry.org>
Subject: [cf-dev] Re: Re: Re: Re: Re: Re: Re: Re: Diego IAAS settings for
vsphere

Hi, Eric,

I tried to enable debug using the debugAddr but the log_level remained
as 1. Not sure why?

curl -v http://0.0.0.0:17001/log-level -X PUT -d debug

* Hostname was NOT found in DNS cache

* Trying 0.0.0.0...

* Connected to 0.0.0.0 (127.0.0.1) port 17001 (#0)

PUT /log-level HTTP/1.1
User-Agent: curl/7.35.0
Host: 0.0.0.0:17001
Accept: */*
Content-Length: 5
Content-Type: application/x-www-form-urlencoded
* upload completely sent off: 5 out of 5 bytes

< HTTP/1.1 200 OK

< Date: Wed, 27 Jan 2016 21:44:44 GMT

< Content-Length: 0

< Content-Type: text/plain; charset=utf-8

<

- Connection #0 to host 0.0.0.0 left intact

var/vcap/packages/*auction*eer/bin/*auction*eer
-bbsClientCert=/var/vcap/jobs/*auction*eer/config/certs/bbs/client.crt
-bbsClientKey=/var/vcap/jobs/*auction*eer/config/certs/bbs/client.key
-bbsCACert=/var/vcap/jobs/*auction*eer/config/certs/bbs/ca.crt
-bbsAddress=https://bbs.service.cf.internal:8889 -consulCluster=
http://127.0.0.1:8500 -debugAddr=0.0.0.0:17001 -listenAddr=0.0.0.0:9016
-logLevel=info

I will restart the processes mentioned, with —logLevel=debug by changing
rhe relevant _ctl files, In the meantime here is the snapshot of the
auctioneer log while trying to start a buildpack based app.


Here is the auctioneer.stdout log

root(a)fb99aa6d-3a9a-48c5-b6e9-dccfbc035c7a:/var/vcap/data/sys/log/auctioneer#
!ta

tail -f auctioneer.stdout.log


{"timestamp":"1453931031.236279488","source":"auctioneer","message":"auctioneer.request.done","log_level":1,"data":{"method":"POST","request":"/v1/tasks","session":"36"}}


{"timestamp":"1453931031.236993074","source":"auctioneer","message":"auctioneer.auction.fetching-cell-reps","log_level":1,"data":{"session":"37"}}


{"timestamp":"1453931031.240644693","source":"auctioneer","message":"auctioneer.auction.fetched-cell-reps","log_level":1,"data":{"cell-reps-count":1,"session":"37"}}


{"timestamp":"1453931031.240906715","source":"auctioneer","message":"auctioneer.auction.fetching-zone-state","log_level":1,"data":{"session":"37"}}


{"timestamp":"1453931031.248399019","source":"auctioneer","message":"auctioneer.auction.zone-state","log_level":1,"data":{"cell-count":1,"session":"37","zone":"z1"}}


{"timestamp":"1453931031.249227762","source":"auctioneer","message":"auctioneer.auction.fetched-zone-state","log_level":1,"data":{"cell-state-count":1,"duration":"6.511719ms","num-failed-requests":0,"session":"37"}}


{"timestamp":"1453931031.249767065","source":"auctioneer","message":"auctioneer.auction.fetching-auctions","log_level":1,"data":{"session":"37"}}


{"timestamp":"1453931031.250007153","source":"auctioneer","message":"auctioneer.auction.fetched-auctions","log_level":1,"data":{"lrp-start-auctions":0,"session":"37","task-auctions":1}}


{"timestamp":"1453931031.250244141","source":"auctioneer","message":"auctioneer.auction.scheduling","log_level":1,"data":{"session":"37"}}


{"timestamp":"1453931031.250724792","source":"auctioneer","message":"auctioneer.auction.scheduled","log_level":1,"data":{"failed-lrp-start-auctions":0,"failed-task-auctions":1,"session":"37","successful-lrp-start-auctions":0,"successful-task-auctions":0}}


{"timestamp":"1453931106.945099831","source":"auctioneer","message":"auctioneer.request.serving","log_level":1,"data":{"method":"POST","request":"/v1/tasks","session":"38"}}


{"timestamp":"1453931106.946578026","source":"auctioneer","message":"auctioneer.request.task-auction-handler.create.submitted","log_level":1,"data":{"method":"POST","request":"/v1/tasks","session":"38.1.1","tasks":["da383b70-f3da-4248-86b5-9a82e6d7d809-2d07956c22c142849c2b160c46ddb480"]}}


{"timestamp":"1453931106.948045492","source":"auctioneer","message":"auctioneer.request.done","log_level":1,"data":{"method":"POST","request":"/v1/tasks","session":"38"}}


{"timestamp":"1453931106.948791265","source":"auctioneer","message":"auctioneer.auction.fetching-cell-reps","log_level":1,"data":{"session":"39"}}


{"timestamp":"1453931106.952327490","source":"auctioneer","message":"auctioneer.auction.fetched-cell-reps","log_level":1,"data":{"cell-reps-count":1,"session":"39"}}


{"timestamp":"1453931106.952946186","source":"auctioneer","message":"auctioneer.auction.fetching-zone-state","log_level":1,"data":{"session":"39"}}


{"timestamp":"1453931106.959112644","source":"auctioneer","message":"auctioneer.auction.zone-state","log_level":1,"data":{"cell-count":1,"session":"39","zone":"z1"}}


{"timestamp":"1453931106.959350586","source":"auctioneer","message":"auctioneer.auction.fetched-zone-state","log_level":1,"data":{"cell-state-count":1,"duration":"5.455613ms","num-failed-requests":0,"session":"39"}}


{"timestamp":"1453931106.959569216","source":"auctioneer","message":"auctioneer.auction.fetching-auctions","log_level":1,"data":{"session":"39"}}


{"timestamp":"1453931106.959800005","source":"auctioneer","message":"auctioneer.auction.fetched-auctions","log_level":1,"data":{"lrp-start-auctions":0,"session":"39","task-auctions":1}}


{"timestamp":"1453931106.960027695","source":"auctioneer","message":"auctioneer.auction.scheduling","log_level":1,"data":{"session":"39"}}


{"timestamp":"1453931106.960274935","source":"auctioneer","message":"auctioneer.auction.scheduled","log_level":1,"data":{"failed-lrp-start-auctions":0,"failed-task-auctions":1,"session":"39","successful-lrp-start-auctions":0,"successful-task-auctions":0}}

Regards
Daya


From: Eric Malm <emalm(a)pivotal.io>
Reply-To: "Discussions about Cloud Foundry projects and the system
overall." <cf-dev(a)lists.cloudfoundry.org>
Date: Wednesday, January 27, 2016 at 12:29 PM
To: "Discussions about Cloud Foundry projects and the system overall." <
cf-dev(a)lists.cloudfoundry.org>
Subject: [cf-dev] Re: Re: Re: Re: Re: Re: Re: Diego IAAS settings for
vsphere

Hi, Daya,

Thanks for the log entries and the state response from the cell. It does
look like that cell should have sufficient capacity to take on the staging
task. From the BBS logs for the 'da383b70-f3da-4248-86b5-9a82e6d7d809-
6db1e598fc2f466287a8f29c01a71bd8' task guid, it does look like the task
is getting saved in the BBS, and then failed by something else (presumably
the auctioneer) almost immediately thereafter. Could you also look at the
auctioneer logs for that period of time?

Also, it might help to turn on debug logging dynamically on these
components: if you do a `netstat -anp` on the bosh VM, you'll see each
Diego component process listening on a port 170XX in the 17000 range. If
you then run `curl http://localhost:170XX/log-level -X PUT -d debug` on
that VM, that will change the log-level on that component to 'debug' (it
normally defaults to 'info'). So if we're not seeing anything useful at the
info/error level for this period of time, you could switch the active bbs
and auctioneer to the debug log-level, start tailing their logs in
real-time with `tail –f`, and try submitting another staging task.

Thanks,
Eric

On Wed, Jan 27, 2016 at 11:31 AM, Daya Shetty <daya.shetty(a)bnymellon.com>
wrote:

Hi, Eric,

I further checked the logs you mentioned and can't see what could be the
problem.

I only have one instance of cell running and here is the state info:
curl 0.0.0.0:1800/state

{"RootFSProviders":{"docker":{"type":"arbitrary"},"preloaded":{"set":{"cflinuxfs2":{}},"type":"fixed_set"}},"AvailableResources":{"MemoryMB":7984,"DiskMB":24263,"Containers":256},"TotalResources":{"MemoryMB":7984,"DiskMB":24263,"Containers":256},"LRPs":[],"Tasks":null,"Zone":"z1","Evacuating":false}


I see TLS handshake errors in the bbs.stderr.log, don't know if this can
be the cause for" Insufficient resources"
2016/01/27 19:12:03 http: TLS handshake error from 127.0.0.1:50475: EOF
2016/01/27 19:12:06 http: TLS handshake error from 127.0.0.1:50477: EOF

Not able to deduce anything from the bbs.stdout.log here is a snapshot.
Will dig further.


"timestamp":"1453921657.161973238","source":"bbs","message":"bbs.request.serving","log_level":1,"data":{"method":"POST","request":"/v1/actual_lrp_groups/list","session":"19946"}}

{"timestamp":"1453921657.162120819","source":"bbs","message":"bbs.request.serving","log_level":1,"data":{"method":"POST","request":"/v1/tasks/list","session":"19947"}}

{"timestamp":"1453921657.167166233","source":"bbs","message":"bbs.request.done","log_level":1,"data":{"method":"POST","request":"/v1/tasks/list","session":"19947"}}

{"timestamp":"1453921657.167504787","source":"bbs","message":"bbs.request.done","log_level":1,"data":{"method":"POST","request":"/v1/actual_lrp_groups/list","session":"19946"}}

{"timestamp":"1453921674.952786684","source":"bbs","message":"bbs.request.serving","log_level":1,"data":{"method":"POST","request":"/v1/actual_lrp_groups/list","session":"19948"}}

{"timestamp":"1453921674.953184366","source":"bbs","message":"bbs.request.serving","log_level":1,"data":{"method":"POST","request":"/v1/desired_lrp_scheduling_infos/list","session":"19949"}}

{"timestamp":"1453921674.953428984","source":"bbs","message":"bbs.desiredlrp-handler.desired-lrps.desired-lrp-scheduling-infos.start","log_level":1,"data":{"filter":{"Domain":""},"session":"6.3791.1"}}

{"timestamp":"1453921674.954186201","source":"bbs","message":"bbs.desiredlrp-handler.desired-lrps.desired-lrp-scheduling-infos.complete","log_level":1,"data":{"filter":{"Domain":""},"session":"6.3791.1"}}

{"timestamp":"1453921674.954262018","source":"bbs","message":"bbs.request.done","log_level":1,"data":{"method":"POST","request":"/v1/desired_lrp_scheduling_infos/list","session":"19949"}}

{"timestamp":"1453921674.959604502","source":"bbs","message":"bbs.request.done","log_level":1,"data":{"method":"POST","request":"/v1/actual_lrp_groups/list","session":"19948"}}

{"timestamp":"1453921681.551017284","source":"bbs","message":"bbs.request.serving","log_level":1,"data":{"method":"POST","request":"/v1/tasks/desire","session":"19950"}}

{"timestamp":"1453921681.551474571","source":"bbs","message":"bbs.task-handler.desire-task.desire-task.starting","log_level":1,"data":{"session":"7.5881.1","task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}}

{"timestamp":"1453921681.561704636","source":"bbs","message":"bbs.task-handler.desire-task.desire-task.finished","log_level":1,"data":{"session":"7.5881.1","task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}}

{"timestamp":"1453921681.561844826","source":"bbs","message":"bbs.request.done","log_level":1,"data":{"method":"POST","request":"/v1/tasks/desire","session":"19950"}}

{"timestamp":"1453921681.575248480","source":"bbs","message":"bbs.request.serving","log_level":1,"data":{"method":"POST","request":"/v1/tasks/fail","session":"19951"}}

{"timestamp":"1453921681.575376987","source":"bbs","message":"bbs.task-handler.fail-task.fail-task.starting","log_level":1,"data":{"session":"7.5882.1","task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}}

{"timestamp":"1453921681.575446367","source":"bbs","message":"bbs.task-handler.fail-task.fail-task.getting-task","log_level":1,"data":{"session":"7.5882.1","task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}}

{"timestamp":"1453921681.577617884","source":"bbs","message":"bbs.task-handler.fail-task.fail-task.succeeded-getting-task","log_level":1,"data":{"session":"7.5882.1","task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}}

{"timestamp":"1453921681.578390121","source":"bbs","message":"bbs.task-handler.fail-task.fail-task.persisting-task","log_level":1,"data":{"session":"7.5882.1","task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}}

{"timestamp":"1453921681.582517862","source":"bbs","message":"bbs.task-handler.fail-task.fail-task.succeded-persisting-task","log_level":1,"data":{"session":"7.5882.1","task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}}

{"timestamp":"1453921681.583042145","source":"bbs","message":"bbs.task-handler.fail-task.fail-task.task-client-completing-task","log_level":1,"data":{"session":"7.5882.1","task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}}

{"timestamp":"1453921681.583634138","source":"bbs","message":"bbs.task-handler.fail-task.fail-task.finished","log_level":1,"data":{"session":"7.5882.1","task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}}

{"timestamp":"1453921681.583706856","source":"bbs","message":"bbs.resolving-task","log_level":1,"data":{"task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}}

{"timestamp":"1453921681.584264040","source":"bbs","message":"bbs.resolving-task.starting","log_level":1,"data":{"session":"1","task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}}

{"timestamp":"1453921681.584349394","source":"bbs","message":"bbs.resolving-task.getting-task","log_level":1,"data":{"session":"1","task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}}

{"timestamp":"1453921681.584867001","source":"bbs","message":"bbs.request.done","log_level":1,"data":{"method":"POST","request":"/v1/tasks/fail","session":"19951"}}

{"timestamp":"1453921681.587889433","source":"bbs","message":"bbs.resolving-task.succeeded-getting-task","log_level":1,"data":{"session":"1","task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}}

{"timestamp":"1453921681.591948748","source":"bbs","message":"bbs.resolving-task.finished","log_level":1,"data":{"session":"1","task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}}

{"timestamp":"1453921681.629187346","source":"bbs","message":"bbs.request.serving","log_level":1,"data":{"method":"POST","request":"/v1/desired_lrp/remove","session":"19952"}}

{"timestamp":"1453921681.629388809","source":"bbs","message":"bbs.desiredlrp-handler.remove-desired-lrp.remove-desired-lrp.starting","log_level":1,"data":{"process-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-c398c85b-3965-4fd7-91e9-3112bafe3394","session":"6.3792.1"}}
{"timestamp":"1453921681.630566120","source":"bbs","message":"bbs.desiredlrp-handler.remove-desired-lrp.remove-desired-lrp.failed","log_level":2,"data":{"error":"100:
Key not found (/v1/desired_lrp)
[64]","process-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-c398c85b-3965-4fd7-91e9-3112bafe3394","session":"6.3792.1"}}

{"timestamp":"1453921681.631013870","source":"bbs","message":"bbs.desiredlrp-handler.remove-desired-lrp.remove-desired-lrp.complete","log_level":1,"data":{"process-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-c398c85b-3965-4fd7-91e9-3112bafe3394","session":"6.3792.1"}}

{"timestamp":"1453921681.631548882","source":"bbs","message":"bbs.request.done","log_level":1,"data":{"method":"POST","request":"/v1/desired_lrp/remove","session":"19952"}}

{"timestamp":"1453921681.639199495","source":"bbs","message":"bbs.delete-task.starting","log_level":1,"data":{"callback_url":"
http://stager.service.cf.internal:8888/v1/staging/da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8/completed
","session":"1","task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}}

5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8/completed","status_code":200,"task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}}

{"timestamp":"1453921686.492143154","source":"bbs","message":"bbs.request.serving","log_level":1,"data":{"method":"POST","request":"/v1/lrps/converge","session":"19953"}}

{"timestamp":"1453921686.492546797","source":"bbs","message":"bbs.converge-lrps.starting-convergence","log_level":1,"data":{"session":"19954"}}

{"timestamp":"1453921686.492759466","source":"bbs","message":"bbs.converge-lrps.gathering-and-pruning-actual-lrps","log_level":1,"data":{"session":"19954"}}

{"timestamp":"1453921686.493501425","source":"bbs","message":"bbs.request.serving","log_level":1,"data":{"method":"POST","request":"/v1/tasks/converge","session":"19955"}}

{"timestamp":"1453921686.493688107","source":"bbs","message":"bbs.task-handler.converge-tasks.converge-tasks.starting-convergence","log_level":1,"data":{"session":"7.5883.1"}}

{"timestamp":"1453921686.495135784","source":"bbs","message":"bbs.converge-lrps.actual-lrp-schema-root-not-found","log_level":1,"data":{"session":"19954"}}

{"timestamp":"1453921686.495184660","source":"bbs","message":"bbs.converge-lrps.succeeded-gathering-and-pruning-actual-lrps","log_level":1,"data":{"session":"19954"}}

{"timestamp":"1453921686.495222807","source":"bbs","message":"bbs.converge-lrps.gathering-desired-lrps","log_level":1,"data":{"session":"19954"}}

{"timestamp":"1453921686.499060631","source":"bbs","message":"bbs.converge-lrps.actual-lrp-schema-root-not-found","log_level":1,"data":{"session":"19954"}}

{"timestamp":"1453921686.499403954","source":"bbs","message":"bbs.converge-lrps.succeeded-gathering-desired-lrps","log_level":1,"data":{"session":"19954"}}

{"timestamp":"1453921686.501797915","source":"bbs","message":"bbs.task-handler.converge-tasks.converge-tasks.finished-convergence","log_level":1,"data":{"session":"7.5883.1"}}

{"timestamp":"1453921686.502251863","source":"bbs","message":"bbs.request.done","log_level":1,"data":{"method":"POST","request":"/v1/tasks/converge","session":"19955"}}

{"timestamp":"1453921686.504722357","source":"bbs","message":"bbs.converge-lrps.calculate-convergence.start","log_level":1,"data":{"session":"19954.3"}}

{"timestamp":"1453921686.504907846","source":"bbs","message":"bbs.converge-lrps.calculate-convergence.done","log_level":1,"data":{"session":"19954.3"}}

{"timestamp":"1453921686.505190134","source":"bbs","message":"bbs.converge-lrps.finished-convergence","log_level":1,"data":{"session":"19954"}}

{"timestamp":"1453921686.505403280","source":"bbs","message":"bbs.request.done","log_level":1,"data":{"method":"POST","request":"/v1/lrps/converge","session":"19953"}}

{"timestamp":"1453921687.169877052","source":"bbs","message":"bbs.request.serving","log_level":1,"data":{"method":"POST","request":"/v1/tasks/list","session":"19956"}}

{"timestamp":"1453921687.171530724","source":"bbs","message":"bbs.request.done","log_level":1,"data":{"method":"POST","request":"/v1/tasks/list","session":"19956"}}

{"timestamp":"1453921687.172345877","source":"bbs","message":"bbs.request.serving","log_level":1,"data":{"method":"POST","request":"/v1/actual_lrp_groups/list","session":"19957"}}

{"timestamp":"1453921687.176663637","source":"bbs","message":"bbs.request.done","log_level":1,"data":{"method":"POST","request":"/v1/actual_lrp_groups/list","session":"19957"}}

{"timestamp":"1453921692.995214939","source":"bbs","message":"bbs.request.serving","log_level":1,"data":{"method":"POST","request":"/v1/desired_lrp_scheduling_infos/list","session":"19958"}}

{"timestamp":"1453921692.995497704","source":"bbs","message":"bbs.desiredlrp-handler.desired-lrps.desired-lrp-scheduling-infos.start","log_level":1,"data":{"filter":{"Domain":"cf-apps"},"session":"6.3793.1"}}

{"timestamp":"1453921692.996849060","source":"bbs","message":"bbs.desiredlrp-handler.desired-lrps.desired-lrp-scheduling-infos.complete","log_level":1,"data":{"filter":{"Domain":"cf-apps"},"session":"6.3793.1"}}

{"timestamp":"1453921692.997039795","source":"bbs","message":"bbs.request.done","log_level":1,"data":{"method":"POST","request":"/v1/desired_lrp_scheduling_infos/list","session":"19958"}}

The information contained in this e-mail, and any attachment, is
confidential and is intended solely for the use of the intended recipient.
Access, copying or re-use of the e-mail or any attachment, or any
information contained therein, by any other person is not authorized. If
you are not the intended recipient please return the e-mail to the sender
and delete it from your computer. Although we attempt to sweep e-mail and
attachments for viruses, we do not guarantee that either are virus-free and
accept no liability for any damage sustained as a result of viruses.

Please refer to http://disclaimer.bnymellon.com/eu.htm for certain
disclosures relating to European legal entities.

The information contained in this e-mail, and any attachment, is
confidential and is intended solely for the use of the intended recipient.
Access, copying or re-use of the e-mail or any attachment, or any
information contained therein, by any other person is not authorized. If
you are not the intended recipient please return the e-mail to the sender
and delete it from your computer. Although we attempt to sweep e-mail and
attachments for viruses, we do not guarantee that either are virus-free and
accept no liability for any damage sustained as a result of viruses.

Please refer to http://disclaimer.bnymellon.com/eu.htm for certain
disclosures relating to European legal entities.

Join cf-dev@lists.cloudfoundry.org to automatically receive all group messages.