Date
1 - 15 of 15
Diego IAAS settings for vsphere
|
Daya Shetty <daya.shetty@...>
Is there a reference vsphere iaas-settings.yml for Diego?
Thanks Daya |
|
|
|
Eric Malm <emalm@...>
Hi, Daya,
The Diego team doesn't currently maintain a Diego iaas-settings stub specifically for vSphere environments. Filling one out for your environment should require specifying only the following sets of information, though: - Stemcell name (presumably 'bosh-vsphere-esxi-ubuntu-trusty-go_agent') and version. - Network information for diego1, diego2, and diego3 subnets (cloud_properties, range, dns, gateway, reserved, and static sections). For vSphere, the only relevant cloud property should be 'name'. Also, as of Diego 0.1446.0, the Diego deployment-manifest templates no longer specify static IP addresses for any of the VMs. - vSphere-specific cloud properties for the Diego VM resource pools (cpu, disk, and ram). - vSphere-specific cloud properties for compilation VMs (cpu, disk, and ram). These VMs will be allocated in the diego1 subnet. - vSphere-specific cloud properties for the database_disks disk pool to be attached to the Diego database VMs. For vSphere, the only relevant cloud property should be 'type'. I've included below an attempt at such a stub based on the vSphere CPI docs (http://bosh.io/docs/vsphere-cpi.html), as well as the instance sizing contained in the CF vSphere infrastructure stub ( https://github.com/cloudfoundry/cf-release/blob/develop/templates/cf-infrastructure-vsphere.yml). The network configuration details would of course have to be replaced with ones for your environment. If that iaas-settings stub or something like it works correctly for your environment, please let me know, and we can then include it in the 'manifest-generation/examples' directory in diego-release. Best, Eric, CF Runtime Diego PM ***** iaas_settings: compilation_cloud_properties: cpu: 2 disk: 8192 ram: 1024 disk_pools: - cloud_properties: type: thick name: database_disks resource_pool_cloud_properties: - cloud_properties: cpu: 1 disk: 4096 ram: 1024 name: access_z1 - cloud_properties: cpu: 1 disk: 4096 ram: 1024 name: access_z2 - cloud_properties: cpu: 1 disk: 4096 ram: 1024 name: access_z3 - cloud_properties: cpu: 1 disk: 4096 ram: 1024 name: brain_z1 - cloud_properties: cpu: 1 disk: 4096 ram: 1024 name: brain_z2 - cloud_properties: cpu: 1 disk: 4096 ram: 1024 name: brain_z3 - cloud_properties: cpu: 1 disk: 4096 ram: 1024 name: cc_bridge_z1 - cloud_properties: cpu: 1 disk: 4096 ram: 1024 name: cc_bridge_z2 - cloud_properties: cpu: 1 disk: 4096 ram: 1024 name: cc_bridge_z3 - cloud_properties: cpu: 2 disk: 32768 ram: 8192 name: cell_z1 - cloud_properties: cpu: 2 disk: 32768 ram: 8192 name: cell_z2 - cloud_properties: cpu: 2 disk: 32768 ram: 8192 name: cell_z3 - cloud_properties: cpu: 2 disk: 32768 ram: 8192 name: colocated_z1 - cloud_properties: cpu: 2 disk: 32768 ram: 8192 name: colocated_z2 - cloud_properties: cpu: 2 disk: 32768 ram: 8192 name: colocated_z3 - cloud_properties: cpu: 2 disk: 16384 ram: 4096 name: database_z1 - cloud_properties: cpu: 2 disk: 16384 ram: 4096 name: database_z2 - cloud_properties: cpu: 2 disk: 16384 ram: 4096 name: database_z3 - cloud_properties: cpu: 1 disk: 4096 ram: 1024 name: route_emitter_z1 - cloud_properties: cpu: 1 disk: 4096 ram: 1024 name: route_emitter_z2 - cloud_properties: cpu: 1 disk: 4096 ram: 1024 name: route_emitter_z3 stemcell: name: bosh-vsphere-esxi-ubuntu-trusty-go_agent version: latest subnet_configs: - name: diego1 subnets: - cloud_properties: name: PLACEHOLDER-diego1-network-name dns: - 10.10.0.2 gateway: 10.10.5.1 range: 10.10.5.0/24 reserved: - 10.10.5.2 - 10.10.5.9 static: - 10.10.5.10 - 10.10.5.63 - name: diego2 subnets: - cloud_properties: name: PLACEHOLDER-diego2-network-name dns: - 10.10.0.2 gateway: 10.10.6.1 range: 10.10.6.0/24 reserved: - 10.10.6.2 - 10.10.6.9 static: - 10.10.6.10 - 10.10.6.63 - name: diego3 subnets: - cloud_properties: name: PLACEHOLDER-diego3-network-name dns: - 10.10.0.2 gateway: 10.10.7.1 range: 10.10.7.0/24 reserved: - 10.10.7.2 - 10.10.7.9 static: - 10.10.7.10 - 10.10.7.63 On Tue, Jan 19, 2016 at 9:53 PM, Daya Shetty <daya.shetty(a)bnymellon.com> wrote: Is there a reference vsphere iaas-settings.yml for Diego? |
|
|
|
Eric Malm <emalm@...>
Hi, Daya,
toggle quoted message
Show quoted text
The stub I included in-line in my previous email may not have come through so well for all mail clients, so I've also included it in a public gist at https://gist.github.com/ematpl/236ed28870c680ac7d4c. Thanks, Eric On Sun, Jan 24, 2016 at 5:07 PM, Eric Malm <emalm(a)pivotal.io> wrote:
Hi, Daya, |
|
|
|
Daya Shetty <daya.shetty@...>
Eric,
Thanks for the response, I was already working on one with help from your previous response for Diego on AWS but this helps a lot to speed it up. Regards Daya |
|
|
|
Daya Shetty <daya.shetty@...>
Hi Eric,
Thanks for your helpI I could conjure up an iass-settings for vsphere and was able to deploy diego successfully but was having the same access_z1 VM failing problem but was able to bring up diego after incorporating your comments on this thread . https://lists.cloudfoundry.org/archives/list/cf-dev(a)lists.cloudfoundry.org/thread/JDD3SEP7CV6ONGED3DFAFYN2I54JQSIT/ Trying to push a docker app and I’m getting cf push my-tcp-receiver -o cloudfoundry/tcp-sample-receiver Creating app my-tcp-receiver in org BNYMellon / space testspace as admin... OK Creating route my-tcp-receiver.poc-apps.bnymellon.net... OK Binding my-tcp-receiver.poc-apps.bnymellon.net to my-tcp-receiver... OK Starting app my-tcp-receiver in org BNYMellon / space testspace as admin... FAILED InsufficientResources I checked the stager.stdout.log on the cc-bridge and it shows the following: "timestamp":"1453833835.995078564","source":"stager","message":"stager.starting","log_level":1,"data":{}} {"timestamp":"1453833835.995591879","source":"stager","message":"stager.Listening for staging requests!","log_level":1,"data":{}} {"timestamp":"1453850050.902931690","source":"stager","message":"stager.docker.build-recipe.staging-request","log_level":1,"data":{"app-id":"6992baee-ef29-4805-92ce-acac5268b78e","session":"2.1","staging-guid":"6992baee-ef29-4805-92ce-acac5268b78e-6e72ac5779384f68a04a245ade7184b6"}} {"timestamp":"1453850050.903894424","source":"stager","message":"stager.staging-handler.staging-request.desiring-task","log_level":1,"data":{"callback_url":"http://stager.service.cf.internal:8888/v1/staging/6992baee-ef29-4805-92ce-acac5268b78e-6e72ac5779384f68a04a245ade7184b6/completed","session":"3.1","staging-guid":"6992baee-ef29-4805-92ce-acac5268b78e-6e72ac5779384f68a04a245ade7184b6","task_guid":"6992baee-ef29-4805-92ce-acac5268b78e-6e72ac5779384f68a04a245ade7184b6"}} {"timestamp":"1453850051.046576500","source":"stager","message":"stager.completion-handler.task-complete-callback-received.posting-staging-complete","log_level":1,"data":{"guid":"6992baee-ef29-4805-92ce-acac5268b78e-6e72ac5779384f68a04a245ade7184b6","payload":"eyJlcnJvciI6eyJpZCI6Ikluc3VmZmljaWVudFJlc291cmNlcyIsIm1lc3NhZ2UiOiJpbnN1ZmZpY2llbnQgcmVzb3VyY2VzIn19","session":"4.1"}} {"timestamp":"1453850051.047005653","source":"stager","message":"stager.completion-handler.task-complete-callback-received.cc-client.delivering-staging-response","log_level":1,"data":{"guid":"6992baee-ef29-4805-92ce-acac5268b78e-6e72ac5779384f68a04a245ade7184b6","payload":"{\"error\":{\"id\":\"InsufficientResources\",\"message\":\"insufficient resources\"}}","session":"4.1.1"}} {"timestamp":"1453850051.143590689","source":"stager","message":"stager.completion-handler.task-complete-callback-received.cc-client.delivered-staging-response","log_level":1,"data":{"guid":"6992baee-ef29-4805-92ce-acac5268b78e-6e72ac5779384f68a04a245ade7184b6","session":"4.1.1"}} {"timestamp":"1453850051.143915415","source":"stager","message":"stager.completion-handler.task-complete-callback-received.posted-staging-complete","log_level":1,"data":{"guid":"6992baee-ef29-4805-92ce-acac5268b78e-6e72ac5779384f68a04a245ade7184b6","session":"4.1”}} I have the following releases deployed: root(a)rsomtapae181 cf-release]# bosh releases Acting as user 'admin' on 'bosh2' +----------------------------+------------+-------------+ | Name | Versions | Commit Hash | +----------------------------+------------+-------------+ | cf | 222+dev.1* | e4eb9f4b+ | | diego | 0.1437.0* | 7a972628 | | etcd | 16* | 793d1c6b | | garden-linux | 0.308.0* | b5eced17 | +----------------------------+------------+-------------+ (*) Currently deployed (+) Uncommitted changes Any ideas on where I could look to debug this issue? Thanks Daya |
|
|
|
Eric Malm <emalm@...>
Hi, Daya,
I'd start by looking at the auctioneer logs at `/var/vcap/sys/log/auctioneer/auctioneer.stdout.log` on the brain VMs and the BBS logs at `/var/vcap/sys/log/bbs/bbs.stdout.log` on the database VMs. You could also get the state response from each cell by curling ` http://CELL_IP:1800/state` <http://CELL_IP:1800/state> to see the resources it advertises to the auctioneer. I think CC by default configures staging tasks to use 6G of disk, so it's possible that none of the cells in the deployment have the remaining capacity to accept the work, especially if they're already running other workloads. Are you able to stage or run a buildpack-based app against the Diego backend? Thanks, Eric On Tue, Jan 26, 2016 at 4:00 PM, Daya Shetty <daya.shetty(a)bnymellon.com> wrote: Hi Eric, |
|
|
|
Daya Shetty <daya.shetty@...>
Hi, Eric,
FYI.. staging a buildpack-based app against the Diego backend also gave me the same Failure (InsufficientResources) but I was able to deploy and start the same buildpack based app using the Warden container. I will check the logs you mentioned above to see if I can debug this problem. Surprising thing is that I was able to download the same apps successfully on my bosh-lite instance using the Diego backend. Thanks Daya |
|
|
|
Daya Shetty <daya.shetty@...>
Hi, Eric,
FYI.. staging a buildpack-based app against the Diego backend also gave me the same Failure (InsufficientResources) but I was able to deploy and start the same buildpack based app using the Warden container. I will check the logs you mentioned above to see if I can debug this problem. Surprising thing is that I was able to download the same apps successfully on my bosh-lite instance using the Diego backend. Thanks Daya |
|
|
|
Daya Shetty <daya.shetty@...>
Hi, Eric,
I further checked the logs you mentioned and can't see what could be the problem. I only have one instance of cell running and here is the state info: curl 0.0.0.0:1800/state {"RootFSProviders":{"docker":{"type":"arbitrary"},"preloaded":{"set":{"cflinuxfs2":{}},"type":"fixed_set"}},"AvailableResources":{"MemoryMB":7984,"DiskMB":24263,"Containers":256},"TotalResources":{"MemoryMB":7984,"DiskMB":24263,"Containers":256},"LRPs":[],"Tasks":null,"Zone":"z1","Evacuating":false} I see TLS handshake errors in the bbs.stderr.log, don't know if this can be the cause for" Insufficient resources" 2016/01/27 19:12:03 http: TLS handshake error from 127.0.0.1:50475: EOF 2016/01/27 19:12:06 http: TLS handshake error from 127.0.0.1:50477: EOF Not able to deduce anything from the bbs.stdout.log here is a snapshot. Will dig further. "timestamp":"1453921657.161973238","source":"bbs","message":"bbs.request.serving","log_level":1,"data":{"method":"POST","request":"/v1/actual_lrp_groups/list","session":"19946"}} {"timestamp":"1453921657.162120819","source":"bbs","message":"bbs.request.serving","log_level":1,"data":{"method":"POST","request":"/v1/tasks/list","session":"19947"}} {"timestamp":"1453921657.167166233","source":"bbs","message":"bbs.request.done","log_level":1,"data":{"method":"POST","request":"/v1/tasks/list","session":"19947"}} {"timestamp":"1453921657.167504787","source":"bbs","message":"bbs.request.done","log_level":1,"data":{"method":"POST","request":"/v1/actual_lrp_groups/list","session":"19946"}} {"timestamp":"1453921674.952786684","source":"bbs","message":"bbs.request.serving","log_level":1,"data":{"method":"POST","request":"/v1/actual_lrp_groups/list","session":"19948"}} {"timestamp":"1453921674.953184366","source":"bbs","message":"bbs.request.serving","log_level":1,"data":{"method":"POST","request":"/v1/desired_lrp_scheduling_infos/list","session":"19949"}} {"timestamp":"1453921674.953428984","source":"bbs","message":"bbs.desiredlrp-handler.desired-lrps.desired-lrp-scheduling-infos.start","log_level":1,"data":{"filter":{"Domain":""},"session":"6.3791.1"}} {"timestamp":"1453921674.954186201","source":"bbs","message":"bbs.desiredlrp-handler.desired-lrps.desired-lrp-scheduling-infos.complete","log_level":1,"data":{"filter":{"Domain":""},"session":"6.3791.1"}} {"timestamp":"1453921674.954262018","source":"bbs","message":"bbs.request.done","log_level":1,"data":{"method":"POST","request":"/v1/desired_lrp_scheduling_infos/list","session":"19949"}} {"timestamp":"1453921674.959604502","source":"bbs","message":"bbs.request.done","log_level":1,"data":{"method":"POST","request":"/v1/actual_lrp_groups/list","session":"19948"}} {"timestamp":"1453921681.551017284","source":"bbs","message":"bbs.request.serving","log_level":1,"data":{"method":"POST","request":"/v1/tasks/desire","session":"19950"}} {"timestamp":"1453921681.551474571","source":"bbs","message":"bbs.task-handler.desire-task.desire-task.starting","log_level":1,"data":{"session":"7.5881.1","task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}} {"timestamp":"1453921681.561704636","source":"bbs","message":"bbs.task-handler.desire-task.desire-task.finished","log_level":1,"data":{"session":"7.5881.1","task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}} {"timestamp":"1453921681.561844826","source":"bbs","message":"bbs.request.done","log_level":1,"data":{"method":"POST","request":"/v1/tasks/desire","session":"19950"}} {"timestamp":"1453921681.575248480","source":"bbs","message":"bbs.request.serving","log_level":1,"data":{"method":"POST","request":"/v1/tasks/fail","session":"19951"}} {"timestamp":"1453921681.575376987","source":"bbs","message":"bbs.task-handler.fail-task.fail-task.starting","log_level":1,"data":{"session":"7.5882.1","task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}} {"timestamp":"1453921681.575446367","source":"bbs","message":"bbs.task-handler.fail-task.fail-task.getting-task","log_level":1,"data":{"session":"7.5882.1","task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}} {"timestamp":"1453921681.577617884","source":"bbs","message":"bbs.task-handler.fail-task.fail-task.succeeded-getting-task","log_level":1,"data":{"session":"7.5882.1","task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}} {"timestamp":"1453921681.578390121","source":"bbs","message":"bbs.task-handler.fail-task.fail-task.persisting-task","log_level":1,"data":{"session":"7.5882.1","task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}} {"timestamp":"1453921681.582517862","source":"bbs","message":"bbs.task-handler.fail-task.fail-task.succeded-persisting-task","log_level":1,"data":{"session":"7.5882.1","task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}} {"timestamp":"1453921681.583042145","source":"bbs","message":"bbs.task-handler.fail-task.fail-task.task-client-completing-task","log_level":1,"data":{"session":"7.5882.1","task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}} {"timestamp":"1453921681.583634138","source":"bbs","message":"bbs.task-handler.fail-task.fail-task.finished","log_level":1,"data":{"session":"7.5882.1","task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}} {"timestamp":"1453921681.583706856","source":"bbs","message":"bbs.resolving-task","log_level":1,"data":{"task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}} {"timestamp":"1453921681.584264040","source":"bbs","message":"bbs.resolving-task.starting","log_level":1,"data":{"session":"1","task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}} {"timestamp":"1453921681.584349394","source":"bbs","message":"bbs.resolving-task.getting-task","log_level":1,"data":{"session":"1","task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}} {"timestamp":"1453921681.584867001","source":"bbs","message":"bbs.request.done","log_level":1,"data":{"method":"POST","request":"/v1/tasks/fail","session":"19951"}} {"timestamp":"1453921681.587889433","source":"bbs","message":"bbs.resolving-task.succeeded-getting-task","log_level":1,"data":{"session":"1","task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}} {"timestamp":"1453921681.591948748","source":"bbs","message":"bbs.resolving-task.finished","log_level":1,"data":{"session":"1","task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}} {"timestamp":"1453921681.629187346","source":"bbs","message":"bbs.request.serving","log_level":1,"data":{"method":"POST","request":"/v1/desired_lrp/remove","session":"19952"}} {"timestamp":"1453921681.629388809","source":"bbs","message":"bbs.desiredlrp-handler.remove-desired-lrp.remove-desired-lrp.starting","log_level":1,"data":{"process-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-c398c85b-3965-4fd7-91e9-3112bafe3394","session":"6.3792.1"}} {"timestamp":"1453921681.630566120","source":"bbs","message":"bbs.desiredlrp-handler.remove-desired-lrp.remove-desired-lrp.failed","log_level":2,"data":{"error":"100: Key not found (/v1/desired_lrp) [64]","process-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-c398c85b-3965-4fd7-91e9-3112bafe3394","session":"6.3792.1"}} {"timestamp":"1453921681.631013870","source":"bbs","message":"bbs.desiredlrp-handler.remove-desired-lrp.remove-desired-lrp.complete","log_level":1,"data":{"process-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-c398c85b-3965-4fd7-91e9-3112bafe3394","session":"6.3792.1"}} {"timestamp":"1453921681.631548882","source":"bbs","message":"bbs.request.done","log_level":1,"data":{"method":"POST","request":"/v1/desired_lrp/remove","session":"19952"}} {"timestamp":"1453921681.639199495","source":"bbs","message":"bbs.delete-task.starting","log_level":1,"data":{"callback_url":"http://stager.service.cf.internal:8888/v1/staging/da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8/completed","session":"1","task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}} 5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8/completed","status_code":200,"task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}} {"timestamp":"1453921686.492143154","source":"bbs","message":"bbs.request.serving","log_level":1,"data":{"method":"POST","request":"/v1/lrps/converge","session":"19953"}} {"timestamp":"1453921686.492546797","source":"bbs","message":"bbs.converge-lrps.starting-convergence","log_level":1,"data":{"session":"19954"}} {"timestamp":"1453921686.492759466","source":"bbs","message":"bbs.converge-lrps.gathering-and-pruning-actual-lrps","log_level":1,"data":{"session":"19954"}} {"timestamp":"1453921686.493501425","source":"bbs","message":"bbs.request.serving","log_level":1,"data":{"method":"POST","request":"/v1/tasks/converge","session":"19955"}} {"timestamp":"1453921686.493688107","source":"bbs","message":"bbs.task-handler.converge-tasks.converge-tasks.starting-convergence","log_level":1,"data":{"session":"7.5883.1"}} {"timestamp":"1453921686.495135784","source":"bbs","message":"bbs.converge-lrps.actual-lrp-schema-root-not-found","log_level":1,"data":{"session":"19954"}} {"timestamp":"1453921686.495184660","source":"bbs","message":"bbs.converge-lrps.succeeded-gathering-and-pruning-actual-lrps","log_level":1,"data":{"session":"19954"}} {"timestamp":"1453921686.495222807","source":"bbs","message":"bbs.converge-lrps.gathering-desired-lrps","log_level":1,"data":{"session":"19954"}} {"timestamp":"1453921686.499060631","source":"bbs","message":"bbs.converge-lrps.actual-lrp-schema-root-not-found","log_level":1,"data":{"session":"19954"}} {"timestamp":"1453921686.499403954","source":"bbs","message":"bbs.converge-lrps.succeeded-gathering-desired-lrps","log_level":1,"data":{"session":"19954"}} {"timestamp":"1453921686.501797915","source":"bbs","message":"bbs.task-handler.converge-tasks.converge-tasks.finished-convergence","log_level":1,"data":{"session":"7.5883.1"}} {"timestamp":"1453921686.502251863","source":"bbs","message":"bbs.request.done","log_level":1,"data":{"method":"POST","request":"/v1/tasks/converge","session":"19955"}} {"timestamp":"1453921686.504722357","source":"bbs","message":"bbs.converge-lrps.calculate-convergence.start","log_level":1,"data":{"session":"19954.3"}} {"timestamp":"1453921686.504907846","source":"bbs","message":"bbs.converge-lrps.calculate-convergence.done","log_level":1,"data":{"session":"19954.3"}} {"timestamp":"1453921686.505190134","source":"bbs","message":"bbs.converge-lrps.finished-convergence","log_level":1,"data":{"session":"19954"}} {"timestamp":"1453921686.505403280","source":"bbs","message":"bbs.request.done","log_level":1,"data":{"method":"POST","request":"/v1/lrps/converge","session":"19953"}} {"timestamp":"1453921687.169877052","source":"bbs","message":"bbs.request.serving","log_level":1,"data":{"method":"POST","request":"/v1/tasks/list","session":"19956"}} {"timestamp":"1453921687.171530724","source":"bbs","message":"bbs.request.done","log_level":1,"data":{"method":"POST","request":"/v1/tasks/list","session":"19956"}} {"timestamp":"1453921687.172345877","source":"bbs","message":"bbs.request.serving","log_level":1,"data":{"method":"POST","request":"/v1/actual_lrp_groups/list","session":"19957"}} {"timestamp":"1453921687.176663637","source":"bbs","message":"bbs.request.done","log_level":1,"data":{"method":"POST","request":"/v1/actual_lrp_groups/list","session":"19957"}} {"timestamp":"1453921692.995214939","source":"bbs","message":"bbs.request.serving","log_level":1,"data":{"method":"POST","request":"/v1/desired_lrp_scheduling_infos/list","session":"19958"}} {"timestamp":"1453921692.995497704","source":"bbs","message":"bbs.desiredlrp-handler.desired-lrps.desired-lrp-scheduling-infos.start","log_level":1,"data":{"filter":{"Domain":"cf-apps"},"session":"6.3793.1"}} {"timestamp":"1453921692.996849060","source":"bbs","message":"bbs.desiredlrp-handler.desired-lrps.desired-lrp-scheduling-infos.complete","log_level":1,"data":{"filter":{"Domain":"cf-apps"},"session":"6.3793.1"}} {"timestamp":"1453921692.997039795","source":"bbs","message":"bbs.request.done","log_level":1,"data":{"method":"POST","request":"/v1/desired_lrp_scheduling_infos/list","session":"19958"}} |
|
|
|
Eric Malm <emalm@...>
Hi, Daya,
Thanks for the log entries and the state response from the cell. It does look like that cell should have sufficient capacity to take on the staging task. From the BBS logs for the 'da383b70-f3da-4248-86b5-9a82e6d7d809- 6db1e598fc2f466287a8f29c01a71bd8' task guid, it does look like the task is getting saved in the BBS, and then failed by something else (presumably the auctioneer) almost immediately thereafter. Could you also look at the auctioneer logs for that period of time? Also, it might help to turn on debug logging dynamically on these components: if you do a `netstat -anp` on the bosh VM, you'll see each Diego component process listening on a port 170XX in the 17000 range. If you then run `curl http://localhost:170XX/log-level -X PUT -d debug` on that VM, that will change the log-level on that component to 'debug' (it normally defaults to 'info'). So if we're not seeing anything useful at the info/error level for this period of time, you could switch the active bbs and auctioneer to the debug log-level, start tailing their logs in real-time with `tail -f`, and try submitting another staging task. Thanks, Eric On Wed, Jan 27, 2016 at 11:31 AM, Daya Shetty <daya.shetty(a)bnymellon.com> wrote: Hi, Eric, |
|
|
|
Shetty, Daya <Daya.Shetty@...>
Hi, Eric,
I tried to enable debug using the debugAddr but the log_level remained as 1. Not sure why? curl -v http://0.0.0.0:17001/log-level -X PUT -d debug * Hostname was NOT found in DNS cache * Trying 0.0.0.0... * Connected to 0.0.0.0 (127.0.0.1) port 17001 (#0) PUT /log-level HTTP/1.1 User-Agent: curl/7.35.0 Host: 0.0.0.0:17001 Accept: */* Content-Length: 5 Content-Type: application/x-www-form-urlencoded* upload completely sent off: 5 out of 5 bytes < HTTP/1.1 200 OK < Date: Wed, 27 Jan 2016 21:44:44 GMT < Content-Length: 0 < Content-Type: text/plain; charset=utf-8 < * Connection #0 to host 0.0.0.0 left intact var/vcap/packages/auctioneer/bin/auctioneer -bbsClientCert=/var/vcap/jobs/auctioneer/config/certs/bbs/client.crt -bbsClientKey=/var/vcap/jobs/auctioneer/config/certs/bbs/client.key -bbsCACert=/var/vcap/jobs/auctioneer/config/certs/bbs/ca.crt -bbsAddress=https://bbs.service.cf.internal:8889 -consulCluster=http://127.0.0.1:8500 -debugAddr=0.0.0.0:17001 -listenAddr=0.0.0.0:9016 -logLevel=info I will restart the processes mentioned, with —logLevel=debug by changing rhe relevant _ctl files, In the meantime here is the snapshot of the auctioneer log while trying to start a buildpack based app. Here is the auctioneer.stdout log root(a)fb99aa6d-3a9a-48c5-b6e9-dccfbc035c7a:/var/vcap/data/sys/log/auctioneer# !ta tail -f auctioneer.stdout.log {"timestamp":"1453931031.236279488","source":"auctioneer","message":"auctioneer.request.done","log_level":1,"data":{"method":"POST","request":"/v1/tasks","session":"36"}} {"timestamp":"1453931031.236993074","source":"auctioneer","message":"auctioneer.auction.fetching-cell-reps","log_level":1,"data":{"session":"37"}} {"timestamp":"1453931031.240644693","source":"auctioneer","message":"auctioneer.auction.fetched-cell-reps","log_level":1,"data":{"cell-reps-count":1,"session":"37"}} {"timestamp":"1453931031.240906715","source":"auctioneer","message":"auctioneer.auction.fetching-zone-state","log_level":1,"data":{"session":"37"}} {"timestamp":"1453931031.248399019","source":"auctioneer","message":"auctioneer.auction.zone-state","log_level":1,"data":{"cell-count":1,"session":"37","zone":"z1"}} {"timestamp":"1453931031.249227762","source":"auctioneer","message":"auctioneer.auction.fetched-zone-state","log_level":1,"data":{"cell-state-count":1,"duration":"6.511719ms","num-failed-requests":0,"session":"37"}} {"timestamp":"1453931031.249767065","source":"auctioneer","message":"auctioneer.auction.fetching-auctions","log_level":1,"data":{"session":"37"}} {"timestamp":"1453931031.250007153","source":"auctioneer","message":"auctioneer.auction.fetched-auctions","log_level":1,"data":{"lrp-start-auctions":0,"session":"37","task-auctions":1}} {"timestamp":"1453931031.250244141","source":"auctioneer","message":"auctioneer.auction.scheduling","log_level":1,"data":{"session":"37"}} {"timestamp":"1453931031.250724792","source":"auctioneer","message":"auctioneer.auction.scheduled","log_level":1,"data":{"failed-lrp-start-auctions":0,"failed-task-auctions":1,"session":"37","successful-lrp-start-auctions":0,"successful-task-auctions":0}} {"timestamp":"1453931106.945099831","source":"auctioneer","message":"auctioneer.request.serving","log_level":1,"data":{"method":"POST","request":"/v1/tasks","session":"38"}} {"timestamp":"1453931106.946578026","source":"auctioneer","message":"auctioneer.request.task-auction-handler.create.submitted","log_level":1,"data":{"method":"POST","request":"/v1/tasks","session":"38.1.1","tasks":["da383b70-f3da-4248-86b5-9a82e6d7d809-2d07956c22c142849c2b160c46ddb480"]}} {"timestamp":"1453931106.948045492","source":"auctioneer","message":"auctioneer.request.done","log_level":1,"data":{"method":"POST","request":"/v1/tasks","session":"38"}} {"timestamp":"1453931106.948791265","source":"auctioneer","message":"auctioneer.auction.fetching-cell-reps","log_level":1,"data":{"session":"39"}} {"timestamp":"1453931106.952327490","source":"auctioneer","message":"auctioneer.auction.fetched-cell-reps","log_level":1,"data":{"cell-reps-count":1,"session":"39"}} {"timestamp":"1453931106.952946186","source":"auctioneer","message":"auctioneer.auction.fetching-zone-state","log_level":1,"data":{"session":"39"}} {"timestamp":"1453931106.959112644","source":"auctioneer","message":"auctioneer.auction.zone-state","log_level":1,"data":{"cell-count":1,"session":"39","zone":"z1"}} {"timestamp":"1453931106.959350586","source":"auctioneer","message":"auctioneer.auction.fetched-zone-state","log_level":1,"data":{"cell-state-count":1,"duration":"5.455613ms","num-failed-requests":0,"session":"39"}} {"timestamp":"1453931106.959569216","source":"auctioneer","message":"auctioneer.auction.fetching-auctions","log_level":1,"data":{"session":"39"}} {"timestamp":"1453931106.959800005","source":"auctioneer","message":"auctioneer.auction.fetched-auctions","log_level":1,"data":{"lrp-start-auctions":0,"session":"39","task-auctions":1}} {"timestamp":"1453931106.960027695","source":"auctioneer","message":"auctioneer.auction.scheduling","log_level":1,"data":{"session":"39"}} {"timestamp":"1453931106.960274935","source":"auctioneer","message":"auctioneer.auction.scheduled","log_level":1,"data":{"failed-lrp-start-auctions":0,"failed-task-auctions":1,"session":"39","successful-lrp-start-auctions":0,"successful-task-auctions":0}} Regards Daya From: Eric Malm <emalm(a)pivotal.io<mailto:emalm(a)pivotal.io>> Reply-To: "Discussions about Cloud Foundry projects and the system overall." <cf-dev(a)lists.cloudfoundry.org<mailto:cf-dev(a)lists.cloudfoundry.org>> Date: Wednesday, January 27, 2016 at 12:29 PM To: "Discussions about Cloud Foundry projects and the system overall." <cf-dev(a)lists.cloudfoundry.org<mailto:cf-dev(a)lists.cloudfoundry.org>> Subject: [cf-dev] Re: Re: Re: Re: Re: Re: Re: Diego IAAS settings for vsphere Hi, Daya, Thanks for the log entries and the state response from the cell. It does look like that cell should have sufficient capacity to take on the staging task. From the BBS logs for the 'da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8' task guid, it does look like the task is getting saved in the BBS, and then failed by something else (presumably the auctioneer) almost immediately thereafter. Could you also look at the auctioneer logs for that period of time? Also, it might help to turn on debug logging dynamically on these components: if you do a `netstat -anp` on the bosh VM, you'll see each Diego component process listening on a port 170XX in the 17000 range. If you then run `curl http://localhost:170XX/log-level -X PUT -d debug` on that VM, that will change the log-level on that component to 'debug' (it normally defaults to 'info'). So if we're not seeing anything useful at the info/error level for this period of time, you could switch the active bbs and auctioneer to the debug log-level, start tailing their logs in real-time with `tail –f`, and try submitting another staging task. Thanks, Eric On Wed, Jan 27, 2016 at 11:31 AM, Daya Shetty <daya.shetty(a)bnymellon.com<mailto:daya.shetty(a)bnymellon.com>> wrote: Hi, Eric, I further checked the logs you mentioned and can't see what could be the problem. I only have one instance of cell running and here is the state info: curl 0.0.0.0:1800/state<http://0.0.0.0:1800/state> {"RootFSProviders":{"docker":{"type":"arbitrary"},"preloaded":{"set":{"cflinuxfs2":{}},"type":"fixed_set"}},"AvailableResources":{"MemoryMB":7984,"DiskMB":24263,"Containers":256},"TotalResources":{"MemoryMB":7984,"DiskMB":24263,"Containers":256},"LRPs":[],"Tasks":null,"Zone":"z1","Evacuating":false} I see TLS handshake errors in the bbs.stderr.log, don't know if this can be the cause for" Insufficient resources" 2016/01/27 19:12:03 http: TLS handshake error from 127.0.0.1:50475<http://127.0.0.1:50475>: EOF 2016/01/27 19:12:06 http: TLS handshake error from 127.0.0.1:50477<http://127.0.0.1:50477>: EOF Not able to deduce anything from the bbs.stdout.log here is a snapshot. Will dig further. "timestamp":"1453921657.161973238","source":"bbs","message":"bbs.request.serving","log_level":1,"data":{"method":"POST","request":"/v1/actual_lrp_groups/list","session":"19946"}} {"timestamp":"1453921657.162120819","source":"bbs","message":"bbs.request.serving","log_level":1,"data":{"method":"POST","request":"/v1/tasks/list","session":"19947"}} {"timestamp":"1453921657.167166233","source":"bbs","message":"bbs.request.done","log_level":1,"data":{"method":"POST","request":"/v1/tasks/list","session":"19947"}} {"timestamp":"1453921657.167504787","source":"bbs","message":"bbs.request.done","log_level":1,"data":{"method":"POST","request":"/v1/actual_lrp_groups/list","session":"19946"}} {"timestamp":"1453921674.952786684","source":"bbs","message":"bbs.request.serving","log_level":1,"data":{"method":"POST","request":"/v1/actual_lrp_groups/list","session":"19948"}} {"timestamp":"1453921674.953184366","source":"bbs","message":"bbs.request.serving","log_level":1,"data":{"method":"POST","request":"/v1/desired_lrp_scheduling_infos/list","session":"19949"}} {"timestamp":"1453921674.953428984","source":"bbs","message":"bbs.desiredlrp-handler.desired-lrps.desired-lrp-scheduling-infos.start","log_level":1,"data":{"filter":{"Domain":""},"session":"6.3791.1"}} {"timestamp":"1453921674.954186201","source":"bbs","message":"bbs.desiredlrp-handler.desired-lrps.desired-lrp-scheduling-infos.complete","log_level":1,"data":{"filter":{"Domain":""},"session":"6.3791.1"}} {"timestamp":"1453921674.954262018","source":"bbs","message":"bbs.request.done","log_level":1,"data":{"method":"POST","request":"/v1/desired_lrp_scheduling_infos/list","session":"19949"}} {"timestamp":"1453921674.959604502","source":"bbs","message":"bbs.request.done","log_level":1,"data":{"method":"POST","request":"/v1/actual_lrp_groups/list","session":"19948"}} {"timestamp":"1453921681.551017284","source":"bbs","message":"bbs.request.serving","log_level":1,"data":{"method":"POST","request":"/v1/tasks/desire","session":"19950"}} {"timestamp":"1453921681.551474571","source":"bbs","message":"bbs.task-handler.desire-task.desire-task.starting","log_level":1,"data":{"session":"7.5881.1","task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}} {"timestamp":"1453921681.561704636","source":"bbs","message":"bbs.task-handler.desire-task.desire-task.finished","log_level":1,"data":{"session":"7.5881.1","task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}} {"timestamp":"1453921681.561844826","source":"bbs","message":"bbs.request.done","log_level":1,"data":{"method":"POST","request":"/v1/tasks/desire","session":"19950"}} {"timestamp":"1453921681.575248480","source":"bbs","message":"bbs.request.serving","log_level":1,"data":{"method":"POST","request":"/v1/tasks/fail","session":"19951"}} {"timestamp":"1453921681.575376987","source":"bbs","message":"bbs.task-handler.fail-task.fail-task.starting","log_level":1,"data":{"session":"7.5882.1","task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}} {"timestamp":"1453921681.575446367","source":"bbs","message":"bbs.task-handler.fail-task.fail-task.getting-task","log_level":1,"data":{"session":"7.5882.1","task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}} {"timestamp":"1453921681.577617884","source":"bbs","message":"bbs.task-handler.fail-task.fail-task.succeeded-getting-task","log_level":1,"data":{"session":"7.5882.1","task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}} {"timestamp":"1453921681.578390121","source":"bbs","message":"bbs.task-handler.fail-task.fail-task.persisting-task","log_level":1,"data":{"session":"7.5882.1","task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}} {"timestamp":"1453921681.582517862","source":"bbs","message":"bbs.task-handler.fail-task.fail-task.succeded-persisting-task","log_level":1,"data":{"session":"7.5882.1","task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}} {"timestamp":"1453921681.583042145","source":"bbs","message":"bbs.task-handler.fail-task.fail-task.task-client-completing-task","log_level":1,"data":{"session":"7.5882.1","task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}} {"timestamp":"1453921681.583634138","source":"bbs","message":"bbs.task-handler.fail-task.fail-task.finished","log_level":1,"data":{"session":"7.5882.1","task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}} {"timestamp":"1453921681.583706856","source":"bbs","message":"bbs.resolving-task","log_level":1,"data":{"task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}} {"timestamp":"1453921681.584264040","source":"bbs","message":"bbs.resolving-task.starting","log_level":1,"data":{"session":"1","task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}} {"timestamp":"1453921681.584349394","source":"bbs","message":"bbs.resolving-task.getting-task","log_level":1,"data":{"session":"1","task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}} {"timestamp":"1453921681.584867001","source":"bbs","message":"bbs.request.done","log_level":1,"data":{"method":"POST","request":"/v1/tasks/fail","session":"19951"}} {"timestamp":"1453921681.587889433","source":"bbs","message":"bbs.resolving-task.succeeded-getting-task","log_level":1,"data":{"session":"1","task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}} {"timestamp":"1453921681.591948748","source":"bbs","message":"bbs.resolving-task.finished","log_level":1,"data":{"session":"1","task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}} {"timestamp":"1453921681.629187346","source":"bbs","message":"bbs.request.serving","log_level":1,"data":{"method":"POST","request":"/v1/desired_lrp/remove","session":"19952"}} {"timestamp":"1453921681.629388809","source":"bbs","message":"bbs.desiredlrp-handler.remove-desired-lrp.remove-desired-lrp.starting","log_level":1,"data":{"process-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-c398c85b-3965-4fd7-91e9-3112bafe3394","session":"6.3792.1"}} {"timestamp":"1453921681.630566120","source":"bbs","message":"bbs.desiredlrp-handler.remove-desired-lrp.remove-desired-lrp.failed","log_level":2,"data":{"error":"100: Key not found (/v1/desired_lrp) [64]","process-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-c398c85b-3965-4fd7-91e9-3112bafe3394","session":"6.3792.1"}} {"timestamp":"1453921681.631013870","source":"bbs","message":"bbs.desiredlrp-handler.remove-desired-lrp.remove-desired-lrp.complete","log_level":1,"data":{"process-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-c398c85b-3965-4fd7-91e9-3112bafe3394","session":"6.3792.1"}} {"timestamp":"1453921681.631548882","source":"bbs","message":"bbs.request.done","log_level":1,"data":{"method":"POST","request":"/v1/desired_lrp/remove","session":"19952"}} {"timestamp":"1453921681.639199495","source":"bbs","message":"bbs.delete-task.starting","log_level":1,"data":{"callback_url":"http://stager.service.cf.internal:8888/v1/staging/da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8/completed","session":"1","task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}} 5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8/completed","status_code":200,"task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}} {"timestamp":"1453921686.492143154","source":"bbs","message":"bbs.request.serving","log_level":1,"data":{"method":"POST","request":"/v1/lrps/converge","session":"19953"}} {"timestamp":"1453921686.492546797","source":"bbs","message":"bbs.converge-lrps.starting-convergence","log_level":1,"data":{"session":"19954"}} {"timestamp":"1453921686.492759466","source":"bbs","message":"bbs.converge-lrps.gathering-and-pruning-actual-lrps","log_level":1,"data":{"session":"19954"}} {"timestamp":"1453921686.493501425","source":"bbs","message":"bbs.request.serving","log_level":1,"data":{"method":"POST","request":"/v1/tasks/converge","session":"19955"}} {"timestamp":"1453921686.493688107","source":"bbs","message":"bbs.task-handler.converge-tasks.converge-tasks.starting-convergence","log_level":1,"data":{"session":"7.5883.1"}} {"timestamp":"1453921686.495135784","source":"bbs","message":"bbs.converge-lrps.actual-lrp-schema-root-not-found","log_level":1,"data":{"session":"19954"}} {"timestamp":"1453921686.495184660","source":"bbs","message":"bbs.converge-lrps.succeeded-gathering-and-pruning-actual-lrps","log_level":1,"data":{"session":"19954"}} {"timestamp":"1453921686.495222807","source":"bbs","message":"bbs.converge-lrps.gathering-desired-lrps","log_level":1,"data":{"session":"19954"}} {"timestamp":"1453921686.499060631","source":"bbs","message":"bbs.converge-lrps.actual-lrp-schema-root-not-found","log_level":1,"data":{"session":"19954"}} {"timestamp":"1453921686.499403954","source":"bbs","message":"bbs.converge-lrps.succeeded-gathering-desired-lrps","log_level":1,"data":{"session":"19954"}} {"timestamp":"1453921686.501797915","source":"bbs","message":"bbs.task-handler.converge-tasks.converge-tasks.finished-convergence","log_level":1,"data":{"session":"7.5883.1"}} {"timestamp":"1453921686.502251863","source":"bbs","message":"bbs.request.done","log_level":1,"data":{"method":"POST","request":"/v1/tasks/converge","session":"19955"}} {"timestamp":"1453921686.504722357","source":"bbs","message":"bbs.converge-lrps.calculate-convergence.start","log_level":1,"data":{"session":"19954.3"}} {"timestamp":"1453921686.504907846","source":"bbs","message":"bbs.converge-lrps.calculate-convergence.done","log_level":1,"data":{"session":"19954.3"}} {"timestamp":"1453921686.505190134","source":"bbs","message":"bbs.converge-lrps.finished-convergence","log_level":1,"data":{"session":"19954"}} {"timestamp":"1453921686.505403280","source":"bbs","message":"bbs.request.done","log_level":1,"data":{"method":"POST","request":"/v1/lrps/converge","session":"19953"}} {"timestamp":"1453921687.169877052","source":"bbs","message":"bbs.request.serving","log_level":1,"data":{"method":"POST","request":"/v1/tasks/list","session":"19956"}} {"timestamp":"1453921687.171530724","source":"bbs","message":"bbs.request.done","log_level":1,"data":{"method":"POST","request":"/v1/tasks/list","session":"19956"}} {"timestamp":"1453921687.172345877","source":"bbs","message":"bbs.request.serving","log_level":1,"data":{"method":"POST","request":"/v1/actual_lrp_groups/list","session":"19957"}} {"timestamp":"1453921687.176663637","source":"bbs","message":"bbs.request.done","log_level":1,"data":{"method":"POST","request":"/v1/actual_lrp_groups/list","session":"19957"}} {"timestamp":"1453921692.995214939","source":"bbs","message":"bbs.request.serving","log_level":1,"data":{"method":"POST","request":"/v1/desired_lrp_scheduling_infos/list","session":"19958"}} {"timestamp":"1453921692.995497704","source":"bbs","message":"bbs.desiredlrp-handler.desired-lrps.desired-lrp-scheduling-infos.start","log_level":1,"data":{"filter":{"Domain":"cf-apps"},"session":"6.3793.1"}} {"timestamp":"1453921692.996849060","source":"bbs","message":"bbs.desiredlrp-handler.desired-lrps.desired-lrp-scheduling-infos.complete","log_level":1,"data":{"filter":{"Domain":"cf-apps"},"session":"6.3793.1"}} {"timestamp":"1453921692.997039795","source":"bbs","message":"bbs.request.done","log_level":1,"data":{"method":"POST","request":"/v1/desired_lrp_scheduling_infos/list","session":"19958"}} The information contained in this e-mail, and any attachment, is confidential and is intended solely for the use of the intended recipient. Access, copying or re-use of the e-mail or any attachment, or any information contained therein, by any other person is not authorized. If you are not the intended recipient please return the e-mail to the sender and delete it from your computer. Although we attempt to sweep e-mail and attachments for viruses, we do not guarantee that either are virus-free and accept no liability for any damage sustained as a result of viruses. Please refer to http://disclaimer.bnymellon.com/eu.htm for certain disclosures relating to European legal entities. |
|
|
|
Shetty, Daya <Daya.Shetty@...>
Hi, Eric,
As per your suggestion I changed the logLevel=debug and did a tail on the auctioneer & bbs logs while doing a cf push of buildpack based app to diego. Unfortunately I still saw only LogLevel: 1 messsages on the Auctioneer logs but did see LogLevel 0,1 & 2 messages on the bbs logs. I have attached the bbs.log. Here are some errors I see in the bbs log. {"timestamp":"1453934586.220547676","source":"bbs","message":"bbs.actuallrp-handler.actual-lrp-groups.etcd-error.resource-not-found","log_level":0,"data":{"error":{"errorCode":100,"message":"Key not found","cause":"/v1","index":5},"session":"3.19.1"}} {"timestamp":"1453934586.224381685","source":"bbs","message":"bbs.task-handler.tasks.etcd-error.resource-not-found","log_level":0,"data":{"error":{"errorCode":100,"message":"Key not found","cause":"/v1","index":5},"session":"7.25.1"}} "timestamp":"1453934595.548539162","source":"bbs","message":"bbs.desiredlrp-handler.remove-desired-lrp.remove-desired-lrp.failed","log_level":2,"data":{"error":"100: Key not found (/v1/desired_lrp) [8]","process-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-e42855a5-0122-4e1a-a003-a4eac1fe3deb","session":"6.19.1"}} {"timestamp":"1453934595.548731089","source":"bbs","message":"bbs.desiredlrp-handler.remove-desired-lrp.remove-desired-lrp.etcd-error.resource-not-found","log_level":0,"data":{"error":{"errorCode":100,"message":"Key not found","cause":"/v1/desired_lrp","index":8},"process-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-e42855a5-0122-4e1a-a003-a4eac1fe3deb","session":"6.19.1.1"}} {"timestamp":"1453934595.735231638","source":"bbs","message":"bbs.actuallrp-handler.actual-lrp-groups.etcd-error.resource-not-found","log_level":0,"data":{"error":{"errorCode":100,"message":"Key not found","cause":"/v1/actual","index":9},"session":"3.20.1"}} {"timestamp":"1453934595.738165140","source":"bbs","message":"bbs.desiredlrp-handler.desired-lrps.desired-lrp-scheduling-infos.etcd-error.resource-not-found","log_level":0,"data":{"error":{"errorCode":100,"message":"Key not found","cause":"/v1/desired_lrp","index":9},"filter":{"Domain":""},"session":"6.20.1.1"}} {"timestamp":"1453934600.773372650","source":"bbs","message":"bbs.desiredlrp-handler.desired-lrps.desired-lrp-scheduling-infos.etcd-error.resource-not-found","log_level":0,"data":{"error":{"errorCode":100,"message":"Key not found","cause":"/v1/desired_lrp","index":9},"filter":{"Domain":"cf-apps"},"session":"6.21.1.1"}} {"timestamp":"1453934610.201195955","source":"bbs","message":"bbs.converge-lrps.etcd-error.resource-not-found","log_level":0,"data":{"error":{"errorCode":100,"message":"Key not found","cause":"/v1/actual","index":9},"session":"111.1"}} {"timestamp":"1453934610.205472708","source":"bbs","message":"bbs.converge-lrps.etcd-error.resource-not-found","log_level":0,"data":{"error":{"errorCode":100,"message":"Key not found","cause":"/v1/desired_lrp","index":9},"session":"111.2"}} {"timestamp":"1453934616.234182835","source":"bbs","message":"bbs.actuallrp-handler.actual-lrp-groups.etcd-error.resource-not-found","log_level":0,"data":{"error":{"errorCode":100,"message":"Key not found","cause":"/v1/actual","index":9},"session":"3.21.1"}} Regards Daya From: Daya Shetty <Daya.Shetty(a)bnymellon.com<mailto:Daya.Shetty(a)bnymellon.com>> Reply-To: "Discussions about Cloud Foundry projects and the system overall." <cf-dev(a)lists.cloudfoundry.org<mailto:cf-dev(a)lists.cloudfoundry.org>> Date: Wednesday, January 27, 2016 at 2:00 PM To: "Discussions about Cloud Foundry projects and the system overall." <cf-dev(a)lists.cloudfoundry.org<mailto:cf-dev(a)lists.cloudfoundry.org>> Subject: [cf-dev] Re: Re: Re: Re: Re: Re: Re: Re: Diego IAAS settings for vsphere Hi, Eric, I tried to enable debug using the debugAddr but the log_level remained as 1. Not sure why? curl -v http://0.0.0.0:17001/log-level -X PUT -d debug * Hostname was NOT found in DNS cache * Trying 0.0.0.0... * Connected to 0.0.0.0 (127.0.0.1) port 17001 (#0) PUT /log-level HTTP/1.1 User-Agent: curl/7.35.0 Host: 0.0.0.0:17001 Accept: */* Content-Length: 5 Content-Type: application/x-www-form-urlencoded* upload completely sent off: 5 out of 5 bytes < HTTP/1.1 200 OK < Date: Wed, 27 Jan 2016 21:44:44 GMT < Content-Length: 0 < Content-Type: text/plain; charset=utf-8 < * Connection #0 to host 0.0.0.0 left intact var/vcap/packages/auctioneer/bin/auctioneer -bbsClientCert=/var/vcap/jobs/auctioneer/config/certs/bbs/client.crt -bbsClientKey=/var/vcap/jobs/auctioneer/config/certs/bbs/client.key -bbsCACert=/var/vcap/jobs/auctioneer/config/certs/bbs/ca.crt -bbsAddress=https://bbs.service.cf.internal:8889 -consulCluster=http://127.0.0.1:8500 -debugAddr=0.0.0.0:17001 -listenAddr=0.0.0.0:9016 -logLevel=info I will restart the processes mentioned, with —logLevel=debug by changing rhe relevant _ctl files, In the meantime here is the snapshot of the auctioneer log while trying to start a buildpack based app. Here is the auctioneer.stdout log root(a)fb99aa6d-3a9a-48c5-b6e9-dccfbc035c7a:/var/vcap/data/sys/log/auctioneer# !ta tail -f auctioneer.stdout.log {"timestamp":"1453931031.236279488","source":"auctioneer","message":"auctioneer.request.done","log_level":1,"data":{"method":"POST","request":"/v1/tasks","session":"36"}} {"timestamp":"1453931031.236993074","source":"auctioneer","message":"auctioneer.auction.fetching-cell-reps","log_level":1,"data":{"session":"37"}} {"timestamp":"1453931031.240644693","source":"auctioneer","message":"auctioneer.auction.fetched-cell-reps","log_level":1,"data":{"cell-reps-count":1,"session":"37"}} {"timestamp":"1453931031.240906715","source":"auctioneer","message":"auctioneer.auction.fetching-zone-state","log_level":1,"data":{"session":"37"}} {"timestamp":"1453931031.248399019","source":"auctioneer","message":"auctioneer.auction.zone-state","log_level":1,"data":{"cell-count":1,"session":"37","zone":"z1"}} {"timestamp":"1453931031.249227762","source":"auctioneer","message":"auctioneer.auction.fetched-zone-state","log_level":1,"data":{"cell-state-count":1,"duration":"6.511719ms","num-failed-requests":0,"session":"37"}} {"timestamp":"1453931031.249767065","source":"auctioneer","message":"auctioneer.auction.fetching-auctions","log_level":1,"data":{"session":"37"}} {"timestamp":"1453931031.250007153","source":"auctioneer","message":"auctioneer.auction.fetched-auctions","log_level":1,"data":{"lrp-start-auctions":0,"session":"37","task-auctions":1}} {"timestamp":"1453931031.250244141","source":"auctioneer","message":"auctioneer.auction.scheduling","log_level":1,"data":{"session":"37"}} {"timestamp":"1453931031.250724792","source":"auctioneer","message":"auctioneer.auction.scheduled","log_level":1,"data":{"failed-lrp-start-auctions":0,"failed-task-auctions":1,"session":"37","successful-lrp-start-auctions":0,"successful-task-auctions":0}} {"timestamp":"1453931106.945099831","source":"auctioneer","message":"auctioneer.request.serving","log_level":1,"data":{"method":"POST","request":"/v1/tasks","session":"38"}} {"timestamp":"1453931106.946578026","source":"auctioneer","message":"auctioneer.request.task-auction-handler.create.submitted","log_level":1,"data":{"method":"POST","request":"/v1/tasks","session":"38.1.1","tasks":["da383b70-f3da-4248-86b5-9a82e6d7d809-2d07956c22c142849c2b160c46ddb480"]}} {"timestamp":"1453931106.948045492","source":"auctioneer","message":"auctioneer.request.done","log_level":1,"data":{"method":"POST","request":"/v1/tasks","session":"38"}} {"timestamp":"1453931106.948791265","source":"auctioneer","message":"auctioneer.auction.fetching-cell-reps","log_level":1,"data":{"session":"39"}} {"timestamp":"1453931106.952327490","source":"auctioneer","message":"auctioneer.auction.fetched-cell-reps","log_level":1,"data":{"cell-reps-count":1,"session":"39"}} {"timestamp":"1453931106.952946186","source":"auctioneer","message":"auctioneer.auction.fetching-zone-state","log_level":1,"data":{"session":"39"}} {"timestamp":"1453931106.959112644","source":"auctioneer","message":"auctioneer.auction.zone-state","log_level":1,"data":{"cell-count":1,"session":"39","zone":"z1"}} {"timestamp":"1453931106.959350586","source":"auctioneer","message":"auctioneer.auction.fetched-zone-state","log_level":1,"data":{"cell-state-count":1,"duration":"5.455613ms","num-failed-requests":0,"session":"39"}} {"timestamp":"1453931106.959569216","source":"auctioneer","message":"auctioneer.auction.fetching-auctions","log_level":1,"data":{"session":"39"}} {"timestamp":"1453931106.959800005","source":"auctioneer","message":"auctioneer.auction.fetched-auctions","log_level":1,"data":{"lrp-start-auctions":0,"session":"39","task-auctions":1}} {"timestamp":"1453931106.960027695","source":"auctioneer","message":"auctioneer.auction.scheduling","log_level":1,"data":{"session":"39"}} {"timestamp":"1453931106.960274935","source":"auctioneer","message":"auctioneer.auction.scheduled","log_level":1,"data":{"failed-lrp-start-auctions":0,"failed-task-auctions":1,"session":"39","successful-lrp-start-auctions":0,"successful-task-auctions":0}} Regards Daya From: Eric Malm <emalm(a)pivotal.io<mailto:emalm(a)pivotal.io>> Reply-To: "Discussions about Cloud Foundry projects and the system overall." <cf-dev(a)lists.cloudfoundry.org<mailto:cf-dev(a)lists.cloudfoundry.org>> Date: Wednesday, January 27, 2016 at 12:29 PM To: "Discussions about Cloud Foundry projects and the system overall." <cf-dev(a)lists.cloudfoundry.org<mailto:cf-dev(a)lists.cloudfoundry.org>> Subject: [cf-dev] Re: Re: Re: Re: Re: Re: Re: Diego IAAS settings for vsphere Hi, Daya, Thanks for the log entries and the state response from the cell. It does look like that cell should have sufficient capacity to take on the staging task. From the BBS logs for the 'da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8' task guid, it does look like the task is getting saved in the BBS, and then failed by something else (presumably the auctioneer) almost immediately thereafter. Could you also look at the auctioneer logs for that period of time? Also, it might help to turn on debug logging dynamically on these components: if you do a `netstat -anp` on the bosh VM, you'll see each Diego component process listening on a port 170XX in the 17000 range. If you then run `curl http://localhost:170XX/log-level -X PUT -d debug` on that VM, that will change the log-level on that component to 'debug' (it normally defaults to 'info'). So if we're not seeing anything useful at the info/error level for this period of time, you could switch the active bbs and auctioneer to the debug log-level, start tailing their logs in real-time with `tail –f`, and try submitting another staging task. Thanks, Eric On Wed, Jan 27, 2016 at 11:31 AM, Daya Shetty <daya.shetty(a)bnymellon.com<mailto:daya.shetty(a)bnymellon.com>> wrote: Hi, Eric, I further checked the logs you mentioned and can't see what could be the problem. I only have one instance of cell running and here is the state info: curl 0.0.0.0:1800/state<http://0.0.0.0:1800/state> {"RootFSProviders":{"docker":{"type":"arbitrary"},"preloaded":{"set":{"cflinuxfs2":{}},"type":"fixed_set"}},"AvailableResources":{"MemoryMB":7984,"DiskMB":24263,"Containers":256},"TotalResources":{"MemoryMB":7984,"DiskMB":24263,"Containers":256},"LRPs":[],"Tasks":null,"Zone":"z1","Evacuating":false} I see TLS handshake errors in the bbs.stderr.log, don't know if this can be the cause for" Insufficient resources" 2016/01/27 19:12:03 http: TLS handshake error from 127.0.0.1:50475<http://127.0.0.1:50475>: EOF 2016/01/27 19:12:06 http: TLS handshake error from 127.0.0.1:50477<http://127.0.0.1:50477>: EOF Not able to deduce anything from the bbs.stdout.log here is a snapshot. Will dig further. "timestamp":"1453921657.161973238","source":"bbs","message":"bbs.request.serving","log_level":1,"data":{"method":"POST","request":"/v1/actual_lrp_groups/list","session":"19946"}} {"timestamp":"1453921657.162120819","source":"bbs","message":"bbs.request.serving","log_level":1,"data":{"method":"POST","request":"/v1/tasks/list","session":"19947"}} {"timestamp":"1453921657.167166233","source":"bbs","message":"bbs.request.done","log_level":1,"data":{"method":"POST","request":"/v1/tasks/list","session":"19947"}} {"timestamp":"1453921657.167504787","source":"bbs","message":"bbs.request.done","log_level":1,"data":{"method":"POST","request":"/v1/actual_lrp_groups/list","session":"19946"}} {"timestamp":"1453921674.952786684","source":"bbs","message":"bbs.request.serving","log_level":1,"data":{"method":"POST","request":"/v1/actual_lrp_groups/list","session":"19948"}} {"timestamp":"1453921674.953184366","source":"bbs","message":"bbs.request.serving","log_level":1,"data":{"method":"POST","request":"/v1/desired_lrp_scheduling_infos/list","session":"19949"}} {"timestamp":"1453921674.953428984","source":"bbs","message":"bbs.desiredlrp-handler.desired-lrps.desired-lrp-scheduling-infos.start","log_level":1,"data":{"filter":{"Domain":""},"session":"6.3791.1"}} {"timestamp":"1453921674.954186201","source":"bbs","message":"bbs.desiredlrp-handler.desired-lrps.desired-lrp-scheduling-infos.complete","log_level":1,"data":{"filter":{"Domain":""},"session":"6.3791.1"}} {"timestamp":"1453921674.954262018","source":"bbs","message":"bbs.request.done","log_level":1,"data":{"method":"POST","request":"/v1/desired_lrp_scheduling_infos/list","session":"19949"}} {"timestamp":"1453921674.959604502","source":"bbs","message":"bbs.request.done","log_level":1,"data":{"method":"POST","request":"/v1/actual_lrp_groups/list","session":"19948"}} {"timestamp":"1453921681.551017284","source":"bbs","message":"bbs.request.serving","log_level":1,"data":{"method":"POST","request":"/v1/tasks/desire","session":"19950"}} {"timestamp":"1453921681.551474571","source":"bbs","message":"bbs.task-handler.desire-task.desire-task.starting","log_level":1,"data":{"session":"7.5881.1","task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}} {"timestamp":"1453921681.561704636","source":"bbs","message":"bbs.task-handler.desire-task.desire-task.finished","log_level":1,"data":{"session":"7.5881.1","task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}} {"timestamp":"1453921681.561844826","source":"bbs","message":"bbs.request.done","log_level":1,"data":{"method":"POST","request":"/v1/tasks/desire","session":"19950"}} {"timestamp":"1453921681.575248480","source":"bbs","message":"bbs.request.serving","log_level":1,"data":{"method":"POST","request":"/v1/tasks/fail","session":"19951"}} {"timestamp":"1453921681.575376987","source":"bbs","message":"bbs.task-handler.fail-task.fail-task.starting","log_level":1,"data":{"session":"7.5882.1","task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}} {"timestamp":"1453921681.575446367","source":"bbs","message":"bbs.task-handler.fail-task.fail-task.getting-task","log_level":1,"data":{"session":"7.5882.1","task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}} {"timestamp":"1453921681.577617884","source":"bbs","message":"bbs.task-handler.fail-task.fail-task.succeeded-getting-task","log_level":1,"data":{"session":"7.5882.1","task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}} {"timestamp":"1453921681.578390121","source":"bbs","message":"bbs.task-handler.fail-task.fail-task.persisting-task","log_level":1,"data":{"session":"7.5882.1","task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}} {"timestamp":"1453921681.582517862","source":"bbs","message":"bbs.task-handler.fail-task.fail-task.succeded-persisting-task","log_level":1,"data":{"session":"7.5882.1","task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}} {"timestamp":"1453921681.583042145","source":"bbs","message":"bbs.task-handler.fail-task.fail-task.task-client-completing-task","log_level":1,"data":{"session":"7.5882.1","task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}} {"timestamp":"1453921681.583634138","source":"bbs","message":"bbs.task-handler.fail-task.fail-task.finished","log_level":1,"data":{"session":"7.5882.1","task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}} {"timestamp":"1453921681.583706856","source":"bbs","message":"bbs.resolving-task","log_level":1,"data":{"task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}} {"timestamp":"1453921681.584264040","source":"bbs","message":"bbs.resolving-task.starting","log_level":1,"data":{"session":"1","task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}} {"timestamp":"1453921681.584349394","source":"bbs","message":"bbs.resolving-task.getting-task","log_level":1,"data":{"session":"1","task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}} {"timestamp":"1453921681.584867001","source":"bbs","message":"bbs.request.done","log_level":1,"data":{"method":"POST","request":"/v1/tasks/fail","session":"19951"}} {"timestamp":"1453921681.587889433","source":"bbs","message":"bbs.resolving-task.succeeded-getting-task","log_level":1,"data":{"session":"1","task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}} {"timestamp":"1453921681.591948748","source":"bbs","message":"bbs.resolving-task.finished","log_level":1,"data":{"session":"1","task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}} {"timestamp":"1453921681.629187346","source":"bbs","message":"bbs.request.serving","log_level":1,"data":{"method":"POST","request":"/v1/desired_lrp/remove","session":"19952"}} {"timestamp":"1453921681.629388809","source":"bbs","message":"bbs.desiredlrp-handler.remove-desired-lrp.remove-desired-lrp.starting","log_level":1,"data":{"process-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-c398c85b-3965-4fd7-91e9-3112bafe3394","session":"6.3792.1"}} {"timestamp":"1453921681.630566120","source":"bbs","message":"bbs.desiredlrp-handler.remove-desired-lrp.remove-desired-lrp.failed","log_level":2,"data":{"error":"100: Key not found (/v1/desired_lrp) [64]","process-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-c398c85b-3965-4fd7-91e9-3112bafe3394","session":"6.3792.1"}} {"timestamp":"1453921681.631013870","source":"bbs","message":"bbs.desiredlrp-handler.remove-desired-lrp.remove-desired-lrp.complete","log_level":1,"data":{"process-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-c398c85b-3965-4fd7-91e9-3112bafe3394","session":"6.3792.1"}} {"timestamp":"1453921681.631548882","source":"bbs","message":"bbs.request.done","log_level":1,"data":{"method":"POST","request":"/v1/desired_lrp/remove","session":"19952"}} {"timestamp":"1453921681.639199495","source":"bbs","message":"bbs.delete-task.starting","log_level":1,"data":{"callback_url":"http://stager.service.cf.internal:8888/v1/staging/da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8/completed","session":"1","task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}} 5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8/completed","status_code":200,"task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}} {"timestamp":"1453921686.492143154","source":"bbs","message":"bbs.request.serving","log_level":1,"data":{"method":"POST","request":"/v1/lrps/converge","session":"19953"}} {"timestamp":"1453921686.492546797","source":"bbs","message":"bbs.converge-lrps.starting-convergence","log_level":1,"data":{"session":"19954"}} {"timestamp":"1453921686.492759466","source":"bbs","message":"bbs.converge-lrps.gathering-and-pruning-actual-lrps","log_level":1,"data":{"session":"19954"}} {"timestamp":"1453921686.493501425","source":"bbs","message":"bbs.request.serving","log_level":1,"data":{"method":"POST","request":"/v1/tasks/converge","session":"19955"}} {"timestamp":"1453921686.493688107","source":"bbs","message":"bbs.task-handler.converge-tasks.converge-tasks.starting-convergence","log_level":1,"data":{"session":"7.5883.1"}} {"timestamp":"1453921686.495135784","source":"bbs","message":"bbs.converge-lrps.actual-lrp-schema-root-not-found","log_level":1,"data":{"session":"19954"}} {"timestamp":"1453921686.495184660","source":"bbs","message":"bbs.converge-lrps.succeeded-gathering-and-pruning-actual-lrps","log_level":1,"data":{"session":"19954"}} {"timestamp":"1453921686.495222807","source":"bbs","message":"bbs.converge-lrps.gathering-desired-lrps","log_level":1,"data":{"session":"19954"}} {"timestamp":"1453921686.499060631","source":"bbs","message":"bbs.converge-lrps.actual-lrp-schema-root-not-found","log_level":1,"data":{"session":"19954"}} {"timestamp":"1453921686.499403954","source":"bbs","message":"bbs.converge-lrps.succeeded-gathering-desired-lrps","log_level":1,"data":{"session":"19954"}} {"timestamp":"1453921686.501797915","source":"bbs","message":"bbs.task-handler.converge-tasks.converge-tasks.finished-convergence","log_level":1,"data":{"session":"7.5883.1"}} {"timestamp":"1453921686.502251863","source":"bbs","message":"bbs.request.done","log_level":1,"data":{"method":"POST","request":"/v1/tasks/converge","session":"19955"}} {"timestamp":"1453921686.504722357","source":"bbs","message":"bbs.converge-lrps.calculate-convergence.start","log_level":1,"data":{"session":"19954.3"}} {"timestamp":"1453921686.504907846","source":"bbs","message":"bbs.converge-lrps.calculate-convergence.done","log_level":1,"data":{"session":"19954.3"}} {"timestamp":"1453921686.505190134","source":"bbs","message":"bbs.converge-lrps.finished-convergence","log_level":1,"data":{"session":"19954"}} {"timestamp":"1453921686.505403280","source":"bbs","message":"bbs.request.done","log_level":1,"data":{"method":"POST","request":"/v1/lrps/converge","session":"19953"}} {"timestamp":"1453921687.169877052","source":"bbs","message":"bbs.request.serving","log_level":1,"data":{"method":"POST","request":"/v1/tasks/list","session":"19956"}} {"timestamp":"1453921687.171530724","source":"bbs","message":"bbs.request.done","log_level":1,"data":{"method":"POST","request":"/v1/tasks/list","session":"19956"}} {"timestamp":"1453921687.172345877","source":"bbs","message":"bbs.request.serving","log_level":1,"data":{"method":"POST","request":"/v1/actual_lrp_groups/list","session":"19957"}} {"timestamp":"1453921687.176663637","source":"bbs","message":"bbs.request.done","log_level":1,"data":{"method":"POST","request":"/v1/actual_lrp_groups/list","session":"19957"}} {"timestamp":"1453921692.995214939","source":"bbs","message":"bbs.request.serving","log_level":1,"data":{"method":"POST","request":"/v1/desired_lrp_scheduling_infos/list","session":"19958"}} {"timestamp":"1453921692.995497704","source":"bbs","message":"bbs.desiredlrp-handler.desired-lrps.desired-lrp-scheduling-infos.start","log_level":1,"data":{"filter":{"Domain":"cf-apps"},"session":"6.3793.1"}} {"timestamp":"1453921692.996849060","source":"bbs","message":"bbs.desiredlrp-handler.desired-lrps.desired-lrp-scheduling-infos.complete","log_level":1,"data":{"filter":{"Domain":"cf-apps"},"session":"6.3793.1"}} {"timestamp":"1453921692.997039795","source":"bbs","message":"bbs.request.done","log_level":1,"data":{"method":"POST","request":"/v1/desired_lrp_scheduling_infos/list","session":"19958"}} The information contained in this e-mail, and any attachment, is confidential and is intended solely for the use of the intended recipient. Access, copying or re-use of the e-mail or any attachment, or any information contained therein, by any other person is not authorized. If you are not the intended recipient please return the e-mail to the sender and delete it from your computer. Although we attempt to sweep e-mail and attachments for viruses, we do not guarantee that either are virus-free and accept no liability for any damage sustained as a result of viruses. Please refer to http://disclaimer.bnymellon.com/eu.htm for certain disclosures relating to European legal entities. The information contained in this e-mail, and any attachment, is confidential and is intended solely for the use of the intended recipient. Access, copying or re-use of the e-mail or any attachment, or any information contained therein, by any other person is not authorized. If you are not the intended recipient please return the e-mail to the sender and delete it from your computer. Although we attempt to sweep e-mail and attachments for viruses, we do not guarantee that either are virus-free and accept no liability for any damage sustained as a result of viruses. Please refer to http://disclaimer.bnymellon.com/eu.htm for certain disclosures relating to European legal entities. |
|
|
|
Eric Malm <emalm@...>
Hi, Daya,
Thanks very much for the additional logs. Those BBS errors look normal (requests often bounce off nonexistent records), and from the auctioneer logs the auction process is retrieving state from your one cell correctly. From the adjacent 'auctioneer.auction.scheduling' and 'auctioneer.auction.scheduled' log lines at the end, though, it does look like the auction is failing to place the desired task on the only available cell. Since those log lines are less than a millisecond apart, it's extremely unlikely that the auctioneer is telling the cell to take on the task and the cell is refusing (although we could also look for that interaction in the cell rep logs). Unfortunately, we intentionally don't have a lot of logging in the code that runs the placement algorithm, so it may be impossible in this version of the codebase to introspect the specific decisions it's making about task placement. One other pathway to try would be to move an existing app from the DEAs to the Diego backend via the Diego-Enabler CLI plugin ( https://github.com/cloudfoundry-incubator/Diego-Enabler), and see if its corresponding LRP is placed correctly. The placement decision should be based entirely on matching the LRP's rootfs to the cell and then validating the cell has enough capacity, just as it is for Tasks. It may be easier to compare (and adjust through CC) the resource requirements of the app's DesiredLRP that it would be on the more ephemeral Task submitted for staging. In any case, if you haven't already resolved the placement problems, please try out moving a test app over and let us know how that turns out. Best, Eric On Wed, Jan 27, 2016 at 3:37 PM, Shetty, Daya <Daya.Shetty(a)bnymellon.com> wrote: Hi, Eric, |
|
|
|
Shetty, Daya <Daya.Shetty@...>
Hi, Eric,
toggle quoted message
Show quoted text
I did manage to successfully push both a buildpack app and a docker app on Diego. I still did not find time to revisit this and make sure that this was a reason for the Insufficient Resource problem but here is what I noticed and changed to make it work. Our CF instance dea_next.staging_memory_limit_mb was defined as 8G, so I increased the diego cell memory from 8G to 16G and after redeploying diego I was able to successfully push these apps. I don’t know whether this makes sense (I.e. dea staging parameter being used to stage apps on diego) but until I can revert and test again, cannot say for sure. Thanks again for all your help! Regards Daya From: Eric Malm <emalm(a)pivotal.io<mailto:emalm(a)pivotal.io>> Reply-To: "Discussions about Cloud Foundry projects and the system overall." <cf-dev(a)lists.cloudfoundry.org<mailto:cf-dev(a)lists.cloudfoundry.org>> Date: Tuesday, February 2, 2016 at 6:46 PM To: "Discussions about Cloud Foundry projects and the system overall." <cf-dev(a)lists.cloudfoundry.org<mailto:cf-dev(a)lists.cloudfoundry.org>> Subject: [cf-dev] Re: Re: Re: Re: Re: Re: Re: Re: Re: Re: Diego IAAS settings for vsphere Hi, Daya, Thanks very much for the additional logs. Those BBS errors look normal (requests often bounce off nonexistent records), and from the auctioneer logs the auction process is retrieving state from your one cell correctly. From the adjacent 'auctioneer.auction.scheduling' and 'auctioneer.auction.scheduled' log lines at the end, though, it does look like the auction is failing to place the desired task on the only available cell. Since those log lines are less than a millisecond apart, it's extremely unlikely that the auctioneer is telling the cell to take on the task and the cell is refusing (although we could also look for that interaction in the cell rep logs). Unfortunately, we intentionally don't have a lot of logging in the code that runs the placement algorithm, so it may be impossible in this version of the codebase to introspect the specific decisions it's making about task placement. One other pathway to try would be to move an existing app from the DEAs to the Diego backend via the Diego-Enabler CLI plugin (https://github.com/cloudfoundry-incubator/Diego-Enabler), and see if its corresponding LRP is placed correctly. The placement decision should be based entirely on matching the LRP's rootfs to the cell and then validating the cell has enough capacity, just as it is for Tasks. It may be easier to compare (and adjust through CC) the resource requirements of the app's DesiredLRP that it would be on the more ephemeral Task submitted for staging. In any case, if you haven't already resolved the placement problems, please try out moving a test app over and let us know how that turns out. Best, Eric On Wed, Jan 27, 2016 at 3:37 PM, Shetty, Daya <Daya.Shetty(a)bnymellon.com<mailto:Daya.Shetty(a)bnymellon.com>> wrote:
Hi, Eric, As per your suggestion I changed the logLevel=debug and did a tail on the auctioneer & bbs logs while doing a cf push of buildpack based app to diego. Unfortunately I still saw only LogLevel: 1 messsages on the Auctioneer logs but did see LogLevel 0,1 & 2 messages on the bbs logs. I have attached the bbs.log. Here are some errors I see in the bbs log. {"timestamp":"1453934586.220547676","source":"bbs","message":"bbs.actuallrp-handler.actual-lrp-groups.etcd-error.resource-not-found","log_level":0,"data":{"error":{"errorCode":100,"message":"Key not found","cause":"/v1","index":5},"session":"3.19.1"}} {"timestamp":"1453934586.224381685","source":"bbs","message":"bbs.task-handler.tasks.etcd-error.resource-not-found","log_level":0,"data":{"error":{"errorCode":100,"message":"Key not found","cause":"/v1","index":5},"session":"7.25.1"}} "timestamp":"1453934595.548539162","source":"bbs","message":"bbs.desiredlrp-handler.remove-desired-lrp.remove-desired-lrp.failed","log_level":2,"data":{"error":"100: Key not found (/v1/desired_lrp) [8]","process-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-e42855a5-0122-4e1a-a003-a4eac1fe3deb","session":"6.19.1"}} {"timestamp":"1453934595.548731089","source":"bbs","message":"bbs.desiredlrp-handler.remove-desired-lrp.remove-desired-lrp.etcd-error.resource-not-found","log_level":0,"data":{"error":{"errorCode":100,"message":"Key not found","cause":"/v1/desired_lrp","index":8},"process-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-e42855a5-0122-4e1a-a003-a4eac1fe3deb","session":"6.19.1.1"}} {"timestamp":"1453934595.735231638","source":"bbs","message":"bbs.actuallrp-handler.actual-lrp-groups.etcd-error.resource-not-found","log_level":0,"data":{"error":{"errorCode":100,"message":"Key not found","cause":"/v1/actual","index":9},"session":"3.20.1"}} {"timestamp":"1453934595.738165140","source":"bbs","message":"bbs.desiredlrp-handler.desired-lrps.desired-lrp-scheduling-infos.etcd-error.resource-not-found","log_level":0,"data":{"error":{"errorCode":100,"message":"Key not found","cause":"/v1/desired_lrp","index":9},"filter":{"Domain":""},"session":"6.20.1.1"}} {"timestamp":"1453934600.773372650","source":"bbs","message":"bbs.desiredlrp-handler.desired-lrps.desired-lrp-scheduling-infos.etcd-error.resource-not-found","log_level":0,"data":{"error":{"errorCode":100,"message":"Key not found","cause":"/v1/desired_lrp","index":9},"filter":{"Domain":"cf-apps"},"session":"6.21.1.1"}} {"timestamp":"1453934610.201195955","source":"bbs","message":"bbs.converge-lrps.etcd-error.resource-not-found","log_level":0,"data":{"error":{"errorCode":100,"message":"Key not found","cause":"/v1/actual","index":9},"session":"111.1"}} {"timestamp":"1453934610.205472708","source":"bbs","message":"bbs.converge-lrps.etcd-error.resource-not-found","log_level":0,"data":{"error":{"errorCode":100,"message":"Key not found","cause":"/v1/desired_lrp","index":9},"session":"111.2"}} {"timestamp":"1453934616.234182835","source":"bbs","message":"bbs.actuallrp-handler.actual-lrp-groups.etcd-error.resource-not-found","log_level":0,"data":{"error":{"errorCode":100,"message":"Key not found","cause":"/v1/actual","index":9},"session":"3.21.1"}} Regards Daya From: Daya Shetty <Daya.Shetty(a)bnymellon.com<mailto:Daya.Shetty(a)bnymellon.com>> Reply-To: "Discussions about Cloud Foundry projects and the system overall." <cf-dev(a)lists.cloudfoundry.org<mailto:cf-dev(a)lists.cloudfoundry.org>> Date: Wednesday, January 27, 2016 at 2:00 PM To: "Discussions about Cloud Foundry projects and the system overall." <cf-dev(a)lists.cloudfoundry.org<mailto:cf-dev(a)lists.cloudfoundry.org>> Subject: [cf-dev] Re: Re: Re: Re: Re: Re: Re: Re: Diego IAAS settings for vsphere Hi, Eric, I tried to enable debug using the debugAddr but the log_level remained as 1. Not sure why? curl -v http://0.0.0.0:17001/log-level -X PUT -d debug * Hostname was NOT found in DNS cache * Trying 0.0.0.0... * Connected to 0.0.0.0 (127.0.0.1) port 17001 (#0) PUT /log-level HTTP/1.1 User-Agent: curl/7.35.0 Host: 0.0.0.0:17001<http://0.0.0.0:17001> Accept: */* Content-Length: 5 Content-Type: application/x-www-form-urlencoded* upload completely sent off: 5 out of 5 bytes < HTTP/1.1 200 OK < Date: Wed, 27 Jan 2016 21:44:44 GMT < Content-Length: 0 < Content-Type: text/plain; charset=utf-8 < * Connection #0 to host 0.0.0.0 left intact var/vcap/packages/auctioneer/bin/auctioneer -bbsClientCert=/var/vcap/jobs/auctioneer/config/certs/bbs/client.crt -bbsClientKey=/var/vcap/jobs/auctioneer/config/certs/bbs/client.key -bbsCACert=/var/vcap/jobs/auctioneer/config/certs/bbs/ca.crt -bbsAddress=https://bbs.service.cf.internal:8889 -consulCluster=http://127.0.0.1:8500 -debugAddr=0.0.0.0:17001<http://0.0.0.0:17001> -listenAddr=0.0.0.0:9016<http://0.0.0.0:9016> -logLevel=info I will restart the processes mentioned, with —logLevel=debug by changing rhe relevant _ctl files, In the meantime here is the snapshot of the auctioneer log while trying to start a buildpack based app. Here is the auctioneer.stdout log root(a)fb99aa6d-3a9a-48c5-b6e9-dccfbc035c7a:/var/vcap/data/sys/log/auctioneer# !ta tail -f auctioneer.stdout.log {"timestamp":"1453931031.236279488","source":"auctioneer","message":"auctioneer.request.done","log_level":1,"data":{"method":"POST","request":"/v1/tasks","session":"36"}} {"timestamp":"1453931031.236993074","source":"auctioneer","message":"auctioneer.auction.fetching-cell-reps","log_level":1,"data":{"session":"37"}} {"timestamp":"1453931031.240644693","source":"auctioneer","message":"auctioneer.auction.fetched-cell-reps","log_level":1,"data":{"cell-reps-count":1,"session":"37"}} {"timestamp":"1453931031.240906715","source":"auctioneer","message":"auctioneer.auction.fetching-zone-state","log_level":1,"data":{"session":"37"}} {"timestamp":"1453931031.248399019","source":"auctioneer","message":"auctioneer.auction.zone-state","log_level":1,"data":{"cell-count":1,"session":"37","zone":"z1"}} {"timestamp":"1453931031.249227762","source":"auctioneer","message":"auctioneer.auction.fetched-zone-state","log_level":1,"data":{"cell-state-count":1,"duration":"6.511719ms","num-failed-requests":0,"session":"37"}} {"timestamp":"1453931031.249767065","source":"auctioneer","message":"auctioneer.auction.fetching-auctions","log_level":1,"data":{"session":"37"}} {"timestamp":"1453931031.250007153","source":"auctioneer","message":"auctioneer.auction.fetched-auctions","log_level":1,"data":{"lrp-start-auctions":0,"session":"37","task-auctions":1}} {"timestamp":"1453931031.250244141","source":"auctioneer","message":"auctioneer.auction.scheduling","log_level":1,"data":{"session":"37"}} {"timestamp":"1453931031.250724792","source":"auctioneer","message":"auctioneer.auction.scheduled","log_level":1,"data":{"failed-lrp-start-auctions":0,"failed-task-auctions":1,"session":"37","successful-lrp-start-auctions":0,"successful-task-auctions":0}} {"timestamp":"1453931106.945099831","source":"auctioneer","message":"auctioneer.request.serving","log_level":1,"data":{"method":"POST","request":"/v1/tasks","session":"38"}} {"timestamp":"1453931106.946578026","source":"auctioneer","message":"auctioneer.request.task-auction-handler.create.submitted","log_level":1,"data":{"method":"POST","request":"/v1/tasks","session":"38.1.1","tasks":["da383b70-f3da-4248-86b5-9a82e6d7d809-2d07956c22c142849c2b160c46ddb480"]}} {"timestamp":"1453931106.948045492","source":"auctioneer","message":"auctioneer.request.done","log_level":1,"data":{"method":"POST","request":"/v1/tasks","session":"38"}} {"timestamp":"1453931106.948791265","source":"auctioneer","message":"auctioneer.auction.fetching-cell-reps","log_level":1,"data":{"session":"39"}} {"timestamp":"1453931106.952327490","source":"auctioneer","message":"auctioneer.auction.fetched-cell-reps","log_level":1,"data":{"cell-reps-count":1,"session":"39"}} {"timestamp":"1453931106.952946186","source":"auctioneer","message":"auctioneer.auction.fetching-zone-state","log_level":1,"data":{"session":"39"}} {"timestamp":"1453931106.959112644","source":"auctioneer","message":"auctioneer.auction.zone-state","log_level":1,"data":{"cell-count":1,"session":"39","zone":"z1"}} {"timestamp":"1453931106.959350586","source":"auctioneer","message":"auctioneer.auction.fetched-zone-state","log_level":1,"data":{"cell-state-count":1,"duration":"5.455613ms","num-failed-requests":0,"session":"39"}} {"timestamp":"1453931106.959569216","source":"auctioneer","message":"auctioneer.auction.fetching-auctions","log_level":1,"data":{"session":"39"}} {"timestamp":"1453931106.959800005","source":"auctioneer","message":"auctioneer.auction.fetched-auctions","log_level":1,"data":{"lrp-start-auctions":0,"session":"39","task-auctions":1}} {"timestamp":"1453931106.960027695","source":"auctioneer","message":"auctioneer.auction.scheduling","log_level":1,"data":{"session":"39"}} {"timestamp":"1453931106.960274935","source":"auctioneer","message":"auctioneer.auction.scheduled","log_level":1,"data":{"failed-lrp-start-auctions":0,"failed-task-auctions":1,"session":"39","successful-lrp-start-auctions":0,"successful-task-auctions":0}} Regards Daya From: Eric Malm <emalm(a)pivotal.io<mailto:emalm(a)pivotal.io>> Reply-To: "Discussions about Cloud Foundry projects and the system overall." <cf-dev(a)lists.cloudfoundry.org<mailto:cf-dev(a)lists.cloudfoundry.org>> Date: Wednesday, January 27, 2016 at 12:29 PM To: "Discussions about Cloud Foundry projects and the system overall." <cf-dev(a)lists.cloudfoundry.org<mailto:cf-dev(a)lists.cloudfoundry.org>> Subject: [cf-dev] Re: Re: Re: Re: Re: Re: Re: Diego IAAS settings for vsphere Hi, Daya, Thanks for the log entries and the state response from the cell. It does look like that cell should have sufficient capacity to take on the staging task. From the BBS logs for the 'da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8' task guid, it does look like the task is getting saved in the BBS, and then failed by something else (presumably the auctioneer) almost immediately thereafter. Could you also look at the auctioneer logs for that period of time? Also, it might help to turn on debug logging dynamically on these components: if you do a `netstat -anp` on the bosh VM, you'll see each Diego component process listening on a port 170XX in the 17000 range. If you then run `curl http://localhost:170XX/log-level -X PUT -d debug` on that VM, that will change the log-level on that component to 'debug' (it normally defaults to 'info'). So if we're not seeing anything useful at the info/error level for this period of time, you could switch the active bbs and auctioneer to the debug log-level, start tailing their logs in real-time with `tail –f`, and try submitting another staging task. Thanks, Eric On Wed, Jan 27, 2016 at 11:31 AM, Daya Shetty <daya.shetty(a)bnymellon.com<mailto:daya.shetty(a)bnymellon.com>> wrote: Hi, Eric, I further checked the logs you mentioned and can't see what could be the problem. I only have one instance of cell running and here is the state info: curl 0.0.0.0:1800/state<http://0.0.0.0:1800/state> {"RootFSProviders":{"docker":{"type":"arbitrary"},"preloaded":{"set":{"cflinuxfs2":{}},"type":"fixed_set"}},"AvailableResources":{"MemoryMB":7984,"DiskMB":24263,"Containers":256},"TotalResources":{"MemoryMB":7984,"DiskMB":24263,"Containers":256},"LRPs":[],"Tasks":null,"Zone":"z1","Evacuating":false} I see TLS handshake errors in the bbs.stderr.log, don't know if this can be the cause for" Insufficient resources" 2016/01/27 19:12:03 http: TLS handshake error from 127.0.0.1:50475<http://127.0.0.1:50475>: EOF 2016/01/27 19:12:06 http: TLS handshake error from 127.0.0.1:50477<http://127.0.0.1:50477>: EOF Not able to deduce anything from the bbs.stdout.log here is a snapshot. Will dig further. "timestamp":"1453921657.161973238","source":"bbs","message":"bbs.request.serving","log_level":1,"data":{"method":"POST","request":"/v1/actual_lrp_groups/list","session":"19946"}} {"timestamp":"1453921657.162120819","source":"bbs","message":"bbs.request.serving","log_level":1,"data":{"method":"POST","request":"/v1/tasks/list","session":"19947"}} {"timestamp":"1453921657.167166233","source":"bbs","message":"bbs.request.done","log_level":1,"data":{"method":"POST","request":"/v1/tasks/list","session":"19947"}} {"timestamp":"1453921657.167504787","source":"bbs","message":"bbs.request.done","log_level":1,"data":{"method":"POST","request":"/v1/actual_lrp_groups/list","session":"19946"}} {"timestamp":"1453921674.952786684","source":"bbs","message":"bbs.request.serving","log_level":1,"data":{"method":"POST","request":"/v1/actual_lrp_groups/list","session":"19948"}} {"timestamp":"1453921674.953184366","source":"bbs","message":"bbs.request.serving","log_level":1,"data":{"method":"POST","request":"/v1/desired_lrp_scheduling_infos/list","session":"19949"}} {"timestamp":"1453921674.953428984","source":"bbs","message":"bbs.desiredlrp-handler.desired-lrps.desired-lrp-scheduling-infos.start","log_level":1,"data":{"filter":{"Domain":""},"session":"6.3791.1"}} {"timestamp":"1453921674.954186201","source":"bbs","message":"bbs.desiredlrp-handler.desired-lrps.desired-lrp-scheduling-infos.complete","log_level":1,"data":{"filter":{"Domain":""},"session":"6.3791.1"}} {"timestamp":"1453921674.954262018","source":"bbs","message":"bbs.request.done","log_level":1,"data":{"method":"POST","request":"/v1/desired_lrp_scheduling_infos/list","session":"19949"}} {"timestamp":"1453921674.959604502","source":"bbs","message":"bbs.request.done","log_level":1,"data":{"method":"POST","request":"/v1/actual_lrp_groups/list","session":"19948"}} {"timestamp":"1453921681.551017284","source":"bbs","message":"bbs.request.serving","log_level":1,"data":{"method":"POST","request":"/v1/tasks/desire","session":"19950"}} {"timestamp":"1453921681.551474571","source":"bbs","message":"bbs.task-handler.desire-task.desire-task.starting","log_level":1,"data":{"session":"7.5881.1","task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}} {"timestamp":"1453921681.561704636","source":"bbs","message":"bbs.task-handler.desire-task.desire-task.finished","log_level":1,"data":{"session":"7.5881.1","task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}} {"timestamp":"1453921681.561844826","source":"bbs","message":"bbs.request.done","log_level":1,"data":{"method":"POST","request":"/v1/tasks/desire","session":"19950"}} {"timestamp":"1453921681.575248480","source":"bbs","message":"bbs.request.serving","log_level":1,"data":{"method":"POST","request":"/v1/tasks/fail","session":"19951"}} {"timestamp":"1453921681.575376987","source":"bbs","message":"bbs.task-handler.fail-task.fail-task.starting","log_level":1,"data":{"session":"7.5882.1","task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}} {"timestamp":"1453921681.575446367","source":"bbs","message":"bbs.task-handler.fail-task.fail-task.getting-task","log_level":1,"data":{"session":"7.5882.1","task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}} {"timestamp":"1453921681.577617884","source":"bbs","message":"bbs.task-handler.fail-task.fail-task.succeeded-getting-task","log_level":1,"data":{"session":"7.5882.1","task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}} {"timestamp":"1453921681.578390121","source":"bbs","message":"bbs.task-handler.fail-task.fail-task.persisting-task","log_level":1,"data":{"session":"7.5882.1","task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}} {"timestamp":"1453921681.582517862","source":"bbs","message":"bbs.task-handler.fail-task.fail-task.succeded-persisting-task","log_level":1,"data":{"session":"7.5882.1","task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}} {"timestamp":"1453921681.583042145","source":"bbs","message":"bbs.task-handler.fail-task.fail-task.task-client-completing-task","log_level":1,"data":{"session":"7.5882.1","task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}} {"timestamp":"1453921681.583634138","source":"bbs","message":"bbs.task-handler.fail-task.fail-task.finished","log_level":1,"data":{"session":"7.5882.1","task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}} {"timestamp":"1453921681.583706856","source":"bbs","message":"bbs.resolving-task","log_level":1,"data":{"task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}} {"timestamp":"1453921681.584264040","source":"bbs","message":"bbs.resolving-task.starting","log_level":1,"data":{"session":"1","task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}} {"timestamp":"1453921681.584349394","source":"bbs","message":"bbs.resolving-task.getting-task","log_level":1,"data":{"session":"1","task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}} {"timestamp":"1453921681.584867001","source":"bbs","message":"bbs.request.done","log_level":1,"data":{"method":"POST","request":"/v1/tasks/fail","session":"19951"}} {"timestamp":"1453921681.587889433","source":"bbs","message":"bbs.resolving-task.succeeded-getting-task","log_level":1,"data":{"session":"1","task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}} {"timestamp":"1453921681.591948748","source":"bbs","message":"bbs.resolving-task.finished","log_level":1,"data":{"session":"1","task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}} {"timestamp":"1453921681.629187346","source":"bbs","message":"bbs.request.serving","log_level":1,"data":{"method":"POST","request":"/v1/desired_lrp/remove","session":"19952"}} {"timestamp":"1453921681.629388809","source":"bbs","message":"bbs.desiredlrp-handler.remove-desired-lrp.remove-desired-lrp.starting","log_level":1,"data":{"process-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-c398c85b-3965-4fd7-91e9-3112bafe3394","session":"6.3792.1"}} {"timestamp":"1453921681.630566120","source":"bbs","message":"bbs.desiredlrp-handler.remove-desired-lrp.remove-desired-lrp.failed","log_level":2,"data":{"error":"100: Key not found (/v1/desired_lrp) [64]","process-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-c398c85b-3965-4fd7-91e9-3112bafe3394","session":"6.3792.1"}} {"timestamp":"1453921681.631013870","source":"bbs","message":"bbs.desiredlrp-handler.remove-desired-lrp.remove-desired-lrp.complete","log_level":1,"data":{"process-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-c398c85b-3965-4fd7-91e9-3112bafe3394","session":"6.3792.1"}} {"timestamp":"1453921681.631548882","source":"bbs","message":"bbs.request.done","log_level":1,"data":{"method":"POST","request":"/v1/desired_lrp/remove","session":"19952"}} {"timestamp":"1453921681.639199495","source":"bbs","message":"bbs.delete-task.starting","log_level":1,"data":{"callback_url":"http://stager.service.cf.internal:8888/v1/staging/da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8/completed","session":"1","task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}} 5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8/completed","status_code":200,"task-guid":"da383b70-f3da-4248-86b5-9a82e6d7d809-6db1e598fc2f466287a8f29c01a71bd8"}} {"timestamp":"1453921686.492143154","source":"bbs","message":"bbs.request.serving","log_level":1,"data":{"method":"POST","request":"/v1/lrps/converge","session":"19953"}} {"timestamp":"1453921686.492546797","source":"bbs","message":"bbs.converge-lrps.starting-convergence","log_level":1,"data":{"session":"19954"}} {"timestamp":"1453921686.492759466","source":"bbs","message":"bbs.converge-lrps.gathering-and-pruning-actual-lrps","log_level":1,"data":{"session":"19954"}} {"timestamp":"1453921686.493501425","source":"bbs","message":"bbs.request.serving","log_level":1,"data":{"method":"POST","request":"/v1/tasks/converge","session":"19955"}} {"timestamp":"1453921686.493688107","source":"bbs","message":"bbs.task-handler.converge-tasks.converge-tasks.starting-convergence","log_level":1,"data":{"session":"7.5883.1"}} {"timestamp":"1453921686.495135784","source":"bbs","message":"bbs.converge-lrps.actual-lrp-schema-root-not-found","log_level":1,"data":{"session":"19954"}} {"timestamp":"1453921686.495184660","source":"bbs","message":"bbs.converge-lrps.succeeded-gathering-and-pruning-actual-lrps","log_level":1,"data":{"session":"19954"}} {"timestamp":"1453921686.495222807","source":"bbs","message":"bbs.converge-lrps.gathering-desired-lrps","log_level":1,"data":{"session":"19954"}} {"timestamp":"1453921686.499060631","source":"bbs","message":"bbs.converge-lrps.actual-lrp-schema-root-not-found","log_level":1,"data":{"session":"19954"}} {"timestamp":"1453921686.499403954","source":"bbs","message":"bbs.converge-lrps.succeeded-gathering-desired-lrps","log_level":1,"data":{"session":"19954"}} {"timestamp":"1453921686.501797915","source":"bbs","message":"bbs.task-handler.converge-tasks.converge-tasks.finished-convergence","log_level":1,"data":{"session":"7.5883.1"}} {"timestamp":"1453921686.502251863","source":"bbs","message":"bbs.request.done","log_level":1,"data":{"method":"POST","request":"/v1/tasks/converge","session":"19955"}} {"timestamp":"1453921686.504722357","source":"bbs","message":"bbs.converge-lrps.calculate-convergence.start","log_level":1,"data":{"session":"19954.3"}} {"timestamp":"1453921686.504907846","source":"bbs","message":"bbs.converge-lrps.calculate-convergence.done","log_level":1,"data":{"session":"19954.3"}} {"timestamp":"1453921686.505190134","source":"bbs","message":"bbs.converge-lrps.finished-convergence","log_level":1,"data":{"session":"19954"}} {"timestamp":"1453921686.505403280","source":"bbs","message":"bbs.request.done","log_level":1,"data":{"method":"POST","request":"/v1/lrps/converge","session":"19953"}} {"timestamp":"1453921687.169877052","source":"bbs","message":"bbs.request.serving","log_level":1,"data":{"method":"POST","request":"/v1/tasks/list","session":"19956"}} {"timestamp":"1453921687.171530724","source":"bbs","message":"bbs.request.done","log_level":1,"data":{"method":"POST","request":"/v1/tasks/list","session":"19956"}} {"timestamp":"1453921687.172345877","source":"bbs","message":"bbs.request.serving","log_level":1,"data":{"method":"POST","request":"/v1/actual_lrp_groups/list","session":"19957"}} {"timestamp":"1453921687.176663637","source":"bbs","message":"bbs.request.done","log_level":1,"data":{"method":"POST","request":"/v1/actual_lrp_groups/list","session":"19957"}} {"timestamp":"1453921692.995214939","source":"bbs","message":"bbs.request.serving","log_level":1,"data":{"method":"POST","request":"/v1/desired_lrp_scheduling_infos/list","session":"19958"}} {"timestamp":"1453921692.995497704","source":"bbs","message":"bbs.desiredlrp-handler.desired-lrps.desired-lrp-scheduling-infos.start","log_level":1,"data":{"filter":{"Domain":"cf-apps"},"session":"6.3793.1"}} {"timestamp":"1453921692.996849060","source":"bbs","message":"bbs.desiredlrp-handler.desired-lrps.desired-lrp-scheduling-infos.complete","log_level":1,"data":{"filter":{"Domain":"cf-apps"},"session":"6.3793.1"}} {"timestamp":"1453921692.997039795","source":"bbs","message":"bbs.request.done","log_level":1,"data":{"method":"POST","request":"/v1/desired_lrp_scheduling_infos/list","session":"19958"}} The information contained in this e-mail, and any attachment, is confidential and is intended solely for the use of the intended recipient. Access, copying or re-use of the e-mail or any attachment, or any information contained therein, by any other person is not authorized. If you are not the intended recipient please return the e-mail to the sender and delete it from your computer. Although we attempt to sweep e-mail and attachments for viruses, we do not guarantee that either are virus-free and accept no liability for any damage sustained as a result of viruses. Please refer to http://disclaimer.bnymellon.com/eu.htm for certain disclosures relating to European legal entities. The information contained in this e-mail, and any attachment, is confidential and is intended solely for the use of the intended recipient. Access, copying or re-use of the e-mail or any attachment, or any information contained therein, by any other person is not authorized. If you are not the intended recipient please return the e-mail to the sender and delete it from your computer. Although we attempt to sweep e-mail and attachments for viruses, we do not guarantee that either are virus-free and accept no liability for any damage sustained as a result of viruses. Please refer to http://disclaimer.bnymellon.com/eu.htm for certain disclosures relating to European legal entities. The information contained in this e-mail, and any attachment, is confidential and is intended solely for the use of the intended recipient. Access, copying or re-use of the e-mail or any attachment, or any information contained therein, by any other person is not authorized. If you are not the intended recipient please return the e-mail to the sender and delete it from your computer. Although we attempt to sweep e-mail and attachments for viruses, we do not guarantee that either are virus-free and accept no liability for any damage sustained as a result of viruses. Please refer to http://disclaimer.bnymellon.com/eu.htm for certain disclosures relating to European legal entities. |
|
|
|
Eric Malm <emalm@...>
Hi, Daya,
That's most likely the problem: if you have a staging task using that 8G = 8192M memory limit, it wouldn't be able to fit on the cell that's advertising only 7984M of available memory. I think that the `staging_memory_limit_mb` property is namespaced under `dea_next` for historical reasons, and it's now more closely associated with CC's config. In any case, from the CC config wiring and how that's interpreted in the CC's DEA and Diego clients, it looks like that property determines the minimum amount of memory allocated to a staging task. If the app requires more memory than that, that app-specific memory limit is used instead. Anyway, glad to hear that solved the problem! I'll keep this in mind as an operator-debugging use-case that we on the Diego team would like to build tooling to help inspect and resolve. Best, Eric On Tue, Feb 9, 2016 at 11:30 AM, Shetty, Daya <Daya.Shetty(a)bnymellon.com> wrote: Hi, Eric, |
|
|