Re: CF Auto-scaling with an external application
@Daniel Mikusa Thank you for your support. I have few more questions and hope that you can help me. If i'll use the library you linked, i'd like to have some suggestions about the task i have to solve: - is there a way to collect in some way the data i need in a struct that i could use for send "the scale command"? I mean, would be great to have a struct in wich i could have fields like "AppName", "CpuUse", "MemoryUse", etc. so i cloud just check the App.CpuUse field, for istance, and send a "cf scale" command to CF to solve resource problems. - i found this client library for CF, do you think could help my work? https://github.com/cloudfoundry/cf-java-client/ I'm asking you cause i'll prefer to use Java, by the way. Is you know, is there a way to have ".jar" library of this repo so i could use as simple as i can? Also i found this http://www.ibm.com/developerworks/cloud/library/cl-bluemix-cloudappswithjava/index.html and a "cloudfoundry-client-lib.jar" online but i don't kwon if could be good for my problem. I would really thank you for your support, and i'm sorry if i made some stupid or specifical question but i really need help. The app i have to develop must be very simple because it is just a mimal study case for the PaaS Cloud Foundry so the easier it is the best would be. Still thank you, Giovanni.
|
|
Serge Kornfeld <serge.kornfeld@...>
Unsubscribe
toggle quoted messageShow quoted text
On Fri, Apr 8, 2016 at 11:51 AM, A Gonzalez <apolonioguy(a)gmail.com> wrote:
|
|
Re: Cloud Controller System Domain vs App Domains
Hi Tom, App developers can currently push an app using api.domain, but that is the problem we are trying to solve. As this is currently allowed, the Gorouter will balance traffic between the Cloud Controller and the user app. This is not desired, all traffic for api.system_domain needs to go to the Cloud Controller. At this point we are going to move forward with Solution 2. We think this will offer the easiest transition for operators. Thanks, Nick On Wed, Apr 6, 2016 at 5:27 PM Tom Sherrod <tom.sherrod(a)gmail.com> wrote: Would solution 2 allow/work with a system domain? Solution #2 means api.domain is now taken so there could be no other app deployed to api.domain. Application developers may not be happy with the limitation, requiring another domain. Defeats the purpose of solution #2?
Tom
On Tue, Apr 5, 2016 at 2:15 PM, Nicholas Calugar <ncalugar(a)pivotal.io> wrote:
Hi John,
The point of seeding is so that there aren't two decisions (code paths) for requests to create a route. Seeding the database requires no changes to route creation logic.
Would anyone else like to comment on either of the two proposals before we make a decision and start implementing a fix?
Thanks,
Nick
On Thu, Mar 31, 2016 at 10:52 AM john mcteague <john.mcteague(a)gmail.com> wrote:
For option 2 would it not be simpler to have a single property such as cc.blacklisted_system_domain_routes that contains the desired list and have the CC deny route requests for those routes?
I don't see what storing them in the DB or creating real routes actually buys us here. Everything would be available via config.
I'm in favour of some form of option 2 for those of us who have existing deployments but would rather not make a change in either app or system domains but are exposed to this issue.
John. On 31 Mar 2016 5:47 a.m., "Nicholas Calugar" <ncalugar(a)pivotal.io> wrote:
Hi Cloud Foundry,
We've had a recurring issue brought to our attention regarding a Cloud Foundry deployment using a system_domain that is in the list of app_domains. When the system domain is in the list of app domains, a Shared Domain is created for the system domain. This is problematic because it allows users to create routes on the system domain, see this [1] recent issue as an example.
[1] https://github.com/cloudfoundry/cloud_controller_ng/issues/568
I'd like to propose two solutions and get some feedback regarding which the community would prefer. Please respond with your preferred solution and a brief reason why.
*Solution 1 - Require a Unique System Domain*
Instead of recommending a unique system domain, we would enforce this in the Cloud Controller. The proposed change is as follows:
1. REQUIRE system_domain to NOT be in the list of app_domains 2. REQUIRE a system_domain_organization
This will create a Private Domain for the system domain. Failure to configure correctly would not allow the Cloud Controller to start.
If we decide to implement this, an operator should ensure their deployment uses a unique system domain and a system_domain_organization and correct DNS entries before upgrading.
Example for BOSH-lite
- app_domains: [bosh-lite.com] - system_domain: system.bosh-lite.com - system_domain_organization: system
- api endpoint: api.system.bosh-lite.com - sample app endpoint: dora.bosh-lite.com
Example for a PaaS:
- app_domains: [yuge-paas-apps.io] - system_domain: yuge-pass.com - system_domain_organization: yuge-system-org
- api endpoint: api.yuge-paas.com - sample app endpoint: dora.yuge-paas-apps.io
*Pro: *Cloud Controller now enforces what was previously the recommended configuration for separate system and apps domains. Con: Second SSL cert for the system domain and possibly a second DNS record if system domain is not covered by the current wildcard record.
*Solution 2 - Cloud Controller Seeds System Routes*
To prevent a non-system app from requesting a hostname on a shared system domain, the Cloud Controller will take a list of hostnames and seed the database with routes. As routes are associated with a space, we will require a system_organization and system_space. An operator could choose to omit hostnames as desired.
cc.system_hostnames: description: List of hostnames for which routes will be created on the system domain. default: [api,uaa,login,doppler,loggregator,hm9000] cc.system_space: description: Space where system routes will be created. default: system
*Pro:* Significantly less change for operators running Cloud Foundry with matching system and apps domains. *Con: *Cloud Controller has knowledge of unrelated system components and the list of defaults needs to be maintained as we add and remove components.
Thanks,
-Nick -- Nicholas Calugar CAPI Product Manager Pivotal Software, Inc.
-- Nicholas Calugar CAPI Product Manager Pivotal Software, Inc.
-- Nicholas Calugar CAPI Product Manager Pivotal Software, Inc.
|
|
Re: App running even after delete. Pointers on finding it and debugging?
Thanks, Tom, glad you were able to use veritas to find and remove the stray apps. I'd like to know how they remained present in the first place. Do you have logs from the nsync-bulker jobs on the cc_bridge VMs in your deployment? That BOSH job has the responsibility of updating the Diego DesiredLRPs to match the current set of CF apps, so if there are synchronization errors they should be present in those logs.
Thanks, Eric, CF Runtime Diego PM
toggle quoted messageShow quoted text
On Fri, Apr 8, 2016 at 8:09 AM, Kris Hicks <khicks(a)pivotal.io> wrote: It would be nice to figure out the root cause here.
Does having two crashed and two apps have some significance as to why the delete failed, though appeared successful?
On Friday, April 8, 2016, Tom Sherrod <tom.sherrod(a)gmail.com> wrote:
Thank you.
Veritas is quite informative. I found 2 apps running and 2 crashed. I deleted them and all appears well.
On Mon, Apr 4, 2016 at 7:03 PM, Amit Gupta <agupta(a)pivotal.io> wrote:
Ok, I would use veritas to look at the Diego BBS, and confirm that it still thinks the app is there. You can also go onto the router and query its HTTP endpoint to confirm that the route you're seeing is also still there: https://github.com/cloudfoundry/gorouter#instrumentation. Lastly I would connect to the CCDB and confirm that the app and route are *not* there. This will reduce the problem to figuring out why Diego isn't being updated to know that the non-existing app is no longer desired.
On Mon, Apr 4, 2016 at 3:47 PM, Tom Sherrod <tom.sherrod(a)gmail.com> wrote:
The route still exists. I was reluctant to delete it and have the "app" still running. I wanted some way to track it down, not that it has helped, other than let me know it is still running.
Pushed the app, with a different name/host, with no problems and it runs as it should.
On Mon, Apr 4, 2016 at 6:17 PM, Amit Gupta <agupta(a)pivotal.io> wrote:
Tom,
So you're saying that none of the org/spaces shows the app or the route, but the app continues to run and be routeable?
I could imagine this happen if some CC Bridge components are not able to talk to either CC or Diego BBS, leaving the data in the Diego BBS stale. In the case of stale info, Diego may not know that the LRP is no longer desired, and it will do the safe thing of keeping it around, and emitting its route to the gorouter, which just does what it's told (it doesn't check whether CC knows about the route or not).
Are you able to push new apps or delete other apps with the Diego backend?
Amit
On Fri, Apr 1, 2016 at 1:00 PM, Tom Sherrod <tom.sherrod(a)gmail.com> wrote:
JT,
Thanks for responding.
This is a test runtime and small. I checked all orgs and spaces. No routes matching the app.
Found the route information and the result:
{
"total_results": 0,
"total_pages": 1,
"prev_url": null,
"next_url": null,
"resources": []
}
To learn what the output may look like, I check existing routes with apps and without. The output appears to be the same as if the app has been deleted.
Even now, the app url still returns a page from the app, even though it is deleted.
Thanks,
Tom
On Fri, Apr 1, 2016 at 1:52 PM, JT Archie <jarchie(a)pivotal.io> wrote:
Tom,
Are you sure the route isn't bound to another application in another org/space?
When you do `cf routes` it only show routes for the current space. You can hit specific API endpoints though to get all the apps for a route.
For example, `cf curl /v2/routes/89fc2a5e-3a9b-4a88-a360-e405cdbd6f87/apps` will show all the apps for a particular route. Obviously replacing the route ID with the correct ID. To find that, I recommend going through `CF_TRACE=true cf routes` and grabbing the ID.
Let see if you can hunt it down that way.
Kind Regards,
JT
On Fri, Apr 1, 2016 at 8:51 AM, Tom Sherrod <tom.sherrod(a)gmail.com> wrote:
cf 230, diego 0.1450.0, etcd 27, garden-linux 0.330.0 Default to diego true.
Developer deployed a java application. Deleted the application: cf delete <app> No errors. The app still responds. The only thing left is the route. I've not encountered this before. Delete has been delete and even if route remains, 404 Not Found: Requested route ('<hostname.domain>') does not exist. is returned.
Pointers on tracking this down appreciated.
Tom
|
|
Re: App running even after delete. Pointers on finding it and debugging?
It would be nice to figure out the root cause here.
Does having two crashed and two apps have some significance as to why the delete failed, though appeared successful?
toggle quoted messageShow quoted text
On Friday, April 8, 2016, Tom Sherrod <tom.sherrod(a)gmail.com> wrote: Thank you.
Veritas is quite informative. I found 2 apps running and 2 crashed. I deleted them and all appears well.
On Mon, Apr 4, 2016 at 7:03 PM, Amit Gupta <agupta(a)pivotal.io <javascript:_e(%7B%7D,'cvml','agupta(a)pivotal.io');>> wrote:
Ok, I would use veritas to look at the Diego BBS, and confirm that it still thinks the app is there. You can also go onto the router and query its HTTP endpoint to confirm that the route you're seeing is also still there: https://github.com/cloudfoundry/gorouter#instrumentation. Lastly I would connect to the CCDB and confirm that the app and route are *not* there. This will reduce the problem to figuring out why Diego isn't being updated to know that the non-existing app is no longer desired.
On Mon, Apr 4, 2016 at 3:47 PM, Tom Sherrod <tom.sherrod(a)gmail.com <javascript:_e(%7B%7D,'cvml','tom.sherrod(a)gmail.com');>> wrote:
The route still exists. I was reluctant to delete it and have the "app" still running. I wanted some way to track it down, not that it has helped, other than let me know it is still running.
Pushed the app, with a different name/host, with no problems and it runs as it should.
On Mon, Apr 4, 2016 at 6:17 PM, Amit Gupta <agupta(a)pivotal.io <javascript:_e(%7B%7D,'cvml','agupta(a)pivotal.io');>> wrote:
Tom,
So you're saying that none of the org/spaces shows the app or the route, but the app continues to run and be routeable?
I could imagine this happen if some CC Bridge components are not able to talk to either CC or Diego BBS, leaving the data in the Diego BBS stale. In the case of stale info, Diego may not know that the LRP is no longer desired, and it will do the safe thing of keeping it around, and emitting its route to the gorouter, which just does what it's told (it doesn't check whether CC knows about the route or not).
Are you able to push new apps or delete other apps with the Diego backend?
Amit
On Fri, Apr 1, 2016 at 1:00 PM, Tom Sherrod <tom.sherrod(a)gmail.com <javascript:_e(%7B%7D,'cvml','tom.sherrod(a)gmail.com');>> wrote:
JT,
Thanks for responding.
This is a test runtime and small. I checked all orgs and spaces. No routes matching the app.
Found the route information and the result:
{
"total_results": 0,
"total_pages": 1,
"prev_url": null,
"next_url": null,
"resources": []
}
To learn what the output may look like, I check existing routes with apps and without. The output appears to be the same as if the app has been deleted.
Even now, the app url still returns a page from the app, even though it is deleted.
Thanks,
Tom
On Fri, Apr 1, 2016 at 1:52 PM, JT Archie <jarchie(a)pivotal.io <javascript:_e(%7B%7D,'cvml','jarchie(a)pivotal.io');>> wrote:
Tom,
Are you sure the route isn't bound to another application in another org/space?
When you do `cf routes` it only show routes for the current space. You can hit specific API endpoints though to get all the apps for a route.
For example, `cf curl /v2/routes/89fc2a5e-3a9b-4a88-a360-e405cdbd6f87/apps` will show all the apps for a particular route. Obviously replacing the route ID with the correct ID. To find that, I recommend going through `CF_TRACE=true cf routes` and grabbing the ID.
Let see if you can hunt it down that way.
Kind Regards,
JT
On Fri, Apr 1, 2016 at 8:51 AM, Tom Sherrod <tom.sherrod(a)gmail.com <javascript:_e(%7B%7D,'cvml','tom.sherrod(a)gmail.com');>> wrote:
cf 230, diego 0.1450.0, etcd 27, garden-linux 0.330.0 Default to diego true.
Developer deployed a java application. Deleted the application: cf delete <app> No errors. The app still responds. The only thing left is the route. I've not encountered this before. Delete has been delete and even if route remains, 404 Not Found: Requested route ('<hostname.domain>') does not exist. is returned.
Pointers on tracking this down appreciated.
Tom
|
|
Re: App running even after delete. Pointers on finding it and debugging?
Tom Sherrod <tom.sherrod@...>
Thank you.
Veritas is quite informative. I found 2 apps running and 2 crashed. I deleted them and all appears well.
toggle quoted messageShow quoted text
On Mon, Apr 4, 2016 at 7:03 PM, Amit Gupta <agupta(a)pivotal.io> wrote: Ok, I would use veritas to look at the Diego BBS, and confirm that it still thinks the app is there. You can also go onto the router and query its HTTP endpoint to confirm that the route you're seeing is also still there: https://github.com/cloudfoundry/gorouter#instrumentation. Lastly I would connect to the CCDB and confirm that the app and route are *not* there. This will reduce the problem to figuring out why Diego isn't being updated to know that the non-existing app is no longer desired.
On Mon, Apr 4, 2016 at 3:47 PM, Tom Sherrod <tom.sherrod(a)gmail.com> wrote:
The route still exists. I was reluctant to delete it and have the "app" still running. I wanted some way to track it down, not that it has helped, other than let me know it is still running.
Pushed the app, with a different name/host, with no problems and it runs as it should.
On Mon, Apr 4, 2016 at 6:17 PM, Amit Gupta <agupta(a)pivotal.io> wrote:
Tom,
So you're saying that none of the org/spaces shows the app or the route, but the app continues to run and be routeable?
I could imagine this happen if some CC Bridge components are not able to talk to either CC or Diego BBS, leaving the data in the Diego BBS stale. In the case of stale info, Diego may not know that the LRP is no longer desired, and it will do the safe thing of keeping it around, and emitting its route to the gorouter, which just does what it's told (it doesn't check whether CC knows about the route or not).
Are you able to push new apps or delete other apps with the Diego backend?
Amit
On Fri, Apr 1, 2016 at 1:00 PM, Tom Sherrod <tom.sherrod(a)gmail.com> wrote:
JT,
Thanks for responding.
This is a test runtime and small. I checked all orgs and spaces. No routes matching the app.
Found the route information and the result:
{
"total_results": 0,
"total_pages": 1,
"prev_url": null,
"next_url": null,
"resources": []
}
To learn what the output may look like, I check existing routes with apps and without. The output appears to be the same as if the app has been deleted.
Even now, the app url still returns a page from the app, even though it is deleted.
Thanks,
Tom
On Fri, Apr 1, 2016 at 1:52 PM, JT Archie <jarchie(a)pivotal.io> wrote:
Tom,
Are you sure the route isn't bound to another application in another org/space?
When you do `cf routes` it only show routes for the current space. You can hit specific API endpoints though to get all the apps for a route.
For example, `cf curl /v2/routes/89fc2a5e-3a9b-4a88-a360-e405cdbd6f87/apps` will show all the apps for a particular route. Obviously replacing the route ID with the correct ID. To find that, I recommend going through `CF_TRACE=true cf routes` and grabbing the ID.
Let see if you can hunt it down that way.
Kind Regards,
JT
On Fri, Apr 1, 2016 at 8:51 AM, Tom Sherrod <tom.sherrod(a)gmail.com> wrote:
cf 230, diego 0.1450.0, etcd 27, garden-linux 0.330.0 Default to diego true.
Developer deployed a java application. Deleted the application: cf delete <app> No errors. The app still responds. The only thing left is the route. I've not encountered this before. Delete has been delete and even if route remains, 404 Not Found: Requested route ('<hostname.domain>') does not exist. is returned.
Pointers on tracking this down appreciated.
Tom
|
|
Re: Static IP setup for routers on AWS
Engelke, Johannes <info@...>
Hi Amit, thanks for your answer. I deployed cloud foundry without using static IP’s. It is working well.
As far as I understood the uaa config the entire 10.x.x.x network is allowed to access the UAA Servers anyway, so there is no reason to place the dedicated static IP's of the routers into the config.
Do you see any security improvements, if only routers are allowed to access the UAA?
Best, Johannes
toggle quoted messageShow quoted text
On 08 Apr 2016, at 02:19, Amit Gupta <agupta(a)pivotal.io> wrote:
The UAA needs to know the router IPs to know which IPs to accept inbound requests from. If you don't care about this, you can try configuring UAA to allow requests from many IPs, and remove the static IPs from gorouter. I would be interested to find out the result of this experiment should you try it out.
Best, Amit
On Thu, Apr 7, 2016 at 6:28 AM, Engelke, Johannes <info(a)johannes-engelke.de <mailto:info(a)johannes-engelke.de>> wrote: Hi, does anybody know, why the routers got static ips in the cf-infrastructure-aws.yml file? https://github.com/cloudfoundry/cf-release/blob/master/templates/cf-infrastructure-aws.yml#L173 <https://github.com/cloudfoundry/cf-release/blob/master/templates/cf-infrastructure-aws.yml#L173>
Bosh is assigning the instances to ELB’s during deploy time, so there should be no need to have static addresses here.
If nobody know’s a good reason should we remove them ;-)
Cheers Johannes
|
|
Re: [Metron Agent] failed to generate job templates with metron agent on top of OpenStack Dynamic network
I have the same issue. It has to do with every release since bosh 248. However, dynamic networks with older bosh releases + cf-231/cf-231 work.
This must be a bug.
Ben R
toggle quoted messageShow quoted text
On Thu, Apr 7, 2016 at 8:55 PM, Yitao Jiang <jiangyt.cn(a)gmail.com> wrote: Hi,guys
When deploy CF on top of OpenStack with dynamic network, the jobs failed with metron-agent Error filling in template 'syslog_forwarder.conf.erb' (line 44: undefined method `strip' for nil:NilClass)
here's related logs
D etecting deployment changes ---------------------------- Releases cf version type changed: String -> Fixnum - 233 + 233
Compilation No changes
Update ± canaries: - 1 + 0
Resource pools No changes
Disk pools No changes
Networks dynamic-net + name: dynamic-net subnets 10.0.0.0/24 cloud_properties + net_id: 0700ae03-4b38-464e-b40d-0a9c8dd18ff0 + security_groups: ["Test OS SG_20160128T070152Z"] + dns: ["114.114.114.114", "8.8.8.8"]
+ range: 10.0.0.0/24 + name: Test OS Sub Internal Network_20160128T070152Z
+ type: dynamic
Jobs stats_z1
± networks:
- {"name"=>"cf1"}
+ {"name"=>"dynamic-net"}
Properties No changes
Meta No changes
Please review all changes carefully
Deploying ---------
Are you sure you want to deploy? (type 'yes' to continue): yes
Director task 57 Started preparing deployment > Preparing deployment. Done (00:00:03)
Error 100: Unable to render instance groups for deployment. Errors are: - Unable to render jobs for instance group 'stats_z1'. Errors are:
- Unable to render templates for job 'metron_agent'. Errors are:
- Error filling in template 'syslog_forwarder.conf.erb' (line 44:
undefined method `strip' for nil:NilClass)
Task 57 error
For a more detailed error report, run: bosh task 57 --debug as the ip manged by OpenStack, bosh cannot get the actual ip address of each vm until vm alive, this lead to the generated job spec doesn't contain ip address infos so, must i have to configure network type to manual?
snippets of deployment yml
1001 - name: dynamic-net 1002 subnets: 1003 - cloud_properties: 1004 net_id: 0700ae03-4b38-464e-b40d-0a9c8dd18ff0 1005 security_groups: 1006 - Test OS SG_20160128T070152Z 1007 dns: 1010 - 114.114.114.114 1011 - 8.8.8.8 1012 range: 10.0.0.0/24 1013 name: Test OS Sub Internal Network_20160128T070152Z 1014 type: dynamic
Rendered job spec
{"deployment"=>"staging-01", "job"=
{"name"=>"stats_z1", "templates"=>[{"name"=>"collector", "version"=>"6c210292f18d129e9a037fe7053836db2d494344", "sha1"=>"38927f47b15c2daf6c8a2e7c760e73e5ff90 dfd4", "blobstore_id"=>"23531029-0ee1-4267-8863-b5f931afaecb"}, {"name"=>"metron_agent", "version"=>"2b80a211127fc642fc8bb0d14d7eb30c37730db3", "sha1"=>"150f2 7445c2ef960951c1f26606525d41ec629b2", "blobstore_id"=>"e87174dc-f3f7-4768-94cd-74f299813528"}], "template"=>"collector", "version"=>"6c210292f18d129e9a037fe70 53836db2d494344", "sha1"=>"38927f47b15c2daf6c8a2e7c760e73e5ff90dfd4", "blobstore_id"=>"23531029-0ee1-4267-8863-b5f931afaecb"}, "index"=>0, "bootstrap"=>true, "name"=>"stats_z1", "id"=>"99f349d0-fb5d-4de7-9912-3de5559d2f19", "az"=>nil,
*"networks"=>{"dynamic-net"=>{"type"=>"dynamic", "cloud_properties"=>{"net_id"=>"0700ae03-4b38-464e-b40d-0a9c8dd18ff0", "security_groups"=>["Test OS SG_20160128T070152Z"]}, "dns"=>["114.114.114.114", "8.8.8.8", "10.0.0.13"], "default"=>["dns", "gateway"], "dns_record_name"=>"0.stats-z1.dynamic-net.staging-01.microbosh"}}*, "properties"=>{"collector"=>{"aws"=>{ "access_key_id"=>nil, "secret_access_key"=>nil}, "datadog"=>{"api_key"=>nil, "application_key"=>nil}, "deployment_name"=>nil, "logging_level"=>"info", "interv als"=>{"discover"=>60, "healthz"=>30, "local_metrics"=>30, "nats_ping"=>30, "prune"=>300, "varz"=>30}, "use_aws_cloudwatch"=>false, "use_datadog"=>false, "use _tsdb"=>false, "opentsdb"=>{"address"=>nil, "port"=>nil}, "use_graphite"=>false, "graphite"=>{"address"=>nil, "port"=>nil}, "memory_threshold"=>800}, "nats"=> {"machines"=>["10.0.0.127"], "password"=>"NATS_PASSWORD", "port"=>4222, "user"=>"NATS_USER"}, "syslog_daemon_config"=>{"address"=>nil, "port"=>nil, "transport "=>"tcp", "fallback_addresses"=>[], "custom_rule"=>"", "max_message_size"=>"4k"}, "metron_agent"=>{"dropsonde_incoming_port"=>3457, "preferred_protocol"=>"udp ", "tls"=>{"client_cert"=>"", "client_key"=>""}, "debug"=>false, "zone"=>"z1", "deployment"=>"ya-staging-01", "tcp"=>{"batching_buffer_bytes"=>10240, "batchin g_buffer_flush_interval_milliseconds"=>100}, "logrotate"=>{"freq_min"=>5, "rotate"=>7, "size"=>"50M"}, "buffer_size"=>10000, "enable_buffer"=>false}, "metron_ endpoint"=>{"shared_secret"=>"LOGGREGATOR_ENDPOINT_SHARED_SECRET"}, "loggregator"=>{"tls"=>{"ca_cert"=>""}, "dropsonde_incoming_port"=>3457, "etcd"=>{"machine s"=>["10.0.0.133"], "maxconcurrentrequests"=>10}}}, "dns_domain_name"=>"microbosh", "links"=>{}, "address"=>"99f349d0-fb5d-4de7-9912-3de5559d2f19.stats-z1.dyn amic-net.ya-staging-01.microbosh", "persistent_disk"=>0, "resource_pool"=>"small_z1"} --
Regards,
Yitao
|
|
Re: cf v233 api_z1/api_z2 failing
HI Kara, Thanks. Let me try with a valid app_domain. On Thu, Apr 7, 2016 at 9:01 PM, Kara Alexandra <ardnaxelarak(a)gmail.com> wrote: Hi Ranga,
The only reason we were using bosh-lite.com for our app_domains was because we were testing to reproduce on our local bosh-lite. Using 'cfapp' I managed to reproduce the issue locally. My guess is that this is because 'cfapp' is not a valid domain (it doesn't end with a valid top-level domain), and I'm guessing that fixing that will at least fix part of the problem if not all of it.
Thanks,
Kara
On Thu, Apr 7, 2016 at 8:27 PM, Ranga Rajagopalan < ranga.rajagopalan(a)gmail.com> wrote:
Here's my deployment manifest. app_domains is set to cfapp. I can't find bost-lite anywhere in the file at all.
On Thu, Apr 7, 2016 at 4:27 PM, Peter Goetz <peter.gtz(a)gmail.com> wrote:
Hi Ranga,
Looking at your logs we found an error that could possibly cause this and it is related to the properties.apps_domain in the deployment manifest. By setting it to 'b%()osh-lite.com' (using special characters), we could reproduce the following error which we also found in your log file:
Encountered error: name format\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/sequel-4.29.0/lib/sequel/model/base.rb:1543:in `save'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/app/models/runtime/shared_domain.rb:35:in `block in find_or_create'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/sequel-4.29.0/lib/sequel/database/transactions.rb:134:in `_transaction'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/sequel-4.29.0/lib/sequel/database/transactions.rb:108:in `block in transaction'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/sequel-4.29.0/lib/sequel/database/connecting.rb:249:in `block in synchronize'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/sequel-4.29.0/lib/sequel/connection_pool/threaded.rb:103:in `hold'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/sequel-4.29.0/lib/sequel/database/connecting.rb:249:in `synchronize'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/sequel-4.29.0/lib/sequel/database/transactions.rb:97:in `transaction'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/app/models/runtime/shared_domain.rb:27:in `find_or_create'\n/var/vcap/data/packages/cloud_controller_ng/da452b34d79be56a0784c7e88d6b9c0e1811a9d8.1-f934c136e6019cf54ab7aa04a3c153657226e729/cloud_controller_ng/lib/cloud_controller/seeds.rb:57:in `block in create_seed_domains'\n/var/vcap/data/packages/cloud_controller_ng/da452b34d79be56a0784c7e88d6b9c0e1811a9d8.1-f934c136e6019cf54ab7aa04a3c153657226e729/cloud_controller_ng/lib/cloud_controller/seeds.rb:56:in `each'\n/var/vcap/data/packages/cloud_controller_ng/da452b34d79be56a0784c7e88d6b9c0e1811a9d8.1-f934c136e6019cf54ab7aa04a3c153657226e729/cloud_controller_ng/lib/cloud_controller/seeds.rb:56:in `create_seed_domains'\n/var/vcap/data/packages/cloud_controller_ng/da452b34d79be56a0784c7e88d6b9c0e1811a9d8.1-f934c136e6019cf54ab7aa04a3c153657226e729/cloud_controller_ng/lib/cloud_controller/seeds.rb:9:in `write_seed_data'\n/var/vcap/data/packages/cloud_controller_ng/da452b34d79be56a0784c7e88d6b9c0e1811a9d8.1-f934c136e6019cf54ab7aa04a3c153657226e729/cloud_controller_ng/lib/cloud_controller/runner.rb:93:in `block in run!'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/eventmachine-1.0.9.1/lib/eventmachine.rb:193:in `call'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/eventmachine-1.0.9.1/lib/eventmachine.rb:193:in `run_machine'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/eventmachine-1.0.9.1/lib/eventmachine.rb:193:in `run'\n/var/vcap/data/packages/cloud_controller_ng/da452b34d79be56a0784c7e88d6b9c0e1811a9d8.1-f934c136e6019cf54ab7aa04a3c153657226e729/cloud_controller_ng/lib/cloud_controller/runner.rb:87:in `run!'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/bin/cloud_controller:8:in `<main>'
Can you check the apps_domain property and see if there is anything suspicious with it?
Thanks, Peter & Kara
On Thu, Apr 7, 2016 at 2:32 PM Ranga Rajagopalan < ranga.rajagopalan(a)gmail.com> wrote:
Hi Peter,
Attaching /var/vcap/sys/log/cloud_controller_worker/cloud_controller_worker.log.gz and /var/vcap/sys/log/cloud_controller_worker_ctl.log.gz. There isn't a /var/vcap/sys/log/cloud_controller_worker/ directory on this node.
vcap(a)572afc33-0735-4727-8ff3-9dc6d7fa8af0:~$ ls /var/vcap/sys/log/ agent_ctl.err.log cloud_controller_worker_ctl.log nginx_cc/ agent_ctl.log consul_agent/ nginx_ctl.err.log cloud_controller_migration/ metron_agent/ nginx_ctl.log cloud_controller_migration_ctl.err.log metron_agent_ctl.err.log route_registrar/ cloud_controller_migration_ctl.log metron_agent_ctl.log route_registrar_ctl.err.log cloud_controller_ng/ monit/ route_registrar_ctl.log cloud_controller_ng_ctl.err.log nfs_mounter/ statsd-injector/ cloud_controller_ng_ctl.log nfs_mounter_ctl.err.log statsd-injector-ctl.err.log cloud_controller_worker_ctl.err.log nfs_mounter_ctl.log statsd-injector-ctl.log
On Thu, Apr 7, 2016 at 12:21 PM, Peter Goetz <peter.gtz(a)gmail.com> wrote:
Hi Ranga,
To trouble-shoot this issue could you also provide the contents of /var/vcap/sys/log/cloud_controller_ng/cloud_controller_ng.log and /var/vcap/sys/log/cloud_controller_worker/cloud_controller_worker.log? This should give us more details about what's going on. The ctl script logs do not provide enough details.
Thanks, Peter
On Wed, Apr 6, 2016 at 6:12 PM Ranga Rajagopalan < ranga.rajagopalan(a)gmail.com> wrote:
I tried v231. Unfortunately, same issue.
-- Thanks,
Ranga
-- Thanks,
Ranga
-- Thanks, Ranga
|
|
Re: cf v233 api_z1/api_z2 failing
Kara Alexandra <ardnaxelarak@...>
Hi Ranga, The only reason we were using bosh-lite.com for our app_domains was because we were testing to reproduce on our local bosh-lite. Using 'cfapp' I managed to reproduce the issue locally. My guess is that this is because 'cfapp' is not a valid domain (it doesn't end with a valid top-level domain), and I'm guessing that fixing that will at least fix part of the problem if not all of it. Thanks, Kara On Thu, Apr 7, 2016 at 8:27 PM, Ranga Rajagopalan < ranga.rajagopalan(a)gmail.com> wrote: Here's my deployment manifest. app_domains is set to cfapp. I can't find bost-lite anywhere in the file at all.
On Thu, Apr 7, 2016 at 4:27 PM, Peter Goetz <peter.gtz(a)gmail.com> wrote:
Hi Ranga,
Looking at your logs we found an error that could possibly cause this and it is related to the properties.apps_domain in the deployment manifest. By setting it to 'b%()osh-lite.com' (using special characters), we could reproduce the following error which we also found in your log file:
Encountered error: name format\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/sequel-4.29.0/lib/sequel/model/base.rb:1543:in `save'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/app/models/runtime/shared_domain.rb:35:in `block in find_or_create'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/sequel-4.29.0/lib/sequel/database/transactions.rb:134:in `_transaction'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/sequel-4.29.0/lib/sequel/database/transactions.rb:108:in `block in transaction'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/sequel-4.29.0/lib/sequel/database/connecting.rb:249:in `block in synchronize'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/sequel-4.29.0/lib/sequel/connection_pool/threaded.rb:103:in `hold'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/sequel-4.29.0/lib/sequel/database/connecting.rb:249:in `synchronize'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/sequel-4.29.0/lib/sequel/database/transactions.rb:97:in `transaction'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/app/models/runtime/shared_domain.rb:27:in `find_or_create'\n/var/vcap/data/packages/cloud_controller_ng/da452b34d79be56a0784c7e88d6b9c0e1811a9d8.1-f934c136e6019cf54ab7aa04a3c153657226e729/cloud_controller_ng/lib/cloud_controller/seeds.rb:57:in `block in create_seed_domains'\n/var/vcap/data/packages/cloud_controller_ng/da452b34d79be56a0784c7e88d6b9c0e1811a9d8.1-f934c136e6019cf54ab7aa04a3c153657226e729/cloud_controller_ng/lib/cloud_controller/seeds.rb:56:in `each'\n/var/vcap/data/packages/cloud_controller_ng/da452b34d79be56a0784c7e88d6b9c0e1811a9d8.1-f934c136e6019cf54ab7aa04a3c153657226e729/cloud_controller_ng/lib/cloud_controller/seeds.rb:56:in `create_seed_domains'\n/var/vcap/data/packages/cloud_controller_ng/da452b34d79be56a0784c7e88d6b9c0e1811a9d8.1-f934c136e6019cf54ab7aa04a3c153657226e729/cloud_controller_ng/lib/cloud_controller/seeds.rb:9:in `write_seed_data'\n/var/vcap/data/packages/cloud_controller_ng/da452b34d79be56a0784c7e88d6b9c0e1811a9d8.1-f934c136e6019cf54ab7aa04a3c153657226e729/cloud_controller_ng/lib/cloud_controller/runner.rb:93:in `block in run!'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/eventmachine-1.0.9.1/lib/eventmachine.rb:193:in `call'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/eventmachine-1.0.9.1/lib/eventmachine.rb:193:in `run_machine'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/eventmachine-1.0.9.1/lib/eventmachine.rb:193:in `run'\n/var/vcap/data/packages/cloud_controller_ng/da452b34d79be56a0784c7e88d6b9c0e1811a9d8.1-f934c136e6019cf54ab7aa04a3c153657226e729/cloud_controller_ng/lib/cloud_controller/runner.rb:87:in `run!'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/bin/cloud_controller:8:in `<main>'
Can you check the apps_domain property and see if there is anything suspicious with it?
Thanks, Peter & Kara
On Thu, Apr 7, 2016 at 2:32 PM Ranga Rajagopalan < ranga.rajagopalan(a)gmail.com> wrote:
Hi Peter,
Attaching /var/vcap/sys/log/cloud_controller_worker/cloud_controller_worker.log.gz and /var/vcap/sys/log/cloud_controller_worker_ctl.log.gz. There isn't a /var/vcap/sys/log/cloud_controller_worker/ directory on this node.
vcap(a)572afc33-0735-4727-8ff3-9dc6d7fa8af0:~$ ls /var/vcap/sys/log/ agent_ctl.err.log cloud_controller_worker_ctl.log nginx_cc/ agent_ctl.log consul_agent/ nginx_ctl.err.log cloud_controller_migration/ metron_agent/ nginx_ctl.log cloud_controller_migration_ctl.err.log metron_agent_ctl.err.log route_registrar/ cloud_controller_migration_ctl.log metron_agent_ctl.log route_registrar_ctl.err.log cloud_controller_ng/ monit/ route_registrar_ctl.log cloud_controller_ng_ctl.err.log nfs_mounter/ statsd-injector/ cloud_controller_ng_ctl.log nfs_mounter_ctl.err.log statsd-injector-ctl.err.log cloud_controller_worker_ctl.err.log nfs_mounter_ctl.log statsd-injector-ctl.log
On Thu, Apr 7, 2016 at 12:21 PM, Peter Goetz <peter.gtz(a)gmail.com> wrote:
Hi Ranga,
To trouble-shoot this issue could you also provide the contents of /var/vcap/sys/log/cloud_controller_ng/cloud_controller_ng.log and /var/vcap/sys/log/cloud_controller_worker/cloud_controller_worker.log? This should give us more details about what's going on. The ctl script logs do not provide enough details.
Thanks, Peter
On Wed, Apr 6, 2016 at 6:12 PM Ranga Rajagopalan < ranga.rajagopalan(a)gmail.com> wrote:
I tried v231. Unfortunately, same issue.
-- Thanks,
Ranga
-- Thanks,
Ranga
|
|
[Metron Agent] failed to generate job templates with metron agent on top of OpenStack Dynamic network
Hi,guys When deploy CF on top of OpenStack with dynamic network, the jobs failed with metron-agent Error filling in template 'syslog_forwarder.conf.erb' (line 44: undefined method `strip' for nil:NilClass) here's related logs D etecting deployment changes ---------------------------- Releases cf version type changed: String -> Fixnum - 233 + 233
Compilation No changes
Update ± canaries: - 1 + 0
Resource pools No changes
Disk pools No changes
Networks dynamic-net + name: dynamic-net subnets 10.0.0.0/24 cloud_properties + net_id: 0700ae03-4b38-464e-b40d-0a9c8dd18ff0 + security_groups: ["Test OS SG_20160128T070152Z"] + dns: ["114.114.114.114", "8.8.8.8"]
+ range: 10.0.0.0/24 + name: Test OS Sub Internal Network_20160128T070152Z
+ type: dynamic
Jobs stats_z1
± networks:
- {"name"=>"cf1"}
+ {"name"=>"dynamic-net"}
Properties No changes
Meta No changes
Please review all changes carefully
Deploying ---------
Are you sure you want to deploy? (type 'yes' to continue): yes
Director task 57 Started preparing deployment > Preparing deployment. Done (00:00:03)
Error 100: Unable to render instance groups for deployment. Errors are: - Unable to render jobs for instance group 'stats_z1'. Errors are:
- Unable to render templates for job 'metron_agent'. Errors are:
- Error filling in template 'syslog_forwarder.conf.erb' (line 44:
undefined method `strip' for nil:NilClass)
Task 57 error
For a more detailed error report, run: bosh task 57 --debug as the ip manged by OpenStack, bosh cannot get the actual ip address of each vm until vm alive, this lead to the generated job spec doesn't contain ip address infos so, must i have to configure network type to manual? snippets of deployment yml 1001 - name: dynamic-net 1002 subnets: 1003 - cloud_properties: 1004 net_id: 0700ae03-4b38-464e-b40d-0a9c8dd18ff0 1005 security_groups: 1006 - Test OS SG_20160128T070152Z 1007 dns: 1010 - 114.114.114.114 1011 - 8.8.8.8 1012 range: 10.0.0.0/24 1013 name: Test OS Sub Internal Network_20160128T070152Z 1014 type: dynamic Rendered job spec {"deployment"=>"staging-01", "job"= {"name"=>"stats_z1", "templates"=>[{"name"=>"collector", "version"=>"6c210292f18d129e9a037fe7053836db2d494344", "sha1"=>"38927f47b15c2daf6c8a2e7c760e73e5ff90 dfd4", "blobstore_id"=>"23531029-0ee1-4267-8863-b5f931afaecb"}, {"name"=>"metron_agent", "version"=>"2b80a211127fc642fc8bb0d14d7eb30c37730db3", "sha1"=>"150f2 7445c2ef960951c1f26606525d41ec629b2", "blobstore_id"=>"e87174dc-f3f7-4768-94cd-74f299813528"}], "template"=>"collector", "version"=>"6c210292f18d129e9a037fe70 53836db2d494344", "sha1"=>"38927f47b15c2daf6c8a2e7c760e73e5ff90dfd4", "blobstore_id"=>"23531029-0ee1-4267-8863-b5f931afaecb"}, "index"=>0, "bootstrap"=>true, "name"=>"stats_z1", "id"=>"99f349d0-fb5d-4de7-9912-3de5559d2f19", "az"=>nil,
*"networks"=>{"dynamic-net"=>{"type"=>"dynamic", "cloud_properties"=>{"net_id"=>"0700ae03-4b38-464e-b40d-0a9c8dd18ff0", "security_groups"=>["Test OS SG_20160128T070152Z"]}, "dns"=>["114.114.114.114", "8.8.8.8", "10.0.0.13"], "default"=>["dns", "gateway"], "dns_record_name"=>"0.stats-z1.dynamic-net.staging-01.microbosh"}}*, "properties"=>{"collector"=>{"aws"=>{ "access_key_id"=>nil, "secret_access_key"=>nil}, "datadog"=>{"api_key"=>nil, "application_key"=>nil}, "deployment_name"=>nil, "logging_level"=>"info", "interv als"=>{"discover"=>60, "healthz"=>30, "local_metrics"=>30, "nats_ping"=>30, "prune"=>300, "varz"=>30}, "use_aws_cloudwatch"=>false, "use_datadog"=>false, "use _tsdb"=>false, "opentsdb"=>{"address"=>nil, "port"=>nil}, "use_graphite"=>false, "graphite"=>{"address"=>nil, "port"=>nil}, "memory_threshold"=>800}, "nats"=> {"machines"=>["10.0.0.127"], "password"=>"NATS_PASSWORD", "port"=>4222, "user"=>"NATS_USER"}, "syslog_daemon_config"=>{"address"=>nil, "port"=>nil, "transport "=>"tcp", "fallback_addresses"=>[], "custom_rule"=>"", "max_message_size"=>"4k"}, "metron_agent"=>{"dropsonde_incoming_port"=>3457, "preferred_protocol"=>"udp ", "tls"=>{"client_cert"=>"", "client_key"=>""}, "debug"=>false, "zone"=>"z1", "deployment"=>"ya-staging-01", "tcp"=>{"batching_buffer_bytes"=>10240, "batchin g_buffer_flush_interval_milliseconds"=>100}, "logrotate"=>{"freq_min"=>5, "rotate"=>7, "size"=>"50M"}, "buffer_size"=>10000, "enable_buffer"=>false}, "metron_ endpoint"=>{"shared_secret"=>"LOGGREGATOR_ENDPOINT_SHARED_SECRET"}, "loggregator"=>{"tls"=>{"ca_cert"=>""}, "dropsonde_incoming_port"=>3457, "etcd"=>{"machine s"=>["10.0.0.133"], "maxconcurrentrequests"=>10}}}, "dns_domain_name"=>"microbosh", "links"=>{}, "address"=>"99f349d0-fb5d-4de7-9912-3de5559d2f19.stats-z1.dyn amic-net.ya-staging-01.microbosh", "persistent_disk"=>0, "resource_pool"=>"small_z1"} -- Regards, Yitao
|
|
Re: cf v233 api_z1/api_z2 failing
Here's my deployment manifest. app_domains is set to cfapp. I can't find bost-lite anywhere in the file at all.
toggle quoted messageShow quoted text
On Thu, Apr 7, 2016 at 4:27 PM, Peter Goetz <peter.gtz(a)gmail.com> wrote: Hi Ranga,
Looking at your logs we found an error that could possibly cause this and it is related to the properties.apps_domain in the deployment manifest. By setting it to 'b%()osh-lite.com' (using special characters), we could reproduce the following error which we also found in your log file:
Encountered error: name format\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/sequel-4.29.0/lib/sequel/model/base.rb:1543:in `save'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/app/models/runtime/shared_domain.rb:35:in `block in find_or_create'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/sequel-4.29.0/lib/sequel/database/transactions.rb:134:in `_transaction'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/sequel-4.29.0/lib/sequel/database/transactions.rb:108:in `block in transaction'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/sequel-4.29.0/lib/sequel/database/connecting.rb:249:in `block in synchronize'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/sequel-4.29.0/lib/sequel/connection_pool/threaded.rb:103:in `hold'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/sequel-4.29.0/lib/sequel/database/connecting.rb:249:in `synchronize'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/sequel-4.29.0/lib/sequel/database/transactions.rb:97:in `transaction'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/app/models/runtime/shared_domain.rb:27:in `find_or_create'\n/var/vcap/data/packages/cloud_controller_ng/da452b34d79be56a0784c7e88d6b9c0e1811a9d8.1-f934c136e6019cf54ab7aa04a3c153657226e729/cloud_controller_ng/lib/cloud_controller/seeds.rb:57:in `block in create_seed_domains'\n/var/vcap/data/packages/cloud_controller_ng/da452b34d79be56a0784c7e88d6b9c0e1811a9d8.1-f934c136e6019cf54ab7aa04a3c153657226e729/cloud_controller_ng/lib/cloud_controller/seeds.rb:56:in `each'\n/var/vcap/data/packages/cloud_controller_ng/da452b34d79be56a0784c7e88d6b9c0e1811a9d8.1-f934c136e6019cf54ab7aa04a3c153657226e729/cloud_controller_ng/lib/cloud_controller/seeds.rb:56:in `create_seed_domains'\n/var/vcap/data/packages/cloud_controller_ng/da452b34d79be56a0784c7e88d6b9c0e1811a9d8.1-f934c136e6019cf54ab7aa04a3c153657226e729/cloud_controller_ng/lib/cloud_controller/seeds.rb:9:in `write_seed_data'\n/var/vcap/data/packages/cloud_controller_ng/da452b34d79be56a0784c7e88d6b9c0e1811a9d8.1-f934c136e6019cf54ab7aa04a3c153657226e729/cloud_controller_ng/lib/cloud_controller/runner.rb:93:in `block in run!'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/eventmachine-1.0.9.1/lib/eventmachine.rb:193:in `call'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/eventmachine-1.0.9.1/lib/eventmachine.rb:193:in `run_machine'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/eventmachine-1.0.9.1/lib/eventmachine.rb:193:in `run'\n/var/vcap/data/packages/cloud_controller_ng/da452b34d79be56a0784c7e88d6b9c0e1811a9d8.1-f934c136e6019cf54ab7aa04a3c153657226e729/cloud_controller_ng/lib/cloud_controller/runner.rb:87:in `run!'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/bin/cloud_controller:8:in `<main>'
Can you check the apps_domain property and see if there is anything suspicious with it?
Thanks, Peter & Kara
On Thu, Apr 7, 2016 at 2:32 PM Ranga Rajagopalan < ranga.rajagopalan(a)gmail.com> wrote:
Hi Peter,
Attaching /var/vcap/sys/log/cloud_controller_worker/cloud_controller_worker.log.gz and /var/vcap/sys/log/cloud_controller_worker_ctl.log.gz. There isn't a /var/vcap/sys/log/cloud_controller_worker/ directory on this node.
vcap(a)572afc33-0735-4727-8ff3-9dc6d7fa8af0:~$ ls /var/vcap/sys/log/ agent_ctl.err.log cloud_controller_worker_ctl.log nginx_cc/ agent_ctl.log consul_agent/ nginx_ctl.err.log cloud_controller_migration/ metron_agent/ nginx_ctl.log cloud_controller_migration_ctl.err.log metron_agent_ctl.err.log route_registrar/ cloud_controller_migration_ctl.log metron_agent_ctl.log route_registrar_ctl.err.log cloud_controller_ng/ monit/ route_registrar_ctl.log cloud_controller_ng_ctl.err.log nfs_mounter/ statsd-injector/ cloud_controller_ng_ctl.log nfs_mounter_ctl.err.log statsd-injector-ctl.err.log cloud_controller_worker_ctl.err.log nfs_mounter_ctl.log statsd-injector-ctl.log
On Thu, Apr 7, 2016 at 12:21 PM, Peter Goetz <peter.gtz(a)gmail.com> wrote:
Hi Ranga,
To trouble-shoot this issue could you also provide the contents of /var/vcap/sys/log/cloud_controller_ng/cloud_controller_ng.log and /var/vcap/sys/log/cloud_controller_worker/cloud_controller_worker.log? This should give us more details about what's going on. The ctl script logs do not provide enough details.
Thanks, Peter
On Wed, Apr 6, 2016 at 6:12 PM Ranga Rajagopalan < ranga.rajagopalan(a)gmail.com> wrote:
I tried v231. Unfortunately, same issue.
-- Thanks,
Ranga
-- Thanks,
Ranga
|
|
Re: Automated Deployment of window cell for diego
BOSH support for Windows is in the pipeline. In the meantime you hopefully won't have to set up cells very often, but could automate the process with some powershell scripting. Cheers, Steven Benario Windows PM On Thu, Apr 7, 2016 at 8:44 PM, Long Nguyen <long.nguyen11288(a)gmail.com> wrote: Ah was wondering if anyone came up with anything in meantime.
On April 7, 2016 at 8:16:15 PM, Gwenn Etourneau (getourneau(a)pivotal.io) wrote:
Bosh support for windows is coming, instead for now you have the msi installer ..
On Thu, Apr 7, 2016 at 11:27 PM, Long Nguyen <long.nguyen11288(a)gmail.com> wrote:
Anyone know about automated deployment of window cell for diego? I've done it manually a few months back and it was very manual and error prone. Does anyone have anything that automate process?
|
|
Re: Automated Deployment of window cell for diego
Ah was wondering if anyone came up with anything in meantime.
toggle quoted messageShow quoted text
On April 7, 2016 at 8:16:15 PM, Gwenn Etourneau (getourneau(a)pivotal.io) wrote:
Bosh support for windows is coming, instead for now you have the msi installer ..
On Thu, Apr 7, 2016 at 11:27 PM, Long Nguyen <long.nguyen11288(a)gmail.com> wrote: Anyone know about automated deployment of window cell for diego? I've done it manually a few months back and it was very manual and error prone. Does anyone have anything that automate process?
|
|
Re: Static IP setup for routers on AWS
The UAA needs to know the router IPs to know which IPs to accept inbound requests from. If you don't care about this, you can try configuring UAA to allow requests from many IPs, and remove the static IPs from gorouter. I would be interested to find out the result of this experiment should you try it out. Best, Amit On Thu, Apr 7, 2016 at 6:28 AM, Engelke, Johannes <info(a)johannes-engelke.de> wrote: Hi, does anybody know, why the routers got static ips in the cf-infrastructure-aws.yml file? https://github.com/cloudfoundry/cf-release/blob/master/templates/cf-infrastructure-aws.yml#L173
Bosh is assigning the instances to ELB’s during deploy time, so there should be no need to have static addresses here.
If nobody know’s a good reason should we remove them ;-)
Cheers Johannes
|
|
Re: Automated Deployment of window cell for diego
Bosh support for windows is coming, instead for now you have the msi installer .. On Thu, Apr 7, 2016 at 11:27 PM, Long Nguyen <long.nguyen11288(a)gmail.com> wrote: Anyone know about automated deployment of window cell for diego? I've done it manually a few months back and it was very manual and error prone. Does anyone have anything that automate process?
|
|
Re: cf stop not sending SIGTERM
Hi, William,
What version of the Java buildpack are you using? The introduction of the memory calculator in v3.1 resulted in a more complicated start-command that meant the app's java process wouldn't receive a TERM signal when the Diego cell rep stopped the instance, but this was fixed in v3.4 of the buildpack with a judiciously placed `exec`. Diego doesn't call the Stop() method on a garden container, instead opting either to call Signal() on individual processes that it has invoked or to call Destroy() the container.
Thanks, Eric, CF Runtime Diego PM
toggle quoted messageShow quoted text
|
|
Re: cf v233 api_z1/api_z2 failing
Peter Goetz <peter.gtz@...>
Hi Ranga, Looking at your logs we found an error that could possibly cause this and it is related to the properties.apps_domain in the deployment manifest. By setting it to 'b%()osh-lite.com' (using special characters), we could reproduce the following error which we also found in your log file: Encountered error: name format\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/sequel-4.29.0/lib/sequel/model/base.rb:1543:in `save'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/app/models/runtime/shared_domain.rb:35:in `block in find_or_create'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/sequel-4.29.0/lib/sequel/database/transactions.rb:134:in `_transaction'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/sequel-4.29.0/lib/sequel/database/transactions.rb:108:in `block in transaction'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/sequel-4.29.0/lib/sequel/database/connecting.rb:249:in `block in synchronize'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/sequel-4.29.0/lib/sequel/connection_pool/threaded.rb:103:in `hold'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/sequel-4.29.0/lib/sequel/database/connecting.rb:249:in `synchronize'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/sequel-4.29.0/lib/sequel/database/transactions.rb:97:in `transaction'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/app/models/runtime/shared_domain.rb:27:in `find_or_create'\n/var/vcap/data/packages/cloud_controller_ng/da452b34d79be56a0784c7e88d6b9c0e1811a9d8.1-f934c136e6019cf54ab7aa04a3c153657226e729/cloud_controller_ng/lib/cloud_controller/seeds.rb:57:in `block in create_seed_domains'\n/var/vcap/data/packages/cloud_controller_ng/da452b34d79be56a0784c7e88d6b9c0e1811a9d8.1-f934c136e6019cf54ab7aa04a3c153657226e729/cloud_controller_ng/lib/cloud_controller/seeds.rb:56:in `each'\n/var/vcap/data/packages/cloud_controller_ng/da452b34d79be56a0784c7e88d6b9c0e1811a9d8.1-f934c136e6019cf54ab7aa04a3c153657226e729/cloud_controller_ng/lib/cloud_controller/seeds.rb:56:in `create_seed_domains'\n/var/vcap/data/packages/cloud_controller_ng/da452b34d79be56a0784c7e88d6b9c0e1811a9d8.1-f934c136e6019cf54ab7aa04a3c153657226e729/cloud_controller_ng/lib/cloud_controller/seeds.rb:9:in `write_seed_data'\n/var/vcap/data/packages/cloud_controller_ng/da452b34d79be56a0784c7e88d6b9c0e1811a9d8.1-f934c136e6019cf54ab7aa04a3c153657226e729/cloud_controller_ng/lib/cloud_controller/runner.rb:93:in `block in run!'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/eventmachine-1.0.9.1/lib/eventmachine.rb:193:in `call'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/eventmachine-1.0.9.1/lib/eventmachine.rb:193:in `run_machine'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/eventmachine-1.0.9.1/lib/eventmachine.rb:193:in `run'\n/var/vcap/data/packages/cloud_controller_ng/da452b34d79be56a0784c7e88d6b9c0e1811a9d8.1-f934c136e6019cf54ab7aa04a3c153657226e729/cloud_controller_ng/lib/cloud_controller/runner.rb:87:in `run!'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/bin/cloud_controller:8:in `<main>' Can you check the apps_domain property and see if there is anything suspicious with it? Thanks, Peter & Kara On Thu, Apr 7, 2016 at 2:32 PM Ranga Rajagopalan < ranga.rajagopalan(a)gmail.com> wrote: Hi Peter,
Attaching /var/vcap/sys/log/cloud_controller_worker/cloud_controller_worker.log.gz and /var/vcap/sys/log/cloud_controller_worker_ctl.log.gz. There isn't a /var/vcap/sys/log/cloud_controller_worker/ directory on this node.
vcap(a)572afc33-0735-4727-8ff3-9dc6d7fa8af0:~$ ls /var/vcap/sys/log/ agent_ctl.err.log cloud_controller_worker_ctl.log nginx_cc/ agent_ctl.log consul_agent/ nginx_ctl.err.log cloud_controller_migration/ metron_agent/ nginx_ctl.log cloud_controller_migration_ctl.err.log metron_agent_ctl.err.log route_registrar/ cloud_controller_migration_ctl.log metron_agent_ctl.log route_registrar_ctl.err.log cloud_controller_ng/ monit/ route_registrar_ctl.log cloud_controller_ng_ctl.err.log nfs_mounter/ statsd-injector/ cloud_controller_ng_ctl.log nfs_mounter_ctl.err.log statsd-injector-ctl.err.log cloud_controller_worker_ctl.err.log nfs_mounter_ctl.log statsd-injector-ctl.log
On Thu, Apr 7, 2016 at 12:21 PM, Peter Goetz <peter.gtz(a)gmail.com> wrote:
Hi Ranga,
To trouble-shoot this issue could you also provide the contents of /var/vcap/sys/log/cloud_controller_ng/cloud_controller_ng.log and /var/vcap/sys/log/cloud_controller_worker/cloud_controller_worker.log? This should give us more details about what's going on. The ctl script logs do not provide enough details.
Thanks, Peter
On Wed, Apr 6, 2016 at 6:12 PM Ranga Rajagopalan < ranga.rajagopalan(a)gmail.com> wrote:
I tried v231. Unfortunately, same issue.
-- Thanks,
Ranga
|
|
Re: CF 212 - service in "delete failed" -> how to remove it properly?
`cf curl /v2/service_instances/#{service_guid}?purge=true -X DELETE` should delete stuck service records. `cf service --guid #{service_name}` will get you the service guid to use in the above command. Though note that this doesn't fix whatever caused the service to get into the state that kept it from being deleted, and it probably still exists on the system providing the service, so that needs to be dealt with separately. (Or you might continue being charged for that service, for instance.) - Natalie On Thu, Apr 7, 2016 at 6:27 AM Rafal Radecki <radecki.rafal(a)gmail.com> wrote: Hi :)
I have several organizations in which I have services in "delete failed" state. They prevent me from deleting the organization.
What is the best way to remove metadata about such service?
BR, Rafal.
|
|
Re: cf v233 api_z1/api_z2 failing
Hi Peter,
Attaching /var/vcap/sys/log/cloud_controller_worker/cloud_controller_worker.log.gz and /var/vcap/sys/log/cloud_controller_worker_ctl.log.gz. There isn't a /var/vcap/sys/log/cloud_controller_worker/ directory on this node.
vcap(a)572afc33-0735-4727-8ff3-9dc6d7fa8af0:~$ ls /var/vcap/sys/log/ agent_ctl.err.log cloud_controller_worker_ctl.log nginx_cc/ agent_ctl.log consul_agent/ nginx_ctl.err.log cloud_controller_migration/ metron_agent/ nginx_ctl.log cloud_controller_migration_ctl.err.log metron_agent_ctl.err.log route_registrar/ cloud_controller_migration_ctl.log metron_agent_ctl.log route_registrar_ctl.err.log cloud_controller_ng/ monit/ route_registrar_ctl.log cloud_controller_ng_ctl.err.log nfs_mounter/ statsd-injector/ cloud_controller_ng_ctl.log nfs_mounter_ctl.err.log statsd-injector-ctl.err.log cloud_controller_worker_ctl.err.log nfs_mounter_ctl.log statsd-injector-ctl.log
toggle quoted messageShow quoted text
On Thu, Apr 7, 2016 at 12:21 PM, Peter Goetz <peter.gtz(a)gmail.com> wrote: Hi Ranga,
To trouble-shoot this issue could you also provide the contents of /var/vcap/sys/log/cloud_controller_ng/cloud_controller_ng.log and /var/vcap/sys/log/cloud_controller_worker/cloud_controller_worker.log? This should give us more details about what's going on. The ctl script logs do not provide enough details.
Thanks, Peter
On Wed, Apr 6, 2016 at 6:12 PM Ranga Rajagopalan < ranga.rajagopalan(a)gmail.com> wrote:
I tried v231. Unfortunately, same issue.
-- Thanks,
Ranga
|
|