Date   

diego error: failed to initialize container --where to look?

Tom Sherrod <tom.sherrod@...>
 

After upgrade to/deploy diego in multiple environments with success, I'm a bit stumped on error for a new install.

A cf docker-push results in:
015-09-26T16:02:01.65+0000 [API/0] OUT Updated app with guid 7aa5800b-96d3-48a0-b707-1d85680e10c9 ({"state"=>"STARTED"})
2015-09-26T16:02:01.74+0000 [API/0] ERR Failed to stage application: insufficient resources

I back up a step and diego enable an app. Fails on start of app:
2015-09-26T16:14:26.38+0000 [API/0] OUT App instance exited with guid be3f6daa-96d1-41a3-8a3a-2d0ff697f8c4 payload: {"instance"=>"8d1947c6-84cb-4144-4a97-9038a7978ebf", "index"=>0, "reason"=>"CRASHED", "exit_description"=>"failed to initialize container", "crash_count"=>3, "crash_timestamp"=>1443284066323748334, "version"=>"b5a0a268-82f7-445d-937d-4349415b2a70"}

I'm betting it is one machine not being able to talk to another.
What log may provide any additional details on what's failing?

Thanks,
Tom


Re: postgres out of disk space

Aleksey Zalesov
 

It will be handy to have a configuration option for CC to prune unused droplets after 7 days of inactivity, for example.

On 26 Sep 2015, at 01:47, CF Runtime <cfruntime(a)gmail.com<mailto:cfruntime(a)gmail.com>> wrote:

cf-release v213 fixed a bug where buildpack caches were not getting cleaned up properly. Are you running a version prior to 213?

If so, you might need to go into the `shared/run.10.10.2.37.xip.io-cc-droplets` directory and delete the `buildpack_cache` subdirectory. Deleting the contents of buildpack_cache will not cause failures in the system, the caches will simply get generated from scratch the next time apps stage.

If you upgrade to 213 or later, you can just call the api endpoint to clean out the buildpack cache. You can find the details in the v213 release notes:
https://github.com/cloudfoundry/cf-release/releases/tag/v213

Joseph & Natalie
CF Release Integration Team

On Fri, Sep 25, 2015 at 11:53 AM, Matthias Ender <Matthias.Ender(a)sas.com<mailto:Matthias.Ender(a)sas.com>> wrote:
ah, it’s not postgres:

90M postgres

it’s actually:

94G shared/run.10.10.2.37.xip.io-cc-droplets

Does it keep a copy of each artifact ever pushed?
Or is this part of the event storage, controlled by these properties:

app_events:
cutoff_age_in_days: 31
app_usage_events:
cutoff_age_in_days: 31
audit_events:
cutoff_age_in_days: 31

??


From: Aleksey Zalesov [mailto:aleksey.zalesov(a)altoros.com<mailto:aleksey.zalesov(a)altoros.com>]
Sent: Friday, September 25, 2015 12:23 PM
To: Discussions about Cloud Foundry projects and the system overall. <cf-dev(a)lists.cloudfoundry.org<mailto:cf-dev(a)lists.cloudfoundry.org>>
Subject: [cf-dev] Re: postgres out of disk space

Does your database occupy this 100 GB? Or something else like logs?

Aleksey Zalesov | CloudFoundry Engineer | Altoros
Tel: (617) 841-2121 ext. 5707<tel:%28617%29%20841-2121%20ext.%205707> | Toll free: 855-ALTOROS
Fax: (866) 201-3646<tel:%28866%29%20201-3646> | Skype: aleksey_zalesov
www.altoros.com<http://www.altoros.com/> | blog.altoros.com<http://blog.altoros.com/> | twitter.com/altoros<http://twitter.com/altoros>

On 25 Sep 2015, at 14:22, Matthias Ender <Matthias.Ender(a)sas.com<mailto:Matthias.Ender(a)sas.com>> wrote:

I have a cf-aws-tiny cf-boshrelease deployment, and it’s been running well for over 4 months.
We have about 40 apps, with a couple of dozen of cf pushes each day.
Yesterday pushing apps became spotty and then impossible, with various errors.
Turned out the 100GB disk for the postgres instance on the data note was full.
I increased the disk size and things a running again.
But – what happened there? 100G and growing seems like awfully large database for a rather modest use.
And I’m worried it’ll just happen again in a few months.

thanks,
Matthias


Re: postgres out of disk space

CF Runtime
 

cf-release v213 fixed a bug where buildpack caches were not getting cleaned
up properly. Are you running a version prior to 213?

If so, you might need to go into the
`shared/run.10.10.2.37.xip.io-cc-droplets` directory and delete the
`buildpack_cache` subdirectory. Deleting the contents of buildpack_cache
will not cause failures in the system, the caches will simply get generated
from scratch the next time apps stage.

If you upgrade to 213 or later, you can just call the api endpoint to clean
out the buildpack cache. You can find the details in the v213 release notes:
https://github.com/cloudfoundry/cf-release/releases/tag/v213

Joseph & Natalie
CF Release Integration Team

On Fri, Sep 25, 2015 at 11:53 AM, Matthias Ender <Matthias.Ender(a)sas.com>
wrote:

ah, it’s not postgres:



90M postgres



it’s actually:



94G shared/run.10.10.2.37.xip.io-cc-droplets



Does it keep a copy of each artifact ever pushed?

Or is this part of the event storage, controlled by these properties:



app_events:

cutoff_age_in_days: 31

app_usage_events:

cutoff_age_in_days: 31

audit_events:

cutoff_age_in_days: 31



??





*From:* Aleksey Zalesov [mailto:aleksey.zalesov(a)altoros.com]
*Sent:* Friday, September 25, 2015 12:23 PM
*To:* Discussions about Cloud Foundry projects and the system overall. <
cf-dev(a)lists.cloudfoundry.org>
*Subject:* [cf-dev] Re: postgres out of disk space



Does your database occupy this 100 GB? Or something else like logs?



Aleksey Zalesov | CloudFoundry Engineer | Altoros

Tel: (617) 841-2121 ext. 5707 | Toll free: 855-ALTOROS
Fax: (866) 201-3646 | Skype: aleksey_zalesov

www.altoros.com | blog.altoros.com | twitter.com/altoros



On 25 Sep 2015, at 14:22, Matthias Ender <Matthias.Ender(a)sas.com> wrote:



I have a cf-aws-tiny cf-boshrelease deployment, and it’s been running well
for over 4 months.

We have about 40 apps, with a couple of dozen of cf pushes each day.

Yesterday pushing apps became spotty and then impossible, with various
errors.

Turned out the 100GB disk for the postgres instance on the data note was
full.

I increased the disk size and things a running again.

But – what happened there? 100G and growing seems like awfully large
database for a rather modest use.

And I’m worried it’ll just happen again in a few months.



thanks,

Matthias



Re: postgres out of disk space

Matthew Sykes <matthew.sykes@...>
 

Droplets and app packages should be pruned when the applications they're
associated with are deleted. Only one or two versions of the droplet are
ever held and only copy of the application bits.

There's another blob store that holds application resources for resource
match flow when uploading an app. Those resources are not pruned so you
should make sure that you disable that in your manifest if you don't want
that behavior.

The droplet blob store is clearly your issue - not events.

On Fri, Sep 25, 2015 at 2:53 PM, Matthias Ender <Matthias.Ender(a)sas.com>
wrote:

ah, it’s not postgres:



90M postgres



it’s actually:



94G shared/run.10.10.2.37.xip.io-cc-droplets



Does it keep a copy of each artifact ever pushed?

Or is this part of the event storage, controlled by these properties:



app_events:

cutoff_age_in_days: 31

app_usage_events:

cutoff_age_in_days: 31

audit_events:

cutoff_age_in_days: 31



??





*From:* Aleksey Zalesov [mailto:aleksey.zalesov(a)altoros.com]
*Sent:* Friday, September 25, 2015 12:23 PM
*To:* Discussions about Cloud Foundry projects and the system overall. <
cf-dev(a)lists.cloudfoundry.org>
*Subject:* [cf-dev] Re: postgres out of disk space



Does your database occupy this 100 GB? Or something else like logs?



Aleksey Zalesov | CloudFoundry Engineer | Altoros

Tel: (617) 841-2121 ext. 5707 | Toll free: 855-ALTOROS
Fax: (866) 201-3646 | Skype: aleksey_zalesov

www.altoros.com | blog.altoros.com | twitter.com/altoros



On 25 Sep 2015, at 14:22, Matthias Ender <Matthias.Ender(a)sas.com> wrote:



I have a cf-aws-tiny cf-boshrelease deployment, and it’s been running well
for over 4 months.

We have about 40 apps, with a couple of dozen of cf pushes each day.

Yesterday pushing apps became spotty and then impossible, with various
errors.

Turned out the 100GB disk for the postgres instance on the data note was
full.

I increased the disk size and things a running again.

But – what happened there? 100G and growing seems like awfully large
database for a rather modest use.

And I’m worried it’ll just happen again in a few months.



thanks,

Matthias




--
Matthew Sykes
matthew.sykes(a)gmail.com


Re: postgres out of disk space

Matthias Ender <Matthias.Ender@...>
 

ah, it’s not postgres:

90M postgres

it’s actually:

94G shared/run.10.10.2.37.xip.io-cc-droplets

Does it keep a copy of each artifact ever pushed?
Or is this part of the event storage, controlled by these properties:

app_events:
cutoff_age_in_days: 31
app_usage_events:
cutoff_age_in_days: 31
audit_events:
cutoff_age_in_days: 31

??


From: Aleksey Zalesov [mailto:aleksey.zalesov(a)altoros.com]
Sent: Friday, September 25, 2015 12:23 PM
To: Discussions about Cloud Foundry projects and the system overall. <cf-dev(a)lists.cloudfoundry.org>
Subject: [cf-dev] Re: postgres out of disk space

Does your database occupy this 100 GB? Or something else like logs?

Aleksey Zalesov | CloudFoundry Engineer | Altoros
Tel: (617) 841-2121 ext. 5707 | Toll free: 855-ALTOROS
Fax: (866) 201-3646 | Skype: aleksey_zalesov
www.altoros.com<http://www.altoros.com/> | blog.altoros.com<http://blog.altoros.com/> | twitter.com/altoros<http://twitter.com/altoros>

On 25 Sep 2015, at 14:22, Matthias Ender <Matthias.Ender(a)sas.com<mailto:Matthias.Ender(a)sas.com>> wrote:

I have a cf-aws-tiny cf-boshrelease deployment, and it’s been running well for over 4 months.
We have about 40 apps, with a couple of dozen of cf pushes each day.
Yesterday pushing apps became spotty and then impossible, with various errors.
Turned out the 100GB disk for the postgres instance on the data note was full.
I increased the disk size and things a running again.
But – what happened there? 100G and growing seems like awfully large database for a rather modest use.
And I’m worried it’ll just happen again in a few months.

thanks,
Matthias


Re: Environment variable changes in DIEGO

Mike Heath
 

By the `application_uris` data are broken logic, VCAP_SERVICES is also
broken and should be removed as it doesn't reflect service binding changes.

In my (strong) opinion it doesn't make any sense to run an application in a
dynamic environment and NOT expose its route(s). This has a lot of
implications. For example, an application running on Diego can no longer
register itself with a dynamic service registry since the information
needed to route to the application is totally obscured. (i.e. Spring
Cloud's integration with Eureka.)

It makes more sense to me to match the behavior of service bind/unbind by
notifying the user at route map/unmap time that they need to restart their
app for it to see the changes.

-Mike

On Wed, Sep 16, 2015 at 9:09 AM Matthew Sykes <matthew.sykes(a)gmail.com>
wrote:

The changes, in general, were intentional. The `application_uris` data was
always broken as it didn't reflect route changes. I can't speak directly to
the time stamp data.

The host is present still so I don't know why you don't see it.

We also have a migration guide [1]. If you think additional information is
needed there, pull requests would be welcome.

[1]:
https://github.com/cloudfoundry-incubator/diego-design-notes/blob/master/migrating-to-diego.md

On Wed, Sep 16, 2015 at 10:19 AM, Jack Cai <greensight(a)gmail.com> wrote:

I notice the below changes in the environment variables of DIEGO:
1. VCAP_APP_HOST & VCAP_APP_PORT are removed.
2. These fields are removed from VCAP_APPLICATION value:
application_uris, started_at, start, started_at_timestamp, host,
state_timestamp

I suppose #1 is intentional. How about #2?

Jack


--
Matthew Sykes
matthew.sykes(a)gmail.com


Re: Running/Deploying UAA as a standalone service (not within CloudFoundry environment)

Filip Hanik
 

hi TJ,
because the UAA supports multi tenancy and it denotes a tenant by using a
subdomain, we need to know what the base URL is.
By default it is localhost

so

http://tenant1.localhost:8080/uaa
http://tenant2.localhost:8080/uaa

would support the 'default' zone at localhost, and tenant1,localhost and
tenant2.localhost tenants.

You can add more default host names to (in addition to localhost) by adding
hostnames to a Yaml configuration file.

zones:
internal:
hostnames:
- example.com
- test.com

and this would now also support

example.com (default tenant/zone)
tenant1.example.com (tenant1)
tenant2.example.com (tenant2)

On Fri, Sep 25, 2015 at 10:20 AM, TJ Brown <tjbrown(a)gmail.com> wrote:

I'm trying to evaluate UAA to be used as a user authentication /
authorization service to be used within a microservice architecture but not
within a CloudFoundry environment. When running UAA using

$ ./gradlew run

the application seems to work locally but I can't access it externally.
So, what do I need to configure to allow access from other hosts?

Also, are there instructions / guides / tutorial for how to deploy uaa on
an existing Tomcat server?

Thanks for your help.


Re: postgres out of disk space

Aleksey Zalesov
 

Does your database occupy this 100 GB? Or something else like logs?

Aleksey Zalesov | CloudFoundry Engineer | Altoros
Tel: (617) 841-2121 ext. 5707 | Toll free: 855-ALTOROS
Fax: (866) 201-3646 | Skype: aleksey_zalesov
www.altoros.com <http://www.altoros.com/> | blog.altoros.com <http://blog.altoros.com/> | twitter.com/altoros <http://twitter.com/altoros>

On 25 Sep 2015, at 14:22, Matthias Ender <Matthias.Ender(a)sas.com> wrote:

I have a cf-aws-tiny cf-boshrelease deployment, and it’s been running well for over 4 months.
We have about 40 apps, with a couple of dozen of cf pushes each day.
Yesterday pushing apps became spotty and then impossible, with various errors.
Turned out the 100GB disk for the postgres instance on the data note was full.
I increased the disk size and things a running again.
But – what happened there? 100G and growing seems like awfully large database for a rather modest use.
And I’m worried it’ll just happen again in a few months.

thanks,
Matthias


Re: Running/Deploying UAA as a standalone service (not within CloudFoundry environment)

Frans Thamura
 

we modify the uaa (spring security) become OAuth2 Server independently

take a look www.merv.id

the code in github.com/meruvian/yama

F

--
Frans Thamura (曽志胜)
Java Champion
Shadow Master and Lead Investor
Meruvian.
Integrated Hypermedia Java Solution Provider.

Mobile: +628557888699
Blog: http://blogs.mervpolis.com/roller/flatburger (id)

FB: http://www.facebook.com/meruvian
TW: http://www.twitter.com/meruvian / @meruvian
Website: http://www.meruvian.org

"We grow because we share the same belief."

On Fri, Sep 25, 2015 at 11:20 PM, TJ Brown <tjbrown(a)gmail.com> wrote:

I'm trying to evaluate UAA to be used as a user authentication /
authorization service to be used within a microservice architecture but not
within a CloudFoundry environment. When running UAA using

$ ./gradlew run

the application seems to work locally but I can't access it externally.
So, what do I need to configure to allow access from other hosts?

Also, are there instructions / guides / tutorial for how to deploy uaa on
an existing Tomcat server?

Thanks for your help.


Running/Deploying UAA as a standalone service (not within CloudFoundry environment)

TJ Brown
 

I'm trying to evaluate UAA to be used as a user authentication / authorization service to be used within a microservice architecture but not within a CloudFoundry environment. When running UAA using

$ ./gradlew run

the application seems to work locally but I can't access it externally. So, what do I need to configure to allow access from other hosts?

Also, are there instructions / guides / tutorial for how to deploy uaa on an existing Tomcat server?

Thanks for your help.


Re: F5 Load Balancer Configuration for Cloud Foundry Loggregator

Anthony
 

Thanks for all the responses. We ended up finding an f5 device that is on 11.5. With that, things worked out of the box with only the irule to insert x-forwarded-proto. Everything is working over 443.

Regards,
Anthony

On Sep 22, 2015, at 11:35 PM, Mike Youngstrom <youngm(a)gmail.com> wrote:

If you are sharing a vip for http and websocket then 443 would be correct. But Anthony, you can try creating a layer 4 virtual server on 4443 that goes to the same pool on the back end and configure the CC to use that port instead for loggregator connections.

Mike

On Tue, Sep 22, 2015 at 10:32 PM, Johannes Hiemer <jvhiemer(a)gmail.com> wrote:
Are you sure your logregator endpoint is configured on 443 and not 4443?



On 23.09.2015, at 05:26, Anthony <lee.apc(a)gmail.com> wrote:

Yep. --recent works. Other cf commands and cf curl also works.

It definitely is the websockets for loggregator. Just not sure what the right config for F5 (version 10.4) should be.

Regards,
Anthony

On Sep 22, 2015, at 9:53 PM, Rohit Kumar <rokumar(a)pivotal.io> wrote:

Does `cf logs --recent` work for you? The recent logs request goes over HTTP. If that goes through that means only the websocket requests to loggregator servers are a problem.

Rohit

On Tue, Sep 22, 2015 at 8:18 PM, Anthony <lee.apc(a)gmail.com> wrote:
Thanks Mike! Unfortunately, upgrading is not an option since its a really loaded enterprise device. The interesting part is that there is a similarly set up websockets vip (plain old server i think .net) that is working on the same device.

We'll work with our network folks to find other devices with newer software we can use.

Would appreciate if anyone has other ideas?

Regards,
Anthony

On Sep 22, 2015, at 7:49 PM, Mike Youngstrom <youngm(a)gmail.com> wrote:

We are running 11.4 and 11.6. I'd give an upgrade a try before digging too much deeper.

Mike

On Sep 22, 2015 6:36 PM, "Anthony" <lee.apc(a)gmail.com> wrote:
The version we are testing in is 10.4.

Regards,
Anthony

On Sep 22, 2015, at 6:41 PM, Mike Youngstrom <youngm(a)gmail.com> wrote:

What version of F5 software are you running?

Mike

On Tue, Sep 22, 2015 at 5:20 PM, Anthony Lee <lee.apc(a)gmail.com> wrote:
Does any one have any experience configuring F5 load balancers in front of the CF routers? We have configured F5 and app https and cf push requests are working fine. However, the connectivity with loggregator is not working. Taking a look at the documentation, it requires "websocket support" on the load balancer. We've done the configuration specified here:

https://support.f5.com/kb/en-us/solutions/public/14000/800/sol14814.html

With the following irule basically, applying the default TCP profile if it detects websocket traffic:

when HTTP_REQUEST {
if { [string tolower [HTTP::header Upgrade]] contains "websocket" }{
HTTP::disable
}
}

However, we are running into errors. Doing `cf logs myapp1` yields:

Error dialing loggregator server: read tcp <ip redacted>:443: connection reset by peer.
Please ask your Cloud Foundry Operator to check the platform configuration (loggregator endpoint is wss://loggregator.<sys domain redacted>:443).

Does anyone have a clue?

Thanks!
Anthony


Re: postgres out of disk space

Matthew Sykes <matthew.sykes@...>
 

Just a thought: How large is your events table? Is it getting pruned at the
correct interval for you?

I believe events, app usage events, and audit events are pruned when
they're more than 31 days old default. You could change the cc properties
in the deployment manifest to make that a little more aggressive if that's
the problem.

On Fri, Sep 25, 2015 at 7:22 AM, Matthias Ender <Matthias.Ender(a)sas.com>
wrote:

I have a cf-aws-tiny cf-boshrelease deployment, and it’s been running well
for over 4 months.

We have about 40 apps, with a couple of dozen of cf pushes each day.

Yesterday pushing apps became spotty and then impossible, with various
errors.

Turned out the 100GB disk for the postgres instance on the data note was
full.

I increased the disk size and things a running again.

But – what happened there? 100G and growing seems like awfully large
database for a rather modest use.

And I’m worried it’ll just happen again in a few months.



thanks,

Matthias




--
Matthew Sykes
matthew.sykes(a)gmail.com


Re: Running the app test suite within the CATs, and the admin_buildpack_lifecycle_test is failing

JT Archie <jarchie@...>
 

Jordan,

Try the CATs with the deployment, just to be sure, but that being said...

It is also possible that your deployment may not have all the buildpacks
installed.

If you run the command `cf buildpacks`. It returns a list of uploaded
buildpacks on your CF deployment.

If you happen to have one missing that's it.

Cheers,

JT and Amin

On Thu, Sep 24, 2015 at 6:17 PM, CF Runtime <cfruntime(a)gmail.com> wrote:

You can also check out the v208 tag of cf-release, then run the
acceptance-tests from src/github.com/cloudfoundry/cf-acceptance-tests

Joseph
CF Release Integration Team

On Thu, Sep 24, 2015 at 2:53 PM, Christopher Piraino <cpiraino(a)pivotal.io>
wrote:

Jordan,

The Cloud Foundry bosh release comes with an errand called
"acceptance_tests" that contains the version of CATs which that version of
CF was tested with. You can run these by doing "bosh run errand
acceptance_tests".

There are also some manifest properties that you might need to set for
the CATs to run correctly. The list of all possible properties for the
acceptance_tests errand can be found here:
https://github.com/cloudfoundry/cf-release/blob/develop/jobs/acceptance-tests/spec
.


- Chris Piraino

On Thu, Sep 24, 2015 at 11:27 AM, Jordan Collier <
jordanicollier(a)gmail.com> wrote:

I was unclear on what I am asking, the real question is as follows:

What is the best way to run the apps test suite within the CATS on an
older version of cloud foundry? (for example I am running these tests on
version 208)


Re: Instance crashing after running once. Error: "reason"=>"CRASHED", "exit_status"=>0, "exit_description"=>"app instance exited"

Daniel Mikusa
 

On Fri, Sep 25, 2015 at 6:11 AM, zooba Sir <myfakename90(a)gmail.com> wrote:

I've pushed the app which uses redis service (A sample app which simply
send and receives a message thru Redis service).

Does it start and send / receive a set number of messages or does it do
this forever?


Instance is created at start and after successfully running once the
instance getting crashed with error: "reason"=>"CRASHED", "exit_status"=>0,
"exit_description"=>"app instance exited".

Exit code 0 generally means your application shutdown cleanly. i.e. it
finished what it was doing and exited. If that's unexpected, I would
suggest that you increase the log level of the application to get more
details.


And after sometime another instance getting created automatically,
sucessfully running once and crashing with below logs. And this goes on for
sometime.
The system is restarting your application because it exited. The
expectation of an application that runs on CF is that it won't exit, ever.
Because of this if CF sees that one of your applications has stopped, it
assumes the app has "crashed" and will automatically try to restart it.

That's what's happening here. You app has exited, CF sees that and it's
helpfully trying to restart your app.

Dan




my manifest.yml:

name: RedisApp
no-route: true
memory: 512M
random-route: true
instances: 1
path: target/gs-messaging-redis-0.1.0.jar
services:
- redislite


cf logs RedisApp command output:

2015-09-25T11:58:51.85+0200 [DEA/1] OUT Starting app instance (index
0) with guid aec41933-ef0c-4d5b-8e67-da6729ca3005
2015-09-25T11:58:56.58+0200 [App/0] OUT
2015-09-25T11:58:56.58+0200 [App/0] OUT . ____ _
__ _ _
2015-09-25T11:58:56.58+0200 [App/0] OUT /\\ / ___'_ __ _ _(_)_ __
__ _ \ \ \ \
2015-09-25T11:58:56.58+0200 [App/0] OUT ( ( )\___ | '_ | '_| | '_ \/
_` | \ \ \ \
2015-09-25T11:58:56.58+0200 [App/0] OUT \\/ ___)| |_)| | | | | ||
(_| | ) ) ) )
2015-09-25T11:58:56.58+0200 [App/0] OUT ' |____| .__|_| |_|_|
|_\__, | / / / /
2015-09-25T11:58:56.58+0200 [App/0] OUT
=========|_|==============|___/=/_/_/_/
2015-09-25T11:58:56.58+0200 [App/0] OUT :: Spring Boot ::
(v1.2.6.RELEASE)
2015-09-25T11:58:56.69+0200 [App/0] OUT 2015-09-25 09:58:56.690 INFO
29 --- [ main] pertySourceApplicationContextInitializer : Adding
'cloud' PropertySource to ApplicationContext
2015-09-25T11:58:56.78+0200 [App/0] OUT 2015-09-25 09:58:56.786 INFO
29 --- [ main] nfigurationApplicationContextInitializer : Adding
cloud service auto-reconfiguration to ApplicationContext
2015-09-25T11:58:56.80+0200 [App/0] OUT 2015-09-25 09:58:56.804 INFO
29 --- [ main] hello.Application :
Starting Application on 18venf3o9v7 with PID 29 (/home/vcap/app started by
vcap in /home/vcap/app)
2015-09-25T11:58:56.86+0200 [App/0] OUT 2015-09-25 09:58:56.869 INFO
29 --- [ main] s.c.a.AnnotationConfigApplicationContext :
Refreshing
org.springframework.context.annotation.AnnotationConfigApplicationContext(a)6d2ca421:
startup date [Fri Sep 25 09:58:56 UTC 2015]; root of context h
ierarchy
2015-09-25T11:58:57.22+0200 [App/0] OUT 2015-09-25 09:58:57.223 WARN
29 --- [ main] .i.s.PathMatchingResourcePatternResolver :
Skipping
[/home/vcap/app/.java-buildpack/spring_auto_reconfiguration/spring_auto_reconfiguration-1.10.0_RELEASE.jar]
because it does not denote a directory
2015-09-25T11:58:57.78+0200 [App/0] OUT 2015-09-25 09:58:57.780 INFO
29 --- [ main] urceCloudServiceBeanFactoryPostProcessor :
Auto-reconfiguring beans of type javax.sql.DataSource
2015-09-25T11:58:57.79+0200 [App/0] OUT 2015-09-25 09:58:57.789 INFO
29 --- [ main] urceCloudServiceBeanFactoryPostProcessor : No
beans of type javax.sql.DataSource found. Skipping auto-reconfiguration.
2015-09-25T11:58:57.79+0200 [App/0] OUT 2015-09-25 09:58:57.794 INFO
29 --- [ main] edisCloudServiceBeanFactoryPostProcessor :
Auto-reconfiguring beans of type
org.springframework.data.redis.connection.RedisConnectionFactory
2015-09-25T11:58:57.90+0200 [App/0] OUT 2015-09-25 09:58:57.905 INFO
29 --- [ main] edisCloudServiceBeanFactoryPostProcessor :
Reconfigured bean redisConnectionFactory into singleton service connector
org.springframework.data.redis.connection.jedis.JedisConnectionFactory(a)74ca9fd4
2015-09-25T11:58:58.29+0200 [App/0] OUT 2015-09-25 09:58:58.298 INFO
29 --- [ main] o.s.j.e.a.AnnotationMBeanExporter :
Registering beans for JMX exposure on startup
2015-09-25T11:58:58.30+0200 [App/0] OUT 2015-09-25 09:58:58.308 INFO
29 --- [ main] o.s.c.support.DefaultLifecycleProcessor :
Starting beans in phase 2147483647
2015-09-25T11:58:58.37+0200 [App/0] OUT 2015-09-25 09:58:58.375 INFO
29 --- [ main] hello.Application : Started
Application in 2.491 seconds (JVM running for 3.401)
2015-09-25T11:58:58.37+0200 [App/0] OUT 2015-09-25 09:58:58.376 INFO
29 --- [ main] hello.Application : Sending
message...
2015-09-25T11:58:58.39+0200 [App/0] OUT 2015-09-25 09:58:58.398 INFO
29 --- [ container-2] hello.Receiver :
Received <Hello from Redis!>
2015-09-25T11:58:58.40+0200 [App/0] OUT 2015-09-25 09:58:58.400 INFO
29 --- [ Thread-2] s.c.a.AnnotationConfigApplicationContext : Closing
org.springframework.context.annotation.AnnotationConfigApplicationContext(a)6d2ca421:
startup date [Fri Sep 25 09:58:56 UTC 2015]; root of context hier
archy
2015-09-25T11:58:58.40+0200 [App/0] OUT 2015-09-25 09:58:58.401 INFO
29 --- [ Thread-2] o.s.c.support.DefaultLifecycleProcessor :
Stopping beans in phase 2147483647
2015-09-25T11:58:58.40+0200 [App/0] OUT 2015-09-25 09:58:58.404 INFO
29 --- [ Thread-2] o.s.j.e.a.AnnotationMBeanExporter :
Unregistering JMX-exposed beans on shutdown
2015-09-25T11:58:58.44+0200 [App/0] ERR
2015-09-25T11:58:58.49+0200 [API/0] OUT App instance exited with guid
aec41933-ef0c-4d5b-8e67-da6729ca3005 payload: {"cc_partition"=>"default",
"droplet"=>"aec41933-ef0c-4d5b-8e67-da6729ca3005",
"version"=>"8985cc3d-6aa2-4e34-a11c-f64289aeace3",
"instance"=>"0147066c534c4d3bb879fffd2c149529", "
index"=>0, "reason"=>"CRASHED", "exit_status"=>0, "exit_description"=>"app
instance exited", "crash_timestamp"=>1443175138}


postgres out of disk space

Matthias Ender <Matthias.Ender@...>
 

I have a cf-aws-tiny cf-boshrelease deployment, and it's been running well for over 4 months.
We have about 40 apps, with a couple of dozen of cf pushes each day.
Yesterday pushing apps became spotty and then impossible, with various errors.
Turned out the 100GB disk for the postgres instance on the data note was full.
I increased the disk size and things a running again.
But - what happened there? 100G and growing seems like awfully large database for a rather modest use.
And I'm worried it'll just happen again in a few months.

thanks,
Matthias


Instance crashing after running once. Error: "reason"=>"CRASHED", "exit_status"=>0, "exit_description"=>"app instance exited"

Zuba Al <myfakename90@...>
 

I've pushed the app which uses redis service (A sample app which simply send and receives a message thru Redis service). Instance is created at start and after successfully running once the instance getting crashed with error: "reason"=>"CRASHED", "exit_status"=>0, "exit_description"=>"app instance exited". And after sometime another instance getting created automatically, sucessfully running once and crashing with below logs. And this goes on for sometime.

my manifest.yml:

name: RedisApp
no-route: true
memory: 512M
random-route: true
instances: 1
path: target/gs-messaging-redis-0.1.0.jar
services:
- redislite


cf logs RedisApp command output:

2015-09-25T11:58:51.85+0200 [DEA/1] OUT Starting app instance (index 0) with guid aec41933-ef0c-4d5b-8e67-da6729ca3005
2015-09-25T11:58:56.58+0200 [App/0] OUT
2015-09-25T11:58:56.58+0200 [App/0] OUT . ____ _ __ _ _
2015-09-25T11:58:56.58+0200 [App/0] OUT /\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
2015-09-25T11:58:56.58+0200 [App/0] OUT ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
2015-09-25T11:58:56.58+0200 [App/0] OUT \\/ ___)| |_)| | | | | || (_| | ) ) ) )
2015-09-25T11:58:56.58+0200 [App/0] OUT ' |____| .__|_| |_|_| |_\__, | / / / /
2015-09-25T11:58:56.58+0200 [App/0] OUT =========|_|==============|___/=/_/_/_/
2015-09-25T11:58:56.58+0200 [App/0] OUT :: Spring Boot :: (v1.2.6.RELEASE)
2015-09-25T11:58:56.69+0200 [App/0] OUT 2015-09-25 09:58:56.690 INFO 29 --- [ main] pertySourceApplicationContextInitializer : Adding 'cloud' PropertySource to ApplicationContext
2015-09-25T11:58:56.78+0200 [App/0] OUT 2015-09-25 09:58:56.786 INFO 29 --- [ main] nfigurationApplicationContextInitializer : Adding cloud service auto-reconfiguration to ApplicationContext
2015-09-25T11:58:56.80+0200 [App/0] OUT 2015-09-25 09:58:56.804 INFO 29 --- [ main] hello.Application : Starting Application on 18venf3o9v7 with PID 29 (/home/vcap/app started by vcap in /home/vcap/app)
2015-09-25T11:58:56.86+0200 [App/0] OUT 2015-09-25 09:58:56.869 INFO 29 --- [ main] s.c.a.AnnotationConfigApplicationContext : Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext(a)6d2ca421: startup date [Fri Sep 25 09:58:56 UTC 2015]; root of context h
ierarchy
2015-09-25T11:58:57.22+0200 [App/0] OUT 2015-09-25 09:58:57.223 WARN 29 --- [ main] .i.s.PathMatchingResourcePatternResolver : Skipping [/home/vcap/app/.java-buildpack/spring_auto_reconfiguration/spring_auto_reconfiguration-1.10.0_RELEASE.jar] because it does not denote a directory
2015-09-25T11:58:57.78+0200 [App/0] OUT 2015-09-25 09:58:57.780 INFO 29 --- [ main] urceCloudServiceBeanFactoryPostProcessor : Auto-reconfiguring beans of type javax.sql.DataSource
2015-09-25T11:58:57.79+0200 [App/0] OUT 2015-09-25 09:58:57.789 INFO 29 --- [ main] urceCloudServiceBeanFactoryPostProcessor : No beans of type javax.sql.DataSource found. Skipping auto-reconfiguration.
2015-09-25T11:58:57.79+0200 [App/0] OUT 2015-09-25 09:58:57.794 INFO 29 --- [ main] edisCloudServiceBeanFactoryPostProcessor : Auto-reconfiguring beans of type org.springframework.data.redis.connection.RedisConnectionFactory
2015-09-25T11:58:57.90+0200 [App/0] OUT 2015-09-25 09:58:57.905 INFO 29 --- [ main] edisCloudServiceBeanFactoryPostProcessor : Reconfigured bean redisConnectionFactory into singleton service connector org.springframework.data.redis.connection.jedis.JedisConnectionFactory(a)74ca9fd4
2015-09-25T11:58:58.29+0200 [App/0] OUT 2015-09-25 09:58:58.298 INFO 29 --- [ main] o.s.j.e.a.AnnotationMBeanExporter : Registering beans for JMX exposure on startup
2015-09-25T11:58:58.30+0200 [App/0] OUT 2015-09-25 09:58:58.308 INFO 29 --- [ main] o.s.c.support.DefaultLifecycleProcessor : Starting beans in phase 2147483647
2015-09-25T11:58:58.37+0200 [App/0] OUT 2015-09-25 09:58:58.375 INFO 29 --- [ main] hello.Application : Started Application in 2.491 seconds (JVM running for 3.401)
2015-09-25T11:58:58.37+0200 [App/0] OUT 2015-09-25 09:58:58.376 INFO 29 --- [ main] hello.Application : Sending message...
2015-09-25T11:58:58.39+0200 [App/0] OUT 2015-09-25 09:58:58.398 INFO 29 --- [ container-2] hello.Receiver : Received <Hello from Redis!>
2015-09-25T11:58:58.40+0200 [App/0] OUT 2015-09-25 09:58:58.400 INFO 29 --- [ Thread-2] s.c.a.AnnotationConfigApplicationContext : Closing org.springframework.context.annotation.AnnotationConfigApplicationContext(a)6d2ca421: startup date [Fri Sep 25 09:58:56 UTC 2015]; root of context hier
archy
2015-09-25T11:58:58.40+0200 [App/0] OUT 2015-09-25 09:58:58.401 INFO 29 --- [ Thread-2] o.s.c.support.DefaultLifecycleProcessor : Stopping beans in phase 2147483647
2015-09-25T11:58:58.40+0200 [App/0] OUT 2015-09-25 09:58:58.404 INFO 29 --- [ Thread-2] o.s.j.e.a.AnnotationMBeanExporter : Unregistering JMX-exposed beans on shutdown
2015-09-25T11:58:58.44+0200 [App/0] ERR
2015-09-25T11:58:58.49+0200 [API/0] OUT App instance exited with guid aec41933-ef0c-4d5b-8e67-da6729ca3005 payload: {"cc_partition"=>"default", "droplet"=>"aec41933-ef0c-4d5b-8e67-da6729ca3005", "version"=>"8985cc3d-6aa2-4e34-a11c-f64289aeace3", "instance"=>"0147066c534c4d3bb879fffd2c149529", "
index"=>0, "reason"=>"CRASHED", "exit_status"=>0, "exit_description"=>"app instance exited", "crash_timestamp"=>1443175138}


Re: Proposal: Decomposing cf-release and Extracting Deployment Strategies

Mike Youngstrom <youngm@...>
 

Sounds good. Thanks for taking the time to discuss this with me.

Mike

On Mon, Sep 21, 2015 at 7:24 PM, Amit Gupta <agupta(a)pivotal.io> wrote:

This forces us to spread all clusterable nodes across 2 deploys and
certain jobs, like CC, use the job_name+index to uniquely identify a node

I believe they're planning on switching to guids for bosh job
identifiers. I saw in another thread you and Dmitriy discussed this. Any
other reasons for having unique job names we should know about?

How would you feel about the interface allowing for specifying
additional releases, jobs, and templates to be colocated on existing jobs,
along with property configuration for these things?

I don't quite follow what you are proposing here. Can you clarify?
What I mean is the tools we build for generating manifests will support
specifying inputs (probably in the form of a YAML file) that declares what
additional releases you want to add to the deployment, what additional jobs
you may want to add, what additional job templates you may want to colocate
with an existing job, and property configuration for those additional jobs
or colocated job templates. A common example is wanting to colocate some
monitoring agent on all the jobs, and providing some credential
configuration so it can pump metrics into some third party service. This
would be for things not already covered by the LAMB architecture.

Something like that would work for me as long as we were still able to
take advantage of the scripts/tooling in cf-deployment to manage the config
and templates we manage in lds-deployment.

Yes, that'd be the plan.

Cheers,
Amit


On Mon, Sep 21, 2015 at 2:41 PM, Mike Youngstrom <youngm(a)gmail.com> wrote:

Thanks for the response. See comments below:


Sensitive property management as part of manifest generation
(encrypted or acquired from an outside source)

How do you currently get these encrypted or external values into your
manifests? At manifest generation time, would you be able to generate a
stub on the fly from this source, and pass it into the manifest generation
script?
Yes, that would work fine. Just thought I'd call it out as something our
current solution does that we'd have to augment in cf-deployment.


If for some reason we are forced to fork a stock release we'd like to
be able to use that forked release we are building instead of the publicly
available one for manifest generation and release uploads, etc.

Yes, using the stock release will be the default option, but we will
support several other ways of specifying a release, including providing a
URL to a remote tarball, a path to a local release directory, a path to a
local tarball, and maybe a git URL and SHA.
Great!


The job names in each deployment must be unique across the
installation.

Why do the job names need to be unique across deployments?
This is because a single bosh cannot connect to multiple datacenters
which for us represent different availability zones. This forces us to
spread all clusterable nodes across 2 deploys and certain jobs, like CC,
use the job_name+index to uniquely identify a node [0]. Therefore if we
have 2 CCs deployed across 2 AZ we must have one job named
cloud_controller_az1 and the other named cloud_controller_az2. Does that
make sense? I recognize this is mostly the fault of a limitation in Bosh
but until bosh supports connection to multiple vsphere datacenters with a
single director we will need to account for it in our templatin.

[0]
https://github.com/cloudfoundry/cloud_controller_ng/blob/5257a8af6990e71cd1e34ae8978dfe4773b32826/bosh-templates/cloud_controller_worker_ctl.erb#L48

Occasionally we may wish to use some config from a stock release not
currently exposed in a cf-deployment template. I'd like to be sure there
is a way we can add that config, in a not hacky way, without waiting for a
PR to be accepted and subsequent release.

This would be ideal. Currently, a lot of complexity in manifest
generation is around, if you specify a certain value X, then you need to
make sure you specify values Y, Z, etc. in a compatible way. E.g. if you
have 3 etcd instances, then the value for the etcd.machines property needs
to have those 3 IPs. If you specify domain as "mydomain.com", then you
need to specify in other places that the UAA URL is "
https://uaa.mydomain.com". The hope is most of this complexity goes
away with BOSH Links (
https://github.com/cloudfoundry/bosh-notes/blob/master/links.md). My
hope is that, as the complexity goes away, we will have to maintain less
logic and will be able to comfortably expose more, if not all, of the
properties.
Great

We have our own internal bosh releases and config that we'll need to
merge in with the things cf-deployment is doing.

How would you feel about the interface allowing for specifying
additional releases, jobs, and templates to be colocated on existing jobs,
along with property configuration for these things?
I don't quite follow what you are proposing here. Can you clarify?


we'd like to augment this with our own release jobs and config that
we know to work with cf-deployment 250's and perhaps tag it as v250.lds

Would a workflow like this work for you: maintain an lds-deployment
repo, which includes cf-deployment as a submodule, and you can version
lds-deployment and update your submodule pointer to cf-deployment as you
see fit? lds-deployment will probably just need the cf-deployment
submodule, and a config file describing the "blessed" versions of the
non-stock releases you wish to add on. I know this is lacking details, but
does something along those lines sound like a reasonable workflow?
Something like that would work for me as long as we were still able to
take advantage of the scripts/tooling in cf-deployment to manage the config
and templates we manage in lds-deployment.

Thanks,
Mike




On Wed, Sep 16, 2015 at 3:06 PM, Mike Youngstrom <youngm(a)gmail.com>
wrote:

Another situation we have that you may want to keep in mind while
developing cf-deployment:

* We are using vsphere and currently we have a cf installation with 2
AZ using 2 separate vsphere "Datacenters" (more details:
https://github.com/cloudfoundry/bosh-notes/issues/7). This means we
have a CF installation that is actually made up of 2 deployments. So, we
need to generate a manifest for az1 and another for az2. The job names in
each deployment must be unique across the installation (e.g.
cloud_controller_az1 and cloud_controller_az2) would be the cc job names in
each deployment.

Mike

On Wed, Sep 16, 2015 at 3:38 PM, Mike Youngstrom <youngm(a)gmail.com>
wrote:

Here are some of the examples:

* Sensitive property management as part of manifest generation
(encrypted or acquired from an outside source)

* We have our own internal bosh releases and config that we'll need to
merge in with the things cf-deployment is doing. For example, if
cf-deployment tags v250 as including Diego 3333 and etcd 34 with given
templates perhaps we'd like to augment this with our own release jobs and
config that we know to work with cf-deployment 250's and perhaps tag it as
v250.lds and that becomes what we use to generate our manifests and upload
releases.

* Occasionally we may wish to use some config from a stock release not
currently exposed in a cf-deployment template. I'd like to be sure there
is a way we can add that config, in a not hacky way, without waiting for a
PR to be accepted and subsequent release.

* If for some reason we are forced to fork a stock release we'd like
to be able to use that forked release we are building instead of the
publicly available one for manifest generation and release uploads, etc.

Does that help?

Mike



On Tue, Sep 15, 2015 at 9:50 PM, Amit Gupta <agupta(a)pivotal.io> wrote:

Thanks for the feedback Mike!

Can you tell us more specifically what sort of extensions you need?
It would be great if cf-deployment provided an interface that could serve
the needs of essentially all operators of CF.

Thanks,
Amit

On Tue, Sep 15, 2015 at 4:02 PM, Mike Youngstrom <youngm(a)gmail.com>
wrote:

This is great stuff! My organization currently maintains our own
custom ways to generate manifests, include secure properties, and manage
release versions.

We would love to base the next generation of our solution on
cf-deployment. Have you put any thought into how others might customize or
extend cf-deployment? Our needs are very similar to yours just sometimes a
little different.

Perhaps a private fork periodically merged with a known good release
combination (tag) might be appropriate? Or some way to include the same
tools into a wholly private repo?

Mike


On Tue, Sep 8, 2015 at 1:22 PM, Amit Gupta <agupta(a)pivotal.io>
wrote:

Hi all,

The CF OSS Release Integration team (casually referred to as the
"MEGA team") is trying to solve a lot of tightly interrelated problems, and
make many of said problems less interrelated. It is difficult to address
just one issue without touching the others, so the following proposal
addresses several issues, but the most important ones are:

* decompose cf-release into many independently manageable,
independently testable, independently usable releases
* separate manifest generation strategies from the release source,
paving the way for Diego to be part of the standard deployment

This proposal will outline a picture of how manifest generation
will work in a unified manner in development, test, and integration
environments. It will also outline a picture of what each release’s test
pipelines will look like, how they will feed into a common integration
environment, and how feedback from the integration environment will feed
back into the test environments. Finally, it will propose a picture for
what the integration environment will look like, and how we get from the
current integration environment to where we want to be.

For further details, please feel free to view and comment here:


https://docs.google.com/document/d/1Viga_TzUB2nLxN_ILqksmUiILM1hGhq7MBXxgLaUOkY

Thanks,
Amit, CF OSS Release Integration team


Re: Running the app test suite within the CATs, and the admin_buildpack_lifecycle_test is failing

CF Runtime
 

You can also check out the v208 tag of cf-release, then run the
acceptance-tests from src/github.com/cloudfoundry/cf-acceptance-tests

Joseph
CF Release Integration Team

On Thu, Sep 24, 2015 at 2:53 PM, Christopher Piraino <cpiraino(a)pivotal.io>
wrote:

Jordan,

The Cloud Foundry bosh release comes with an errand called
"acceptance_tests" that contains the version of CATs which that version of
CF was tested with. You can run these by doing "bosh run errand
acceptance_tests".

There are also some manifest properties that you might need to set for the
CATs to run correctly. The list of all possible properties for the
acceptance_tests errand can be found here:
https://github.com/cloudfoundry/cf-release/blob/develop/jobs/acceptance-tests/spec
.


- Chris Piraino

On Thu, Sep 24, 2015 at 11:27 AM, Jordan Collier <jordanicollier(a)gmail.com
wrote:
I was unclear on what I am asking, the real question is as follows:

What is the best way to run the apps test suite within the CATS on an
older version of cloud foundry? (for example I am running these tests on
version 208)


Re: Running the app test suite within the CATs, and the admin_buildpack_lifecycle_test is failing

Christopher Piraino <cpiraino@...>
 

Jordan,

The Cloud Foundry bosh release comes with an errand called
"acceptance_tests" that contains the version of CATs which that version of
CF was tested with. You can run these by doing "bosh run errand
acceptance_tests".

There are also some manifest properties that you might need to set for the
CATs to run correctly. The list of all possible properties for the
acceptance_tests errand can be found here:
https://github.com/cloudfoundry/cf-release/blob/develop/jobs/acceptance-tests/spec
.


- Chris Piraino

On Thu, Sep 24, 2015 at 11:27 AM, Jordan Collier <jordanicollier(a)gmail.com>
wrote:

I was unclear on what I am asking, the real question is as follows:

What is the best way to run the apps test suite within the CATS on an
older version of cloud foundry? (for example I am running these tests on
version 208)


Re: Environment variables with special characters not handled correctly?

Jonas Rosland
 

Hi Daniel and Dieu,

Finally after much trial and error I finally got it working. I created a user-provided service and then called on it from my application. I've documented the steps for anyone else wanting to know how to work with these variables (clearer documentation with examples maybe?).

Here's the documentation and example application: https://gist.github.com/jonasrosland/08b5758eaa9098a81cf8

Thanks for all your help!

Best regards,
Jonas Rosland