I have a cf-aws-tiny cf-boshrelease deployment, and it's been running well for over 4 months.
We have about 40 apps, with a couple of dozen of cf pushes each day.
Yesterday pushing apps became spotty and then impossible, with various errors.
Turned out the 100GB disk for the postgres instance on the data note was full.
I increased the disk size and things a running again.
But - what happened there? 100G and growing seems like awfully large database for a rather modest use.
And I'm worried it'll just happen again in a few months.

thanks,
Matthias

Just a thought: How large is your events table? Is it getting pruned at the
correct interval for you?

I believe events, app usage events, and audit events are pruned when
they're more than 31 days old default. You could change the cc properties
in the deployment manifest to make that a little more aggressive if that's
the problem.

On Fri, Sep 25, 2015 at 7:22 AM, Matthias Ender <Matthias.Ender(a)sas.com>
wrote:

I have a cf-aws-tiny cf-boshrelease deployment, and it’s been running well
for over 4 months.

We have about 40 apps, with a couple of dozen of cf pushes each day.

Yesterday pushing apps became spotty and then impossible, with various
errors.

Turned out the 100GB disk for the postgres instance on the data note was
full.

I increased the disk size and things a running again.

But – what happened there? 100G and growing seems like awfully large
database for a rather modest use.

And I’m worried it’ll just happen again in a few months.



thanks,

Matthias

--
Matthew Sykes
matthew.sykes(a)gmail.com

Does your database occupy this 100 GB? Or something else like logs?

Aleksey Zalesov | CloudFoundry Engineer | Altoros
Tel: (617) 841-2121 ext. 5707 | Toll free: 855-ALTOROS
Fax: (866) 201-3646 | Skype: aleksey_zalesov
www.altoros.com <http://www.altoros.com/> | blog.altoros.com <http://blog.altoros.com/> | twitter.com/altoros <http://twitter.com/altoros>

ah, it’s not postgres:

90M postgres

it’s actually:

94G shared/run.10.10.2.37.xip.io-cc-droplets

Does it keep a copy of each artifact ever pushed?
Or is this part of the event storage, controlled by these properties:

app_events:
cutoff_age_in_days: 31
app_usage_events:
cutoff_age_in_days: 31
audit_events:
cutoff_age_in_days: 31

??


From: Aleksey Zalesov [mailto:aleksey.zalesov(a)altoros.com]
Sent: Friday, September 25, 2015 12:23 PM
To: Discussions about Cloud Foundry projects and the system overall. <cf-dev(a)lists.cloudfoundry.org>
Subject: [cf-dev] Re: postgres out of disk space

Does your database occupy this 100 GB? Or something else like logs?

Aleksey Zalesov | CloudFoundry Engineer | Altoros
Tel: (617) 841-2121 ext. 5707 | Toll free: 855-ALTOROS
Fax: (866) 201-3646 | Skype: aleksey_zalesov
www.altoros.com<http://www.altoros.com/> | blog.altoros.com<http://blog.altoros.com/> | twitter.com/altoros<http://twitter.com/altoros>

On 25 Sep 2015, at 14:22, Matthias Ender <Matthias.Ender(a)sas.com<mailto:Matthias.Ender(a)sas.com>> wrote:

I have a cf-aws-tiny cf-boshrelease deployment, and it’s been running well for over 4 months.
We have about 40 apps, with a couple of dozen of cf pushes each day.
Yesterday pushing apps became spotty and then impossible, with various errors.
Turned out the 100GB disk for the postgres instance on the data note was full.
I increased the disk size and things a running again.
But – what happened there? 100G and growing seems like awfully large database for a rather modest use.
And I’m worried it’ll just happen again in a few months.

thanks,
Matthias

Droplets and app packages should be pruned when the applications they're
associated with are deleted. Only one or two versions of the droplet are
ever held and only copy of the application bits.

There's another blob store that holds application resources for resource
match flow when uploading an app. Those resources are not pruned so you
should make sure that you disable that in your manifest if you don't want
that behavior.

The droplet blob store is clearly your issue - not events.

On Fri, Sep 25, 2015 at 2:53 PM, Matthias Ender <Matthias.Ender(a)sas.com>
wrote:

ah, it’s not postgres:



90M postgres



it’s actually:



94G shared/run.10.10.2.37.xip.io-cc-droplets



Does it keep a copy of each artifact ever pushed?

Or is this part of the event storage, controlled by these properties:



app_events:

cutoff_age_in_days: 31

app_usage_events:

cutoff_age_in_days: 31

audit_events:

cutoff_age_in_days: 31



??





*From:* Aleksey Zalesov [mailto:aleksey.zalesov(a)altoros.com]
*Sent:* Friday, September 25, 2015 12:23 PM
*To:* Discussions about Cloud Foundry projects and the system overall. <
cf-dev(a)lists.cloudfoundry.org>
*Subject:* [cf-dev] Re: postgres out of disk space



Does your database occupy this 100 GB? Or something else like logs?



Aleksey Zalesov | CloudFoundry Engineer | Altoros

Tel: (617) 841-2121 ext. 5707 | Toll free: 855-ALTOROS
Fax: (866) 201-3646 | Skype: aleksey_zalesov

www.altoros.com | blog.altoros.com | twitter.com/altoros



On 25 Sep 2015, at 14:22, Matthias Ender <Matthias.Ender(a)sas.com> wrote:



I have a cf-aws-tiny cf-boshrelease deployment, and it’s been running well
for over 4 months.

We have about 40 apps, with a couple of dozen of cf pushes each day.

Yesterday pushing apps became spotty and then impossible, with various
errors.

Turned out the 100GB disk for the postgres instance on the data note was
full.

I increased the disk size and things a running again.

But – what happened there? 100G and growing seems like awfully large
database for a rather modest use.

And I’m worried it’ll just happen again in a few months.



thanks,

Matthias

--
Matthew Sykes
matthew.sykes(a)gmail.com

cf-release v213 fixed a bug where buildpack caches were not getting cleaned
up properly. Are you running a version prior to 213?

If so, you might need to go into the
`shared/run.10.10.2.37.xip.io-cc-droplets` directory and delete the
`buildpack_cache` subdirectory. Deleting the contents of buildpack_cache
will not cause failures in the system, the caches will simply get generated
from scratch the next time apps stage.

If you upgrade to 213 or later, you can just call the api endpoint to clean
out the buildpack cache. You can find the details in the v213 release notes:
https://github.com/cloudfoundry/cf-release/releases/tag/v213

Joseph & Natalie
CF Release Integration Team

On Fri, Sep 25, 2015 at 11:53 AM, Matthias Ender <Matthias.Ender(a)sas.com>
wrote:

ah, it’s not postgres:



90M postgres



it’s actually:



94G shared/run.10.10.2.37.xip.io-cc-droplets



Does it keep a copy of each artifact ever pushed?

Or is this part of the event storage, controlled by these properties:



app_events:

cutoff_age_in_days: 31

app_usage_events:

cutoff_age_in_days: 31

audit_events:

cutoff_age_in_days: 31



??





*From:* Aleksey Zalesov [mailto:aleksey.zalesov(a)altoros.com]
*Sent:* Friday, September 25, 2015 12:23 PM
*To:* Discussions about Cloud Foundry projects and the system overall. <
cf-dev(a)lists.cloudfoundry.org>
*Subject:* [cf-dev] Re: postgres out of disk space



Does your database occupy this 100 GB? Or something else like logs?



Aleksey Zalesov | CloudFoundry Engineer | Altoros

Tel: (617) 841-2121 ext. 5707 | Toll free: 855-ALTOROS
Fax: (866) 201-3646 | Skype: aleksey_zalesov

www.altoros.com | blog.altoros.com | twitter.com/altoros



On 25 Sep 2015, at 14:22, Matthias Ender <Matthias.Ender(a)sas.com> wrote:



I have a cf-aws-tiny cf-boshrelease deployment, and it’s been running well
for over 4 months.

We have about 40 apps, with a couple of dozen of cf pushes each day.

Yesterday pushing apps became spotty and then impossible, with various
errors.

Turned out the 100GB disk for the postgres instance on the data note was
full.

I increased the disk size and things a running again.

But – what happened there? 100G and growing seems like awfully large
database for a rather modest use.

And I’m worried it’ll just happen again in a few months.



thanks,

Matthias

It will be handy to have a configuration option for CC to prune unused droplets after 7 days of inactivity, for example.

Droplets are important, and should not be removed. If an app is deleted,
the droplet will also be deleted. If a droplet exists, that means it is
still needed.

If the app needs to be scaled up or moved to a different runner, the
droplet is reused. In theory droplets could be recreated from the package
if it were deleted, but there is no automated behavior for this. You would
need to `restage` the app to force the system to generate a new droplet
from the existing package.

Joseph
CF Release Integration Team

On Sat, Sep 26, 2015 at 12:24 AM, Zalesov Aleksey <
aleksey.zalesov(a)altoros.com> wrote:

It will be handy to have a configuration option for CC to prune unused
droplets after 7 days of inactivity, for example.

On 26 Sep 2015, at 01:47, CF Runtime <cfruntime(a)gmail.com> wrote:

cf-release v213 fixed a bug where buildpack caches were not getting
cleaned up properly. Are you running a version prior to 213?

If so, you might need to go into the
`shared/run.10.10.2.37.xip.io-cc-droplets` directory and delete the
`buildpack_cache` subdirectory. Deleting the contents of buildpack_cache
will not cause failures in the system, the caches will simply get generated
from scratch the next time apps stage.

If you upgrade to 213 or later, you can just call the api endpoint to
clean out the buildpack cache. You can find the details in the v213 release
notes:
https://github.com/cloudfoundry/cf-release/releases/tag/v213

Joseph & Natalie
CF Release Integration Team

On Fri, Sep 25, 2015 at 11:53 AM, Matthias Ender <Matthias.Ender(a)sas.com>
wrote:

ah, it’s not postgres:



90M postgres



it’s actually:



94G shared/run.10.10.2.37.xip.io-cc-droplets



Does it keep a copy of each artifact ever pushed?

Or is this part of the event storage, controlled by these properties:



app_events:

cutoff_age_in_days: 31

app_usage_events:

cutoff_age_in_days: 31

audit_events:

cutoff_age_in_days: 31



??





*From:* Aleksey Zalesov [mailto:aleksey.zalesov(a)altoros.com]
*Sent:* Friday, September 25, 2015 12:23 PM
*To:* Discussions about Cloud Foundry projects and the system overall. <
cf-dev(a)lists.cloudfoundry.org>
*Subject:* [cf-dev] Re: postgres out of disk space



Does your database occupy this 100 GB? Or something else like logs?



Aleksey Zalesov | CloudFoundry Engineer | Altoros

Tel: (617) 841-2121 ext. 5707 | Toll free: 855-ALTOROS
Fax: (866) 201-3646 | Skype: aleksey_zalesov

www.altoros.com | blog.altoros.com | twitter.com/altoros



On 25 Sep 2015, at 14:22, Matthias Ender <Matthias.Ender(a)sas.com> wrote:



I have a cf-aws-tiny cf-boshrelease deployment, and it’s been running
well for over 4 months.

We have about 40 apps, with a couple of dozen of cf pushes each day.

Yesterday pushing apps became spotty and then impossible, with various
errors.

Turned out the 100GB disk for the postgres instance on the data note was
full.

I increased the disk size and things a running again.

But – what happened there? 100G and growing seems like awfully large
database for a rather modest use.

And I’m worried it’ll just happen again in a few months.



thanks,

Matthias

thank you all!
I’m indeed running v212.
Deleting /var/vcap/store/shared/run.10.10.2.37.xip.io-cc-droplets/buildpack_cache
did the trick.


From: CF Runtime [mailto:cfruntime(a)gmail.com]
Sent: Friday, September 25, 2015 6:47 PM
To: Discussions about Cloud Foundry projects and the system overall. <cf-dev(a)lists.cloudfoundry.org>
Subject: [cf-dev] Re: Re: Re: postgres out of disk space
Importance: High

cf-release v213 fixed a bug where buildpack caches were not getting cleaned up properly. Are you running a version prior to 213?

If so, you might need to go into the `shared/run.10.10.2.37.xip.io-cc-droplets` directory and delete the `buildpack_cache` subdirectory. Deleting the contents of buildpack_cache will not cause failures in the system, the caches will simply get generated from scratch the next time apps stage.

If you upgrade to 213 or later, you can just call the api endpoint to clean out the buildpack cache. You can find the details in the v213 release notes:
https://github.com/cloudfoundry/cf-release/releases/tag/v213

Joseph & Natalie
CF Release Integration Team

On Fri, Sep 25, 2015 at 11:53 AM, Matthias Ender <Matthias.Ender(a)sas.com<mailto:Matthias.Ender(a)sas.com>> wrote:
ah, it’s not postgres:

90M postgres

it’s actually:

94G shared/run.10.10.2.37.xip.io-cc-droplets

Does it keep a copy of each artifact ever pushed?
Or is this part of the event storage, controlled by these properties:

app_events:
cutoff_age_in_days: 31
app_usage_events:
cutoff_age_in_days: 31
audit_events:
cutoff_age_in_days: 31

??


From: Aleksey Zalesov [mailto:aleksey.zalesov(a)altoros.com<mailto:aleksey.zalesov(a)altoros.com>]
Sent: Friday, September 25, 2015 12:23 PM
To: Discussions about Cloud Foundry projects and the system overall. <cf-dev(a)lists.cloudfoundry.org<mailto:cf-dev(a)lists.cloudfoundry.org>>
Subject: [cf-dev] Re: postgres out of disk space

Does your database occupy this 100 GB? Or something else like logs?

Aleksey Zalesov | CloudFoundry Engineer | Altoros
Tel: (617) 841-2121 ext. 5707<tel:%28617%29%20841-2121%20ext.%205707> | Toll free: 855-ALTOROS
Fax: (866) 201-3646<tel:%28866%29%20201-3646> | Skype: aleksey_zalesov
www.altoros.com<http://www.altoros.com/> | blog.altoros.com<http://blog.altoros.com/> | twitter.com/altoros<http://twitter.com/altoros>

On 25 Sep 2015, at 14:22, Matthias Ender <Matthias.Ender(a)sas.com<mailto:Matthias.Ender(a)sas.com>> wrote:

I have a cf-aws-tiny cf-boshrelease deployment, and it’s been running well for over 4 months.
We have about 40 apps, with a couple of dozen of cf pushes each day.
Yesterday pushing apps became spotty and then impossible, with various errors.
Turned out the 100GB disk for the postgres instance on the data note was full.
I increased the disk size and things a running again.
But – what happened there? 100G and growing seems like awfully large database for a rather modest use.
And I’m worried it’ll just happen again in a few months.

thanks,
Matthias