Re: diego: disk filling up over time


Eric Malm <emalm@...>
 

Hi, Tom,

Thanks for asking about this. Could you provide some more details about
your deployment?

- What are the exact errors you're seeing when CF users are trying to make
containers? The errors from CF CLI logs or rep/garden logs would be great
to see.
- What's the total amount of disk space available on the volume attached to
/var/vcap/data? You should be able to see this from `df` command output.
- How much space is the rep configured to allocate for its executor cache?
Is it the default 10GB provided by the rep's job spec in
https://github.com/cloudfoundry-incubator/diego-release/blob/v0.1398.0/jobs/rep/spec#L70-L72?
How much disk is actually used in /var/vcap/data/executor_cache (based on
reporting from `du`, say)?
- How much space have you directed garden-linux to allocate for its btrfs
store? This is provided via the diego.garden-linux.btrfs_store_size_mb BOSH
property, and with Diego 0.1398.0 I believe it has to be specified
explicitly. Also, how much space is actually used in the btrfs filesystem?
You should be able to inspect this with the btrfs tools available on the
cell VM in '/var/vcap/packages/btrfs-tools/bin'. I think running
`/var/vcap/packages/btrfs-tools/bin/btrfs filesystem usage
/var/vcap/data/garden-linux/btrfs_graph` should be a good starting point.

You may also find some useful information in the cf-dev thread from August
about overcommitting disk on Diego cells:
https://lists.cloudfoundry.org/archives/list/cf-dev(a)lists.cloudfoundry.org/thread/VBDM2TMHQSOFILSHRCV4G2CCPRBP5WKA/#VBDM2TMHQSOFILSHRCV4G2CCPRBP5WKA

Thanks,
Eric

On Wed, Nov 18, 2015 at 6:52 AM, Tom Sherrod <tom.sherrod(a)gmail.com> wrote:

diego release 0.1398.0

After a couple of weeks of dev, the cells end up filling their disks. Did
I miss a clean up job somewhere?
Currently, once pushes start failing, I get bosh to recreate the machine.

Other options?

Thanks,
Tom

Join cf-dev@lists.cloudfoundry.org to automatically receive all group messages.