Purge files on NFS or S3?


CF Runtime
 

Hi John,

James is correct, the resources do not get cleaned up. The system does not
have any runtime dependencies on any of them however. They are used when
pushing an app to prevent the user from having to push a large file that
the system has already seen.

You should be able to delete the contents of the resources blobstore. This
is the path defined by
properties.cc.resource_pool.resource_directory_key in your deployment
manifest.

Removing anything in the buildpacks, droplets, or packages folders can
cause problems in the running system.

- CF Runtime Team

john, i think the resource files may grow forever right now without
intervention.

i'm pretty confident that when apps are deleted that their droplets are
deleted with them and that proper garbage collection occurs with that.

i'm unaware of any NFS file system to s3 blob migration. you would need to
update the CC_DB references too i'm pretty sure. i'm interested if you
find
out more.

On Tue, May 5, 2015 at 1:14 PM, John Wong <gokoproject at gmail.com>
wrote:

> Hi
> I just looked at our disk usage on NFS server. We have used like 200G
so
far, and I wonder if there's a systematic way to purge files we don't
need
(or how do I know I don't need them)?

Similarly, if I were to replace NFS server with S3 instead, does the
existing process (if any) work with S3?

Thanks.

_______________________________________________
cf-dev mailing list
cf-dev at lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


--
Thank you,

James Bayer


Jon Price
 

Make sure you only delete the resource files, not everything...

Jon Price
Intel Corp.

On May 11, 2015 10:05 PM, Dieu Cao <dcao(a)pivotal.io> wrote:
An option could be to just delete all the resource files on the blobstore. The effect would be that for binaries that would have been matched, they would be uploaded again on the first new push including those binaries.

On Monday, May 11, 2015, John Wong <gokoproject(a)gmail.com<mailto:gokoproject(a)gmail.com>> wrote:
Hi all

Thanks. No I was just curious if there was a way to identify what to remove in the blobstore because I was surprised the size of my blobstore at this point. I will check what's in there (maybe James is right it is mostly resource files). I am currently using NFS. I can build a CF with S3 as my blobstore.

John


On Mon, May 11, 2015 at 11:36 AM, Chad Woolley <thewoolleyman(a)gmail.com> wrote:
Not sure if this is what you need, but you can manually sync + delete files from a local filesystem (including NFS mount) to/from S3:

http://s3tools.org/s3cmd-sync

... with `—delete-removed` option

-- Chad


On Sat, May 9, 2015 at 12:19 AM, James Bayer <jbayer(a)pivotal.io> wrote:

john, i think the resource files may grow forever right now without intervention.

i'm pretty confident that when apps are deleted that their droplets are deleted with them and that proper garbage collection occurs with that.

i'm unaware of any NFS file system to s3 blob migration. you would need to update the CC_DB references too i'm pretty sure. i'm interested if you find out more.

On Tue, May 5, 2015 at 1:14 PM, John Wong <gokoproject(a)gmail.com> wrote:

Hi

I just looked at our disk usage on NFS server. We have used like 200G so far, and I wonder if there's a systematic way to purge files we don't need (or how do I know I don't need them)?

Similarly, if I were to replace NFS server with S3 instead, does the existing process (if any) work with S3?

Thanks.

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


--
Thank you,

James Bayer

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


Dieu Cao <dcao@...>
 

An option could be to just delete all the resource files on the blobstore.
The effect would be that for binaries that would have been matched, they
would be uploaded again on the first new push including those binaries.

On Monday, May 11, 2015, John Wong <gokoproject(a)gmail.com> wrote:

Hi all

Thanks. No I was just curious if there was a way to identify what to
remove in the blobstore because I was surprised the size of my blobstore at
this point. I will check what's in there (maybe James is right it is mostly
resource files). I am currently using NFS. I can build a CF with S3 as my
blobstore.

John


On Mon, May 11, 2015 at 11:36 AM, Chad Woolley <thewoolleyman(a)gmail.com
<javascript:_e(%7B%7D,'cvml','thewoolleyman(a)gmail.com');>> wrote:

Not sure if this is what you need, but you can manually sync + delete
files from a local filesystem (including NFS mount) to/from S3:

http://s3tools.org/s3cmd-sync

... with `—delete-removed` option

-- Chad


On Sat, May 9, 2015 at 12:19 AM, James Bayer <jbayer(a)pivotal.io
<javascript:_e(%7B%7D,'cvml','jbayer(a)pivotal.io');>> wrote:

john, i think the resource files may grow forever right now without
intervention.

i'm pretty confident that when apps are deleted that their droplets are
deleted with them and that proper garbage collection occurs with that.

i'm unaware of any NFS file system to s3 blob migration. you would need
to update the CC_DB references too i'm pretty sure. i'm interested if you
find out more.

On Tue, May 5, 2015 at 1:14 PM, John Wong <gokoproject(a)gmail.com
<javascript:_e(%7B%7D,'cvml','gokoproject(a)gmail.com');>> wrote:

Hi

I just looked at our disk usage on NFS server. We have used like 200G
so far, and I wonder if there's a systematic way to purge files we don't
need (or how do I know I don't need them)?

Similarly, if I were to replace NFS server with S3 instead, does the
existing process (if any) work with S3?

Thanks.

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
<javascript:_e(%7B%7D,'cvml','cf-dev(a)lists.cloudfoundry.org');>
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


--
Thank you,

James Bayer

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
<javascript:_e(%7B%7D,'cvml','cf-dev(a)lists.cloudfoundry.org');>
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
<javascript:_e(%7B%7D,'cvml','cf-dev(a)lists.cloudfoundry.org');>
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


John Wong
 

Hi all

Thanks. No I was just curious if there was a way to identify what to remove
in the blobstore because I was surprised the size of my blobstore at this
point. I will check what's in there (maybe James is right it is mostly
resource files). I am currently using NFS. I can build a CF with S3 as my
blobstore.

John


On Mon, May 11, 2015 at 11:36 AM, Chad Woolley <thewoolleyman(a)gmail.com>
wrote:

Not sure if this is what you need, but you can manually sync + delete
files from a local filesystem (including NFS mount) to/from S3:

http://s3tools.org/s3cmd-sync

... with `—delete-removed` option

-- Chad


On Sat, May 9, 2015 at 12:19 AM, James Bayer <jbayer(a)pivotal.io> wrote:

john, i think the resource files may grow forever right now without
intervention.

i'm pretty confident that when apps are deleted that their droplets are
deleted with them and that proper garbage collection occurs with that.

i'm unaware of any NFS file system to s3 blob migration. you would need
to update the CC_DB references too i'm pretty sure. i'm interested if you
find out more.

On Tue, May 5, 2015 at 1:14 PM, John Wong <gokoproject(a)gmail.com> wrote:

Hi

I just looked at our disk usage on NFS server. We have used like 200G
so far, and I wonder if there's a systematic way to purge files we don't
need (or how do I know I don't need them)?

Similarly, if I were to replace NFS server with S3 instead, does the
existing process (if any) work with S3?

Thanks.

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


--
Thank you,

James Bayer

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


Chad Woolley <thewoolleyman@...>
 

Not sure if this is what you need, but you can manually sync + delete files
from a local filesystem (including NFS mount) to/from S3:

http://s3tools.org/s3cmd-sync

... with `—delete-removed` option

-- Chad

On Sat, May 9, 2015 at 12:19 AM, James Bayer <jbayer(a)pivotal.io> wrote:

john, i think the resource files may grow forever right now without
intervention.

i'm pretty confident that when apps are deleted that their droplets are
deleted with them and that proper garbage collection occurs with that.

i'm unaware of any NFS file system to s3 blob migration. you would need
to update the CC_DB references too i'm pretty sure. i'm interested if you
find out more.

On Tue, May 5, 2015 at 1:14 PM, John Wong <gokoproject(a)gmail.com> wrote:

Hi

I just looked at our disk usage on NFS server. We have used like 200G so
far, and I wonder if there's a systematic way to purge files we don't need
(or how do I know I don't need them)?

Similarly, if I were to replace NFS server with S3 instead, does the
existing process (if any) work with S3?

Thanks.

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


--
Thank you,

James Bayer

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


James Bayer
 

john, i think the resource files may grow forever right now without
intervention.

i'm pretty confident that when apps are deleted that their droplets are
deleted with them and that proper garbage collection occurs with that.

i'm unaware of any NFS file system to s3 blob migration. you would need to
update the CC_DB references too i'm pretty sure. i'm interested if you find
out more.

On Tue, May 5, 2015 at 1:14 PM, John Wong <gokoproject(a)gmail.com> wrote:

Hi

I just looked at our disk usage on NFS server. We have used like 200G so
far, and I wonder if there's a systematic way to purge files we don't need
(or how do I know I don't need them)?

Similarly, if I were to replace NFS server with S3 instead, does the
existing process (if any) work with S3?

Thanks.

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

--
Thank you,

James Bayer


John Wong
 

Hi

I just looked at our disk usage on NFS server. We have used like 200G so
far, and I wonder if there's a systematic way to purge files we don't need
(or how do I know I don't need them)?

Similarly, if I were to replace NFS server with S3 instead, does the
existing process (if any) work with S3?

Thanks.