Security Question --- Securely wipe data on warden container removal / destruction???


Chris K
 

Hi,

I have a few questions regarding the way data is removed when an application is removed and its corresponding warden container is destroyed. As the Cloud Foundry instance my company is using may be shared with multiple tenants, this is a very critical question for us to be answered.
From Cloud Foundry's GitHub repository I gathered the following information regarding the destruction process:

"When a container is destroyed -- either per user request, or automatically after being idle -- Warden first kills all unprivileged processes running inside the container. These processes first receive a TERM signal followed by a KILL if they haven't exited after a couple of seconds. When these processes have terminated, the root of the container's process tree is sent a KILL . Once all resources the container used have been released, its files are removed and it is considered destroyed." (Quote: https://github.com/cloudfoundry/warden/tree/master/warden)

According to this quote all files of the file system are removed before the resources can be used again. But how are they removed? Are they securely wiped, meaning all blocks are set to zero (or randomized)? And how is data removed from the RAM before it can be assigned to a new warden (i.e. new application).

In case the data is not being securely wiped, how much access does an application have towards the available memory? Is it for example possible to create files of arbitrary size and read / access them?

I'd be thankful for any kind of hints on this topic.

With Regards,
Chris


James Bayer
 

warden/DEAs keeps container file systems for a configured amount of time,
something like 1 hr before removing the containers, i believe with standard
removal tools.

diego cells and garden removes container file system immediately after they
are stopped by the user or the system. when using docker images, the
container images are cached in the garden graph directory and i'm not quite
sure of their cleanup / garbage collection life cycle.

On Wed, Aug 19, 2015 at 1:08 AM, Chris K <christopherkugler2(a)yahoo.de>
wrote:

Hi,

I have a few questions regarding the way data is removed when an
application is removed and its corresponding warden container is destroyed.
As the Cloud Foundry instance my company is using may be shared with
multiple tenants, this is a very critical question for us to be answered.
From Cloud Foundry's GitHub repository I gathered the following
information regarding the destruction process:

"When a container is destroyed -- either per user request, or
automatically after being idle -- Warden first kills all unprivileged
processes running inside the container. These processes first receive a
TERM signal followed by a KILL if they haven't exited after a couple of
seconds. When these processes have terminated, the root of the container's
process tree is sent a KILL . Once all resources the container used have
been released, its files are removed and it is considered destroyed."
(Quote: https://github.com/cloudfoundry/warden/tree/master/warden)

According to this quote all files of the file system are removed before
the resources can be used again. But how are they removed? Are they
securely wiped, meaning all blocks are set to zero (or randomized)? And how
is data removed from the RAM before it can be assigned to a new warden
(i.e. new application).

In case the data is not being securely wiped, how much access does an
application have towards the available memory? Is it for example possible
to create files of arbitrary size and read / access them?

I'd be thankful for any kind of hints on this topic.

With Regards,
Chris


--
Thank you,

James Bayer


Will Pragnell <wpragnell@...>
 

In the Docker image case, the filesystem layer specific to the container is
also deleted immediately when the container stops running (this is the same
for buildpack based apps on Diego/Garden). Lower layers in the image (i.e.
the pre-existing docker image, as pulled from the registry) are not
currently removed, even if not used in any other running containers.

In the coming weeks, we'll define and implement a strategy to remove unused
images, but the details aren't decided yet.

On 19 August 2015 at 14:57, James Bayer <jbayer(a)pivotal.io> wrote:

warden/DEAs keeps container file systems for a configured amount of time,
something like 1 hr before removing the containers, i believe with standard
removal tools.

diego cells and garden removes container file system immediately after
they are stopped by the user or the system. when using docker images, the
container images are cached in the garden graph directory and i'm not quite
sure of their cleanup / garbage collection life cycle.

On Wed, Aug 19, 2015 at 1:08 AM, Chris K <christopherkugler2(a)yahoo.de>
wrote:

Hi,

I have a few questions regarding the way data is removed when an
application is removed and its corresponding warden container is destroyed.
As the Cloud Foundry instance my company is using may be shared with
multiple tenants, this is a very critical question for us to be answered.
From Cloud Foundry's GitHub repository I gathered the following
information regarding the destruction process:

"When a container is destroyed -- either per user request, or
automatically after being idle -- Warden first kills all unprivileged
processes running inside the container. These processes first receive a
TERM signal followed by a KILL if they haven't exited after a couple of
seconds. When these processes have terminated, the root of the container's
process tree is sent a KILL . Once all resources the container used have
been released, its files are removed and it is considered destroyed."
(Quote: https://github.com/cloudfoundry/warden/tree/master/warden)

According to this quote all files of the file system are removed before
the resources can be used again. But how are they removed? Are they
securely wiped, meaning all blocks are set to zero (or randomized)? And how
is data removed from the RAM before it can be assigned to a new warden
(i.e. new application).

In case the data is not being securely wiped, how much access does an
application have towards the available memory? Is it for example possible
to create files of arbitrary size and read / access them?

I'd be thankful for any kind of hints on this topic.

With Regards,
Chris


--
Thank you,

James Bayer


James Bayer
 

this is a relevant tracker story to clean up after containers:
https://www.pivotaltracker.com/n/projects/1158420/stories/96458012

On Wed, Aug 19, 2015 at 8:28 AM, Will Pragnell <wpragnell(a)pivotal.io> wrote:

In the Docker image case, the filesystem layer specific to the container
is also deleted immediately when the container stops running (this is the
same for buildpack based apps on Diego/Garden). Lower layers in the image
(i.e. the pre-existing docker image, as pulled from the registry) are not
currently removed, even if not used in any other running containers.

In the coming weeks, we'll define and implement a strategy to remove
unused images, but the details aren't decided yet.

On 19 August 2015 at 14:57, James Bayer <jbayer(a)pivotal.io> wrote:

warden/DEAs keeps container file systems for a configured amount of time,
something like 1 hr before removing the containers, i believe with standard
removal tools.

diego cells and garden removes container file system immediately after
they are stopped by the user or the system. when using docker images, the
container images are cached in the garden graph directory and i'm not quite
sure of their cleanup / garbage collection life cycle.

On Wed, Aug 19, 2015 at 1:08 AM, Chris K <christopherkugler2(a)yahoo.de>
wrote:

Hi,

I have a few questions regarding the way data is removed when an
application is removed and its corresponding warden container is destroyed.
As the Cloud Foundry instance my company is using may be shared with
multiple tenants, this is a very critical question for us to be answered.
From Cloud Foundry's GitHub repository I gathered the following
information regarding the destruction process:

"When a container is destroyed -- either per user request, or
automatically after being idle -- Warden first kills all unprivileged
processes running inside the container. These processes first receive a
TERM signal followed by a KILL if they haven't exited after a couple of
seconds. When these processes have terminated, the root of the container's
process tree is sent a KILL . Once all resources the container used have
been released, its files are removed and it is considered destroyed."
(Quote: https://github.com/cloudfoundry/warden/tree/master/warden)

According to this quote all files of the file system are removed before
the resources can be used again. But how are they removed? Are they
securely wiped, meaning all blocks are set to zero (or randomized)? And how
is data removed from the RAM before it can be assigned to a new warden
(i.e. new application).

In case the data is not being securely wiped, how much access does an
application have towards the available memory? Is it for example possible
to create files of arbitrary size and read / access them?

I'd be thankful for any kind of hints on this topic.

With Regards,
Chris


--
Thank you,

James Bayer
--
Thank you,

James Bayer


Chris K
 

Hello again,

I'm sorry for having to revive this topic, but I'm still unaware about the deletion process.

[... ] i believe with standard removal tools.
Could you please specify the term "standard removal tool". What's the standard? Is standard just the deletion of the pointer pointing on files / segments, or is secure deletion state of the art?
I'd be thankful for any reference on documentation regarding this topic.

Thanks in advance.

Cheers Chris


Will Pragnell <wpragnell@...>
 

In Diego/Garden, container files are stored on btrfs subvolumes. When a
container is destroyed, the subvolume is removed with btrfs subvolume delete.
I don’t think this does anything particularly fancy, and I don’t think it
classifies as “secure deletion”.

On 17 September 2015 at 11:53, Chris K <christopherkugler2(a)yahoo.de> wrote:

Hello again,

I'm sorry for having to revive this topic, but I'm still unaware about the
deletion process.

[... ] i believe with standard removal tools.
Could you please specify the term "standard removal tool". What's the
standard? Is standard just the deletion of the pointer pointing on files /
segments, or is secure deletion state of the art?
I'd be thankful for any reference on documentation regarding this topic.

Thanks in advance.

Cheers Chris


Guillaume Berche
 

Chris, thanks for bringing up this important security topic.

In terms of secrets an app is handling and carrying, I'd think its code has
generally limited sensitivity (e.g credentials or API key secrets are
rather stored in env vars). I'd expect memory to be much more sensitive
(e.g. holding user data), as well as state handed over to data services (12
factor apps are unlikely to store much state on their ephemeral file
system).

So related to your question about securely wipping data upon app instance
deletion, it may be interesting to consider secure RAM wiping when an app
container exits (sometimes killed by the oomkiller leaving few opportunity
for the app itself to wipe out RAM before exit). See related discussions in
[1] [2] [3] [4]. Quickly searching the bosh stemcell builder, and bosh
tracker I could not find mention of gresec or pax linux kernel
packages/patches that could strengthen RAM wiping after an app instance
exits.

Will, do you know if is there plans to tackle such kernel hardening ?

Related to secrets stored on disk in data services (p-mysql, p-redis), the
services should be designed to not provide access to previous deleted
service instances when normally functionning. The secured data wiping might
be useful if ever the data service itself would get compromised so that an
attacker would not be able to access data from deleted service instances
after hand.

Guillaume.

[1]
http://security.stackexchange.com/questions/42179/is-there-any-linux-distro-or-kernel-patch-that-wipes-a-process-memory-space-afte
[2] https://github.com/coreos/bugs/issues/332#issuecomment-109293958
[3]
https://en.wikibooks.org/wiki/Grsecurity/Appendix/Grsecurity_and_PaX_Configuration_Options#Sanitize_all_freed_memory
[4]
https://blog.docker.com/2013/08/containers-docker-how-secure-are-they/#other-kernel-security-features

On Thu, Sep 17, 2015 at 1:38 PM, Will Pragnell <wpragnell(a)pivotal.io> wrote:

In Diego/Garden, container files are stored on btrfs subvolumes. When a
container is destroyed, the subvolume is removed with btrfs subvolume
delete. I don’t think this does anything particularly fancy, and I don’t
think it classifies as “secure deletion”.


On 17 September 2015 at 11:53, Chris K <christopherkugler2(a)yahoo.de>
wrote:

Hello again,

I'm sorry for having to revive this topic, but I'm still unaware about
the deletion process.

[... ] i believe with standard removal tools.
Could you please specify the term "standard removal tool". What's the
standard? Is standard just the deletion of the pointer pointing on files /
segments, or is secure deletion state of the art?
I'd be thankful for any reference on documentation regarding this topic.

Thanks in advance.

Cheers Chris


Will Pragnell <wpragnell@...>
 

Guillaume, I'm not aware of any plans for secure memory wiping
specifically, but I can say that another track of security work is one of
several candidates for the next phase of work on Garden after OCS/runC
integration is completed.

That said, such a change may fall outside the remit of the Garden team; it
may be a platform wide change that involves changes to the stemcell.

On 23 September 2015 at 13:28, Guillaume Berche <bercheg(a)gmail.com> wrote:

Chris, thanks for bringing up this important security topic.

In terms of secrets an app is handling and carrying, I'd think its code
has generally limited sensitivity (e.g credentials or API key secrets are
rather stored in env vars). I'd expect memory to be much more sensitive
(e.g. holding user data), as well as state handed over to data services (12
factor apps are unlikely to store much state on their ephemeral file
system).

So related to your question about securely wipping data upon app instance
deletion, it may be interesting to consider secure RAM wiping when an app
container exits (sometimes killed by the oomkiller leaving few opportunity
for the app itself to wipe out RAM before exit). See related discussions in
[1] [2] [3] [4]. Quickly searching the bosh stemcell builder, and bosh
tracker I could not find mention of gresec or pax linux kernel
packages/patches that could strengthen RAM wiping after an app instance
exits.

Will, do you know if is there plans to tackle such kernel hardening ?

Related to secrets stored on disk in data services (p-mysql, p-redis), the
services should be designed to not provide access to previous deleted
service instances when normally functionning. The secured data wiping might
be useful if ever the data service itself would get compromised so that an
attacker would not be able to access data from deleted service instances
after hand.

Guillaume.

[1]
http://security.stackexchange.com/questions/42179/is-there-any-linux-distro-or-kernel-patch-that-wipes-a-process-memory-space-afte
[2] https://github.com/coreos/bugs/issues/332#issuecomment-109293958
[3]
https://en.wikibooks.org/wiki/Grsecurity/Appendix/Grsecurity_and_PaX_Configuration_Options#Sanitize_all_freed_memory
[4]
https://blog.docker.com/2013/08/containers-docker-how-secure-are-they/#other-kernel-security-features


On Thu, Sep 17, 2015 at 1:38 PM, Will Pragnell <wpragnell(a)pivotal.io>
wrote:

In Diego/Garden, container files are stored on btrfs subvolumes. When a
container is destroyed, the subvolume is removed with btrfs subvolume
delete. I don’t think this does anything particularly fancy, and I don’t
think it classifies as “secure deletion”.


On 17 September 2015 at 11:53, Chris K <christopherkugler2(a)yahoo.de>
wrote:

Hello again,

I'm sorry for having to revive this topic, but I'm still unaware about
the deletion process.

[... ] i believe with standard removal tools.
Could you please specify the term "standard removal tool". What's the
standard? Is standard just the deletion of the pointer pointing on files /
segments, or is secure deletion state of the art?
I'd be thankful for any reference on documentation regarding this topic.

Thanks in advance.

Cheers Chris