Hi cf-dev, I wanted to write a quick email to make sure the community is aware that we are planning to remove the btrfs driver from Garden/Grootfs in the next Garden release (and to explain why).

Exposition:

- Grootfs is the rootfs management library used in Cloud Foundry. Initially Grootfs planned to use btrfs (a filesystem supporting fast snapshotting) as the main underlying filesystem for managing container layers.  

Conflict!

- Unfortunately as we attempted to run the btrfs-based Grootfs at scale we saw performance and reliability issues that we weren't able to overcome to our satisfaction. 

Rising Action:

- Based on our inability to successfully run the btrfs-based driver at production scale we instead moved to an Overlay-based implementation.  (We'd like to underline that these may not have been fundamental issues with btrfs and may rather have simply reflected our team's lack of skill in it-- but the situation was still that the team was not able to gain confidence in our ability to run btrfs at scale). 

- We kept the btrfs driver in the code with the intention of removing it before merging Groot back in to Garden / creating Grootfs 1.0. This did not happen: the option of opting in to btrfs survived. 

Climax:

- Grootfs has now been running in large environments for several months using the Overlay driver and we've been happy with its performance and stability.  

- The btrfs driver is still in the code, but we have very few tests around it, and feel uncomfortable with people running it, since we don't believe our team can support it. Functionally, we believe all users should be able to use the overlay driver instead (although we’re absolutely aware some users prefer btrfs as they have more knowledge of it in-house). 

- The team is [Planning to adopt Containerd][0] for creating and managing containers inside Garden, this should allow a better way of allowing consumers to support filesystems the garden team doesn’t have the bandwidth to directly test and support, by potentially allowing consumers to use the wide variety of upstream community supported drivers. 

Falling Action:

- We still have the btrfs driver in the code even though there’s a better solution and we don’t feel we can support it. This is storing up problems and technical debt and incurring a cost on all stories which touch the Grootfs code. We’ve decided given the team’s limited bandwidth, we need to stop doing this.  

Resolution:

- We intend to remove the btrfs driver in the next garden version (it should never have survived to a 1.0+ release, and we apologise for the confusion of removing it after this happened). 

- As far as we know, all consumers of the btrfs driver should be able to transition to the Overlay driver.

- Since garden's image plugin API is stable and backwards-compatible, users who wish to continue using btrfs - for example because they have skills in btrfs and prefer to support a btrfs solution rather than an Overlay solution - can use the previous version of the Grootfs release (or their own image plugin), but will not get updates/fixes.

- We've recently begun work on supporting using Containerd [0] to create and manage containers in Garden. Our hope is that when this work is complete users will be able to use any of the drivers the Containerd community provides, including btrfs (though we will likely still recommend, test and support only a specific default configuration). 

Please please please let us know if there is a reason the Overlay driver (or the ability to plug in your own image plugin or an old Grootfs version) does not work for you, or if you have other questions/concerns and we will try our best to find a good solution. We're on slack in the #garden channel, or reply to this email / email me directly.

[0]: https://lists.cloudfoundry.org/g/cf-dev/message/7699?p=,,,20,0,0,0::Created,,containerd,20,2,0,9499071

Thanks!

Julz