Addressing buildpack size


Mike Dalessio
 

Skinny buildpacks have been cut for go, nodejs, php, python and ruby
buildpacks.

| | current | previous |
|--------+---------+----------|
| go | 442MB | 633MB |
| nodejs | 69MB | 417MB |
| php | 804MB | 1.1GB |
| python | 454MB | 654MB |
| ruby | 365MB | 1.3GB |
|--------+---------+----------|
| total | 2.1GB | 4.1GB |

for an aggregate 51% reduction in size. Details follow.
Next Steps

I recognize that every cloud operator may have a different policy on what
versions of interpreters and libraries they want to support, based on the
specific requirements of their users.

These buildpacks reflect a "bare mininum" policy for a cloud to be
operable, and I do not expect these buildpacks to be adopted as-is by many
operators.

These buildpacks have not yet been added to cf-release, specifically so
that the community can prepare their own buildpacks if necessary.

Over the next few days, the buildpacks core team will ship documentation
and tooling to assist you in packaging specific dependencies for your
instance of CF. I'll start a new thread on this list early next week to
communicate this information.
Call to Action

In the meantime, please think about whether the policy implemented in these
buildpacks ("last two patches (or teenies) on all supported major.minor
releases") is suitable for your users; and if not, think about what
dependencies you'll ideally be supporting.
go-buildpack v1.3.0

Release notes are here
<https://github.com/cloudfoundry/go-buildpack/releases/tag/v1.3.0>.

Size reduced 30% from 633MB
<https://github.com/cloudfoundry/go-buildpack/releases/tag/v1.2.0> to 442MB
<https://github.com/cloudfoundry/go-buildpack/releases/tag/v1.3.0>.

Supports (full manifest here
<https://github.com/cloudfoundry/go-buildpack/blob/v1.3.0/manifest.yml>):

- golang 1.4.{1,2}
- golang 1.3.{2,3}
- golang 1.2.{1,2}
- golang 1.1.{1,2}

nodejs-buildpack v1.3.0

Full release notes are here
<https://github.com/cloudfoundry/nodejs-buildpack/releases/tag/v1.3.0>.

Size reduced 83% from 417MB
<https://github.com/cloudfoundry/nodejs-buildpack/releases/tag/v1.2.1> to
69MB <https://github.com/cloudfoundry/nodejs-buildpack/releases/tag/v1.3.0>.

Supports (full manifest here
<https://github.com/cloudfoundry/nodejs-buildpack/blob/v1.3.0/manifest.yml>
):

- 0.8.{27,28}
- 0.9.{11,12}
- 0.10.{37,38}
- 0.11.{15,16}
- 0.12.{1,2}

php-buildpack v3.2.0

Full release notes are here
<https://github.com/cloudfoundry/php-buildpack/releases/tag/v3.2.0>.

Size reduced 27% from 1.1GB
<https://github.com/cloudfoundry/php-buildpack/releases/tag/v3.1.1> to 803MB
<https://github.com/cloudfoundry/php-buildpack/releases/tag/v3.2.0>.

Supports: (full manifest here
<https://github.com/cloudfoundry/php-buildpack/blob/v3.2.0/manifest.yml>)

*PHP*:

- 5.6.{6,7}
- 5.5.{22,23}
- 5.4.{38,39}

*HHVM* (lucid64 stack):

- 3.2.0

*HHVM* (cflinuxfs2 stack):

- 3.5.{0,1}
- 3.6.{0,1}

*Apache HTTPD*:

- 2.4.12

*nginx*:

- 1.7.10
- 1.6.2
- 1.5.13

python-buildpack v1.3.0

Full release notes are here
<https://github.com/cloudfoundry/python-buildpack/releases/tag/v1.3.0>.

Size reduced 30% from 654MB
<https://github.com/cloudfoundry/python-buildpack/releases/tag/v1.2.0> to
454MB <https://github.com/cloudfoundry/python-buildpack/releases/tag/v1.3.0>
.

Supports: (full manifest here
<https://github.com/cloudfoundry/python-buildpack/blob/v1.3.0/manifest.yml>)

- 2.7.{8,9}
- 3.2.{4,5}
- 3.3.{5,6}
- 3.4.{2,3}

ruby-buildpack v1.4.0

Release notes are here
<https://github.com/cloudfoundry/ruby-buildpack/releases/tag/v1.4.0>.

Size reduced 71% from 1.3GB
<https://github.com/cloudfoundry/ruby-buildpack/releases/tag/v1.3.1> to
365MB <https://github.com/cloudfoundry/ruby-buildpack/releases/tag/v1.4.0>.

Supports: (full manifest here
<https://github.com/cloudfoundry/ruby-buildpack/blob/v1.4.0/manifest.yml>)

*MRI*:

- 2.2.{1,2}
- 2.1.{5,6}
- 2.0.0p645

*JRuby*:

- ruby-1.9.3-jruby-1.7.19
- ruby-2.0.0-jruby-1.7.19
- ruby-2.2.0-jruby-9.0.0.0.pre1


---------- Forwarded message ----------
From: Mike Dalessio <mdalessio(a)pivotal.io>
Date: Wed, Apr 8, 2015 at 11:10 AM
Subject: Addressing buildpack size
To: vcap-dev(a)cloudfoundry.org


Hello vcap-dev!

This email details a proposed change to how Cloud Foundry buildpacks are
packaged, with respect to the ever-increasing number of binary dependencies
being cached within them.

This proposal's permanent residence is here:

https://github.com/cloudfoundry-incubator/buildpack-packager/issues/4

Feel free to comment there or reply to this email.
------------------------------
Buildpack SizesWhere we are today

Many of you have seen, and possibly been challenged by, the enormous sizes
of some of the buildpacks that are currently shipping with cf-release.

Here's the state of the world right now, as of v205:

php-buildpack: 1.1G
ruby-buildpack: 922M
go-buildpack: 675M
python-buildpack: 654M
nodejs-buildpack: 403M
----------------------
total: 3.7G

These enormous sizes are the result of the current policy of packaging
every-version-of-everything-ever-supported ("EVOEES") within the buildpack.

Most recently, this problem was exacerbated by the fact that buildpacks now
contain binaries for two rootfses.
Why this is a problem

If continued, buildpacks will only continue to increase in size, leading to
longer and longer build and deploy times, longer test times, slacker
feedback loops, and therefore less frequent buildpack releases.

Additionally, this also means that we're shipping versions of interpreters,
web servers, and libraries that are deprecated, insecure, or both. Feedback
from CF users has made it clear that many companies view this as an
unnecessary security risk.

This policy is clearly unsustainable.
What we can do about it

There are many things being discussed to ameliorate the impact that
buildpack size is having on the operations of CF.

Notably, Onsi has proposed a change to buildpack caching, to improve Diego
staging times (link to proposal
<https://github.com/pivotal-cf-experimental/diego-dev-notes/blob/master/proposals/better-buildpack-caching.md>
).

However, there is an immediate solution available, which addresses both the
size concerns as well as the security concern: packaging fewer binary
dependencies within the buildpack.
The proposal

I'm proposing that we reduce the binary dependencies in each buildpack in a
very specific way.

Aside on terms I'll use below:

- Versions of the form "1.2.3" are broken down as: MAJOR.MINOR.TEENY.
Many language ecosystems refer to the "TEENY" as "PATCH" interchangeably,
but we're going to use "TEENY" in this proposal.
- We'll assume that TEENY gets bumped for API/ABI compatible changes.
- We'll assume that MINOR and MAJOR get bumped when there are API/ABI
*incompatible* changes.

I'd like to move forward soon with the following changes:

1. For language interpreters/compilers, we'll package the two
most-recent TEENY versions on each MAJOR.MINOR release.
2. For all other dependencies, we'll package only the single most-recent
TEENY version on each MAJOR.MINOR release.
3. We will discontinue packaging versions of dependencies that have been
deprecated.
4. We will no longer provide "EVOEES" buildpack releases.
5. We will no longer provide "online" buildpack releases, which download
dependencies from the public internet.
6. We will document the process, and provide tooling, for CF operators
to build their own buildpacks, choosing the dependencies that their
organization wants to support or creating "online" buildpacks at operators'
discretion.

An example for #1 is that we'll go from packaging 34 versions of node v0.10.x
to only packaging two: 0.10.37 and 0.10.38.

An example for #2 is that we'll go from packaging 3 versions of nginx 1.5
in the PHP buildpack to only packaging one: 1.5.12.

An example for #3 is that we'll discontinue packaging ruby 1.9.3 in the
ruby-buildpack, which reached end-of-life in February 2015.
Outcomes

With these changes, the total buildpack size will be reduced greatly. As an
example, we expect the ruby-buildpack size to go from 922M to 338M.

We also want to set the expectation that, as new interpreter versions are
released, either for new features or (more urgently) for security fixes,
we'll release new buildpacks much more quickly than we do today. My hope is
that we'll be able to do it within 24 hours of a new release.
Planning

These changes will be relatively easy to make, since all the buildpacks are
now using a manifest.yml file to declare what's being packaged. We expect
to be able to complete this work within the next two weeks.

Stories are in the Tracker backlog under the Epic named
"skinny-buildpacks", which you can see here:

https://www.pivotaltracker.com/epic/show/1747328

------------------------------

Please let me know how these changes will impact you and your
organizations, and let me know of any counter-proposals or variations you'd
like to consider.

Thanks,

-mike

Join cf-dev@lists.cloudfoundry.org to automatically receive all group messages.