Date
1 - 13 of 13
Addressing buildpack size
Daniel Mikusa
On Fri, May 8, 2015 at 3:09 PM, Mike Dalessio <mdalessio(a)pivotal.io> wrote:
been thinking about the subject a bit, but haven't come up with any great
ideas.
The first thought that came to mind was a transparent network proxy, like
Squid, which would just automatically cache the files as they're accessed.
It's nice and simple, nothing with the build pack would need to change or
be altered to take advantage of it, but I'm not sure how that would work in
a completely offline environments as I'm not sure how you'd seed the cache.
Another thought was for the DEA to provided some additional hints to the
build packs about how they could locate binaries. Perhaps a special
environment variable like CF_BP_REPO=http://repo.system.domain/. The build
pack could then take that and use it to generate URLs to it's binary
resources. A variation on that would be to check this repo first, and then
fall back to some global external repo if available (i.e. most recent stuff
is on CF_BP_REPO, older stuff needs Internet access to download). Yet
another variation would be for the CF_BP_REPO to start small and grow as
things are requested. For example, if you request a file that doesn't
exist CF_BP_REPO would try to download it from the Internet, cache it and
stream it back to the app.
Anyway, I'm just thinking out loud now. Thanks for the update!
Dan
Hey Dan,This sounds cool. Can't wait to see what you guys come up with here. I've
On Tue, May 5, 2015 at 1:33 PM, Daniel Mikusa <dmikusa(a)pivotal.io> wrote:I'm happy to see the size of the build packs dropping, but I have to askYou've nailed it. Yes, it makes a ton of sense to handle binaries as
why do we bundle the build packs with a fixed set of binaries?
The build packs themselves are very small, it's the binaries that are
huge. It seems like it would make sense to handle them as separate
concerns.
separate concerns, and we're heading in that direction.
At one point very recently, we started doing some planning around how we
might cache buildpack assets in a structured way (like a blob store) and
seamlessly have everything Just Work™.
The first step towards separating these concerns was to extract the use of
dependencies out of the (generally upstream) buildpack code and into a
buildpack manifest file. Having done that, the dependencies are now
first-class artifacts that can be managed by operators.
We stopped there, at least for the time being, as it's not terribly clear
how to jam buildpack asset caching into the current API, CC buildpack
model, and staging process (though, again, the manifest is the best first
step, as it enables us to trap network calls and thus redirect them to a
cache either on disk or over the network).
It's also quite possible that the remaining pain will be further
ameliorated by the proposed Diego feature to attach persistent disk (on
which, presumably, the buildpacks and their assets are cached), which means
we're deferring further work until we've got more user feedback and data.
been thinking about the subject a bit, but haven't come up with any great
ideas.
The first thought that came to mind was a transparent network proxy, like
Squid, which would just automatically cache the files as they're accessed.
It's nice and simple, nothing with the build pack would need to change or
be altered to take advantage of it, but I'm not sure how that would work in
a completely offline environments as I'm not sure how you'd seed the cache.
Another thought was for the DEA to provided some additional hints to the
build packs about how they could locate binaries. Perhaps a special
environment variable like CF_BP_REPO=http://repo.system.domain/. The build
pack could then take that and use it to generate URLs to it's binary
resources. A variation on that would be to check this repo first, and then
fall back to some global external repo if available (i.e. most recent stuff
is on CF_BP_REPO, older stuff needs Internet access to download). Yet
another variation would be for the CF_BP_REPO to start small and grow as
things are requested. For example, if you request a file that doesn't
exist CF_BP_REPO would try to download it from the Internet, cache it and
stream it back to the app.
Anyway, I'm just thinking out loud now. Thanks for the update!
Dan
I don't want to come off too harsh, but in addition to the size of the
build packs when bundled with binaries, there are some other disadvantages
to doing things this way.
- Binaries and build packs are updated at different rates. Binaries
are usually updated often, to pick up new runtimes versions & security
fixes; build packs are generally changed at a slower pace, as features or
bug fixes for them are needed. Bundling the two together, requires an
operator to update the build packs more often, just to get updated
binaries. It's been my experience that users don't (or forget) to update
build packs which means they're likely running with older, possibly
insecure runtimes.
- It's difficult to bundle a set of runtime binaries that suite
everyone's needs, different users will update at different rates and will
want different sets of binaries. If build packs and binaries are packaged
together, users will end up needing to find a specific build pack bundle
that contains the runtime they want or users will need to build their own
custom bundles. If build packs and binaries are handled separately, there
will be more flexibility in what binaries a build pack has available as an
operator can manage binaries independently. Wayne's post seems to hit on
this point.
- At some point, I think this has already happened (jruby & java),
build packs are going to start having overlapping sets of binaries. If the
binaries are bundled with the build pack, there's no way that build packs
could ever share binaries.
My personal preference would be to see build packs bundled without
binaries and some other solution, which probably merits a separate thread,
for managing the binaries.
I'm curious to hear what others think or if I've missed something and
bundling build packs and binaries is clearly the way to go.
Dan
PS. If this is something that came up in the PMC, I apologize. I
skimmed the notes, but may have missed it.
On Mon, May 4, 2015 at 2:10 PM, Wayne E. Seguin <
wayneeseguin(a)starkandwayne.com> wrote:Because of very good compatibility between versions (post 1.X) I would
like to make a motion to do the following:
Split the buildpack:
have the default golang buildpack track the latest golang version
Then handle older versions in one of two ways, either:
a) have a large secondary for older versions
or
b) have multiple, one for each version of golang, users can specify a
specific URL if they care about specific versions.
This would improve space/time considerations for operations. Personally
I would prefer b) because it allows you to enable supporting older go
versions out of the box by design but still keeping each golang buildpack
small.
~Wayne
Wayne E. Seguin <wayneeseguin(a)starkandwayne.com>
CTO ; Stark & Wayne, LLC
On May 4, 2015, at 12:40 , Mike Dalessio <mdalessio(a)pivotal.io> wrote:
Hi Wayne,
On Fri, May 1, 2015 at 1:29 PM, Wayne E. Seguin <
wayneeseguin(a)starkandwayne.com> wrote:What an incredible step in the right direction, Awesome!!!Thanks for asking this question.
Out of curiosity, why is the go buildpack still quite so large?
Currently we're including the following binary dependencies in
`go-buildpack`:
```
cache $ ls -lSh *_go*
-rw-r--r-- 1 flavorjones flavorjones 60M 2015-05-04 12:36
https___storage.googleapis.com_golang_go1.4.2.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 60M 2015-05-04 12:36
https___storage.googleapis.com_golang_go1.4.1.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 54M 2015-05-04 12:36
https___storage.googleapis.com_golang_go1.2.2.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 54M 2015-05-04 12:36
http___go.googlecode.com_files_go1.2.1.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 51M 2015-05-04 12:36
https___storage.googleapis.com_golang_go1.3.3.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 51M 2015-05-04 12:36
https___storage.googleapis.com_golang_go1.3.2.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 40M 2015-05-04 12:36
http___go.googlecode.com_files_go1.1.2.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 40M 2015-05-04 12:36
http___go.googlecode.com_files_go1.1.1.linux-amd64.tar.gz
```
One question we should ask, I think, is: should we still be supporting
golang 1.1 and 1.2? Dropping those versions would cut the size of the
buildpack in (approximately) half.
On May 1, 2015, at 11:54 , Mike Dalessio <mdalessio(a)pivotal.io> wrote:
Skinny buildpacks have been cut for go, nodejs, php, python and ruby
buildpacks.
| | current | previous |
|--------+---------+----------|
| go | 442MB | 633MB |
| nodejs | 69MB | 417MB |
| php | 804MB | 1.1GB |
| python | 454MB | 654MB |
| ruby | 365MB | 1.3GB |
|--------+---------+----------|
| total | 2.1GB | 4.1GB |
for an aggregate 51% reduction in size. Details follow.
Next Steps
I recognize that every cloud operator may have a different policy on
what versions of interpreters and libraries they want to support, based on
the specific requirements of their users.
These buildpacks reflect a "bare mininum" policy for a cloud to be
operable, and I do not expect these buildpacks to be adopted as-is by many
operators.
These buildpacks have not yet been added to cf-release, specifically
so that the community can prepare their own buildpacks if necessary.
Over the next few days, the buildpacks core team will ship
documentation and tooling to assist you in packaging specific dependencies
for your instance of CF. I'll start a new thread on this list early next
week to communicate this information.
Call to Action
In the meantime, please think about whether the policy implemented in
these buildpacks ("last two patches (or teenies) on all supported
major.minor releases") is suitable for your users; and if not, think about
what dependencies you'll ideally be supporting.
go-buildpack v1.3.0
Release notes are here
<https://github.com/cloudfoundry/go-buildpack/releases/tag/v1.3.0>.
Size reduced 30% from 633MB
<https://github.com/cloudfoundry/go-buildpack/releases/tag/v1.2.0> to
442MB
<https://github.com/cloudfoundry/go-buildpack/releases/tag/v1.3.0>.
Supports (full manifest here
<https://github.com/cloudfoundry/go-buildpack/blob/v1.3.0/manifest.yml>
):
- golang 1.4.{1,2}
- golang 1.3.{2,3}
- golang 1.2.{1,2}
- golang 1.1.{1,2}
nodejs-buildpack v1.3.0
Full release notes are here
<https://github.com/cloudfoundry/nodejs-buildpack/releases/tag/v1.3.0>.
Size reduced 83% from 417MB
<https://github.com/cloudfoundry/nodejs-buildpack/releases/tag/v1.2.1>
to 69MB
<https://github.com/cloudfoundry/nodejs-buildpack/releases/tag/v1.3.0>.
Supports (full manifest here
<https://github.com/cloudfoundry/nodejs-buildpack/blob/v1.3.0/manifest.yml>
):
- 0.8.{27,28}
- 0.9.{11,12}
- 0.10.{37,38}
- 0.11.{15,16}
- 0.12.{1,2}
php-buildpack v3.2.0
Full release notes are here
<https://github.com/cloudfoundry/php-buildpack/releases/tag/v3.2.0>.
Size reduced 27% from 1.1GB
<https://github.com/cloudfoundry/php-buildpack/releases/tag/v3.1.1> to
803MB
<https://github.com/cloudfoundry/php-buildpack/releases/tag/v3.2.0>.
Supports: (full manifest here
<https://github.com/cloudfoundry/php-buildpack/blob/v3.2.0/manifest.yml>
)
*PHP*:
- 5.6.{6,7}
- 5.5.{22,23}
- 5.4.{38,39}
*HHVM* (lucid64 stack):
- 3.2.0
*HHVM* (cflinuxfs2 stack):
- 3.5.{0,1}
- 3.6.{0,1}
*Apache HTTPD*:
- 2.4.12
*nginx*:
- 1.7.10
- 1.6.2
- 1.5.13
python-buildpack v1.3.0
Full release notes are here
<https://github.com/cloudfoundry/python-buildpack/releases/tag/v1.3.0>.
Size reduced 30% from 654MB
<https://github.com/cloudfoundry/python-buildpack/releases/tag/v1.2.0>
to 454MB
<https://github.com/cloudfoundry/python-buildpack/releases/tag/v1.3.0>.
Supports: (full manifest here
<https://github.com/cloudfoundry/python-buildpack/blob/v1.3.0/manifest.yml>
)
- 2.7.{8,9}
- 3.2.{4,5}
- 3.3.{5,6}
- 3.4.{2,3}
ruby-buildpack v1.4.0
Release notes are here
<https://github.com/cloudfoundry/ruby-buildpack/releases/tag/v1.4.0>.
Size reduced 71% from 1.3GB
<https://github.com/cloudfoundry/ruby-buildpack/releases/tag/v1.3.1>
to 365MB
<https://github.com/cloudfoundry/ruby-buildpack/releases/tag/v1.4.0>.
Supports: (full manifest here
<https://github.com/cloudfoundry/ruby-buildpack/blob/v1.4.0/manifest.yml>
)
*MRI*:
- 2.2.{1,2}
- 2.1.{5,6}
- 2.0.0p645
*JRuby*:
- ruby-1.9.3-jruby-1.7.19
- ruby-2.0.0-jruby-1.7.19
- ruby-2.2.0-jruby-9.0.0.0.pre1
---------- Forwarded message ----------
From: Mike Dalessio <mdalessio(a)pivotal.io>
Date: Wed, Apr 8, 2015 at 11:10 AM
Subject: Addressing buildpack size
To: vcap-dev(a)cloudfoundry.org
Hello vcap-dev!
This email details a proposed change to how Cloud Foundry buildpacks
are packaged, with respect to the ever-increasing number of binary
dependencies being cached within them.
This proposal's permanent residence is here:
https://github.com/cloudfoundry-incubator/buildpack-packager/issues/4
Feel free to comment there or reply to this email.
------------------------------
Buildpack SizesWhere we are today
Many of you have seen, and possibly been challenged by, the enormous
sizes of some of the buildpacks that are currently shipping with cf-release.
Here's the state of the world right now, as of v205:
php-buildpack: 1.1G
ruby-buildpack: 922M
go-buildpack: 675M
python-buildpack: 654M
nodejs-buildpack: 403M
----------------------
total: 3.7G
These enormous sizes are the result of the current policy of packaging
every-version-of-everything-ever-supported ("EVOEES") within the buildpack.
Most recently, this problem was exacerbated by the fact that buildpacks
now contain binaries for two rootfses.
Why this is a problem
If continued, buildpacks will only continue to increase in size,
leading to longer and longer build and deploy times, longer test times,
slacker feedback loops, and therefore less frequent buildpack releases.
Additionally, this also means that we're shipping versions of
interpreters, web servers, and libraries that are deprecated, insecure, or
both. Feedback from CF users has made it clear that many companies view
this as an unnecessary security risk.
This policy is clearly unsustainable.
What we can do about it
There are many things being discussed to ameliorate the impact that
buildpack size is having on the operations of CF.
Notably, Onsi has proposed a change to buildpack caching, to improve
Diego staging times (link to proposal
<https://github.com/pivotal-cf-experimental/diego-dev-notes/blob/master/proposals/better-buildpack-caching.md>
).
However, there is an immediate solution available, which addresses both
the size concerns as well as the security concern: packaging fewer binary
dependencies within the buildpack.
The proposal
I'm proposing that we reduce the binary dependencies in each buildpack
in a very specific way.
Aside on terms I'll use below:
- Versions of the form "1.2.3" are broken down as:
MAJOR.MINOR.TEENY. Many language ecosystems refer to the "TEENY" as "PATCH"
interchangeably, but we're going to use "TEENY" in this proposal.
- We'll assume that TEENY gets bumped for API/ABI compatible
changes.
- We'll assume that MINOR and MAJOR get bumped when there are
API/ABI *incompatible* changes.
I'd like to move forward soon with the following changes:
1. For language interpreters/compilers, we'll package the two
most-recent TEENY versions on each MAJOR.MINOR release.
2. For all other dependencies, we'll package only the single
most-recent TEENY version on each MAJOR.MINOR release.
3. We will discontinue packaging versions of dependencies that have
been deprecated.
4. We will no longer provide "EVOEES" buildpack releases.
5. We will no longer provide "online" buildpack releases, which
download dependencies from the public internet.
6. We will document the process, and provide tooling, for CF
operators to build their own buildpacks, choosing the dependencies that
their organization wants to support or creating "online" buildpacks at
operators' discretion.
An example for #1 is that we'll go from packaging 34 versions of node v0.10.x
to only packaging two: 0.10.37 and 0.10.38.
An example for #2 is that we'll go from packaging 3 versions of nginx 1.5
in the PHP buildpack to only packaging one: 1.5.12.
An example for #3 is that we'll discontinue packaging ruby 1.9.3 in the
ruby-buildpack, which reached end-of-life in February 2015.
Outcomes
With these changes, the total buildpack size will be reduced greatly.
As an example, we expect the ruby-buildpack size to go from 922M to 338M.
We also want to set the expectation that, as new interpreter versions
are released, either for new features or (more urgently) for security
fixes, we'll release new buildpacks much more quickly than we do today. My
hope is that we'll be able to do it within 24 hours of a new release.
Planning
These changes will be relatively easy to make, since all the buildpacks
are now using a manifest.yml file to declare what's being packaged. We
expect to be able to complete this work within the next two weeks.
Stories are in the Tracker backlog under the Epic named
"skinny-buildpacks", which you can see here:
https://www.pivotaltracker.com/epic/show/1747328
------------------------------
Please let me know how these changes will impact you and your
organizations, and let me know of any counter-proposals or variations you'd
like to consider.
Thanks,
-mike
_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
Josh Ghiloni <ghiloni@...>
How does that jive with offline buildpacks? Would it be a matter of the
operator building it with a certain version of the binaries and then
uploading them combined?
toggle quoted message
Show quoted text
operator building it with a certain version of the binaries and then
uploading them combined?
On Fri, May 8, 2015 at 7:01 PM, Patrick Mueller <pmuellr(a)gmail.com> wrote:
Ya, it doesn't seem to make a lot of sense to me to bundle the buildpacks
with their typical binaries. Take io.js for instance [1]; prolly not
required to change the buildpack as often as new releases of the io.js
itself.
[1] https://github.com/iojs/io.js/blob/master/CHANGELOG.md
On Tue, May 5, 2015 at 1:33 PM, Daniel Mikusa <dmikusa(a)pivotal.io> wrote:I'm happy to see the size of the build packs dropping, but I have to ask
why do we bundle the build packs with a fixed set of binaries?
The build packs themselves are very small, it's the binaries that are
huge. It seems like it would make sense to handle them as separate
concerns.
I don't want to come off too harsh, but in addition to the size of the
build packs when bundled with binaries, there are some other disadvantages
to doing things this way.
- Binaries and build packs are updated at different rates. Binaries
are usually updated often, to pick up new runtimes versions & security
fixes; build packs are generally changed at a slower pace, as features or
bug fixes for them are needed. Bundling the two together, requires an
operator to update the build packs more often, just to get updated
binaries. It's been my experience that users don't (or forget) to update
build packs which means they're likely running with older, possibly
insecure runtimes.
- It's difficult to bundle a set of runtime binaries that suite
everyone's needs, different users will update at different rates and will
want different sets of binaries. If build packs and binaries are packaged
together, users will end up needing to find a specific build pack bundle
that contains the runtime they want or users will need to build their own
custom bundles. If build packs and binaries are handled separately, there
will be more flexibility in what binaries a build pack has available as an
operator can manage binaries independently. Wayne's post seems to hit on
this point.
- At some point, I think this has already happened (jruby & java),
build packs are going to start having overlapping sets of binaries. If the
binaries are bundled with the build pack, there's no way that build packs
could ever share binaries.
My personal preference would be to see build packs bundled without
binaries and some other solution, which probably merits a separate thread,
for managing the binaries.
I'm curious to hear what others think or if I've missed something and
bundling build packs and binaries is clearly the way to go.
Dan
PS. If this is something that came up in the PMC, I apologize. I
skimmed the notes, but may have missed it.
On Mon, May 4, 2015 at 2:10 PM, Wayne E. Seguin <
wayneeseguin(a)starkandwayne.com> wrote:Because of very good compatibility between versions (post 1.X) I would_______________________________________________
like to make a motion to do the following:
Split the buildpack:
have the default golang buildpack track the latest golang version
Then handle older versions in one of two ways, either:
a) have a large secondary for older versions
or
b) have multiple, one for each version of golang, users can specify a
specific URL if they care about specific versions.
This would improve space/time considerations for operations. Personally
I would prefer b) because it allows you to enable supporting older go
versions out of the box by design but still keeping each golang buildpack
small.
~Wayne
Wayne E. Seguin <wayneeseguin(a)starkandwayne.com>
CTO ; Stark & Wayne, LLC
On May 4, 2015, at 12:40 , Mike Dalessio <mdalessio(a)pivotal.io> wrote:
Hi Wayne,
On Fri, May 1, 2015 at 1:29 PM, Wayne E. Seguin <
wayneeseguin(a)starkandwayne.com> wrote:What an incredible step in the right direction, Awesome!!!Thanks for asking this question.
Out of curiosity, why is the go buildpack still quite so large?
Currently we're including the following binary dependencies in
`go-buildpack`:
```
cache $ ls -lSh *_go*
-rw-r--r-- 1 flavorjones flavorjones 60M 2015-05-04 12:36
https___storage.googleapis.com_golang_go1.4.2.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 60M 2015-05-04 12:36
https___storage.googleapis.com_golang_go1.4.1.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 54M 2015-05-04 12:36
https___storage.googleapis.com_golang_go1.2.2.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 54M 2015-05-04 12:36
http___go.googlecode.com_files_go1.2.1.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 51M 2015-05-04 12:36
https___storage.googleapis.com_golang_go1.3.3.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 51M 2015-05-04 12:36
https___storage.googleapis.com_golang_go1.3.2.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 40M 2015-05-04 12:36
http___go.googlecode.com_files_go1.1.2.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 40M 2015-05-04 12:36
http___go.googlecode.com_files_go1.1.1.linux-amd64.tar.gz
```
One question we should ask, I think, is: should we still be supporting
golang 1.1 and 1.2? Dropping those versions would cut the size of the
buildpack in (approximately) half.
On May 1, 2015, at 11:54 , Mike Dalessio <mdalessio(a)pivotal.io> wrote:
Skinny buildpacks have been cut for go, nodejs, php, python and ruby
buildpacks.
| | current | previous |
|--------+---------+----------|
| go | 442MB | 633MB |
| nodejs | 69MB | 417MB |
| php | 804MB | 1.1GB |
| python | 454MB | 654MB |
| ruby | 365MB | 1.3GB |
|--------+---------+----------|
| total | 2.1GB | 4.1GB |
for an aggregate 51% reduction in size. Details follow.
Next Steps
I recognize that every cloud operator may have a different policy on
what versions of interpreters and libraries they want to support, based on
the specific requirements of their users.
These buildpacks reflect a "bare mininum" policy for a cloud to be
operable, and I do not expect these buildpacks to be adopted as-is by many
operators.
These buildpacks have not yet been added to cf-release, specifically
so that the community can prepare their own buildpacks if necessary.
Over the next few days, the buildpacks core team will ship
documentation and tooling to assist you in packaging specific dependencies
for your instance of CF. I'll start a new thread on this list early next
week to communicate this information.
Call to Action
In the meantime, please think about whether the policy implemented in
these buildpacks ("last two patches (or teenies) on all supported
major.minor releases") is suitable for your users; and if not, think about
what dependencies you'll ideally be supporting.
go-buildpack v1.3.0
Release notes are here
<https://github.com/cloudfoundry/go-buildpack/releases/tag/v1.3.0>.
Size reduced 30% from 633MB
<https://github.com/cloudfoundry/go-buildpack/releases/tag/v1.2.0> to
442MB
<https://github.com/cloudfoundry/go-buildpack/releases/tag/v1.3.0>.
Supports (full manifest here
<https://github.com/cloudfoundry/go-buildpack/blob/v1.3.0/manifest.yml>
):
- golang 1.4.{1,2}
- golang 1.3.{2,3}
- golang 1.2.{1,2}
- golang 1.1.{1,2}
nodejs-buildpack v1.3.0
Full release notes are here
<https://github.com/cloudfoundry/nodejs-buildpack/releases/tag/v1.3.0>.
Size reduced 83% from 417MB
<https://github.com/cloudfoundry/nodejs-buildpack/releases/tag/v1.2.1>
to 69MB
<https://github.com/cloudfoundry/nodejs-buildpack/releases/tag/v1.3.0>.
Supports (full manifest here
<https://github.com/cloudfoundry/nodejs-buildpack/blob/v1.3.0/manifest.yml>
):
- 0.8.{27,28}
- 0.9.{11,12}
- 0.10.{37,38}
- 0.11.{15,16}
- 0.12.{1,2}
php-buildpack v3.2.0
Full release notes are here
<https://github.com/cloudfoundry/php-buildpack/releases/tag/v3.2.0>.
Size reduced 27% from 1.1GB
<https://github.com/cloudfoundry/php-buildpack/releases/tag/v3.1.1> to
803MB
<https://github.com/cloudfoundry/php-buildpack/releases/tag/v3.2.0>.
Supports: (full manifest here
<https://github.com/cloudfoundry/php-buildpack/blob/v3.2.0/manifest.yml>
)
*PHP*:
- 5.6.{6,7}
- 5.5.{22,23}
- 5.4.{38,39}
*HHVM* (lucid64 stack):
- 3.2.0
*HHVM* (cflinuxfs2 stack):
- 3.5.{0,1}
- 3.6.{0,1}
*Apache HTTPD*:
- 2.4.12
*nginx*:
- 1.7.10
- 1.6.2
- 1.5.13
python-buildpack v1.3.0
Full release notes are here
<https://github.com/cloudfoundry/python-buildpack/releases/tag/v1.3.0>.
Size reduced 30% from 654MB
<https://github.com/cloudfoundry/python-buildpack/releases/tag/v1.2.0>
to 454MB
<https://github.com/cloudfoundry/python-buildpack/releases/tag/v1.3.0>.
Supports: (full manifest here
<https://github.com/cloudfoundry/python-buildpack/blob/v1.3.0/manifest.yml>
)
- 2.7.{8,9}
- 3.2.{4,5}
- 3.3.{5,6}
- 3.4.{2,3}
ruby-buildpack v1.4.0
Release notes are here
<https://github.com/cloudfoundry/ruby-buildpack/releases/tag/v1.4.0>.
Size reduced 71% from 1.3GB
<https://github.com/cloudfoundry/ruby-buildpack/releases/tag/v1.3.1>
to 365MB
<https://github.com/cloudfoundry/ruby-buildpack/releases/tag/v1.4.0>.
Supports: (full manifest here
<https://github.com/cloudfoundry/ruby-buildpack/blob/v1.4.0/manifest.yml>
)
*MRI*:
- 2.2.{1,2}
- 2.1.{5,6}
- 2.0.0p645
*JRuby*:
- ruby-1.9.3-jruby-1.7.19
- ruby-2.0.0-jruby-1.7.19
- ruby-2.2.0-jruby-9.0.0.0.pre1
---------- Forwarded message ----------
From: Mike Dalessio <mdalessio(a)pivotal.io>
Date: Wed, Apr 8, 2015 at 11:10 AM
Subject: Addressing buildpack size
To: vcap-dev(a)cloudfoundry.org
Hello vcap-dev!
This email details a proposed change to how Cloud Foundry buildpacks
are packaged, with respect to the ever-increasing number of binary
dependencies being cached within them.
This proposal's permanent residence is here:
https://github.com/cloudfoundry-incubator/buildpack-packager/issues/4
Feel free to comment there or reply to this email.
------------------------------
Buildpack SizesWhere we are today
Many of you have seen, and possibly been challenged by, the enormous
sizes of some of the buildpacks that are currently shipping with cf-release.
Here's the state of the world right now, as of v205:
php-buildpack: 1.1G
ruby-buildpack: 922M
go-buildpack: 675M
python-buildpack: 654M
nodejs-buildpack: 403M
----------------------
total: 3.7G
These enormous sizes are the result of the current policy of packaging
every-version-of-everything-ever-supported ("EVOEES") within the buildpack.
Most recently, this problem was exacerbated by the fact that buildpacks
now contain binaries for two rootfses.
Why this is a problem
If continued, buildpacks will only continue to increase in size,
leading to longer and longer build and deploy times, longer test times,
slacker feedback loops, and therefore less frequent buildpack releases.
Additionally, this also means that we're shipping versions of
interpreters, web servers, and libraries that are deprecated, insecure, or
both. Feedback from CF users has made it clear that many companies view
this as an unnecessary security risk.
This policy is clearly unsustainable.
What we can do about it
There are many things being discussed to ameliorate the impact that
buildpack size is having on the operations of CF.
Notably, Onsi has proposed a change to buildpack caching, to improve
Diego staging times (link to proposal
<https://github.com/pivotal-cf-experimental/diego-dev-notes/blob/master/proposals/better-buildpack-caching.md>
).
However, there is an immediate solution available, which addresses both
the size concerns as well as the security concern: packaging fewer binary
dependencies within the buildpack.
The proposal
I'm proposing that we reduce the binary dependencies in each buildpack
in a very specific way.
Aside on terms I'll use below:
- Versions of the form "1.2.3" are broken down as:
MAJOR.MINOR.TEENY. Many language ecosystems refer to the "TEENY" as "PATCH"
interchangeably, but we're going to use "TEENY" in this proposal.
- We'll assume that TEENY gets bumped for API/ABI compatible
changes.
- We'll assume that MINOR and MAJOR get bumped when there are
API/ABI *incompatible* changes.
I'd like to move forward soon with the following changes:
1. For language interpreters/compilers, we'll package the two
most-recent TEENY versions on each MAJOR.MINOR release.
2. For all other dependencies, we'll package only the single
most-recent TEENY version on each MAJOR.MINOR release.
3. We will discontinue packaging versions of dependencies that have
been deprecated.
4. We will no longer provide "EVOEES" buildpack releases.
5. We will no longer provide "online" buildpack releases, which
download dependencies from the public internet.
6. We will document the process, and provide tooling, for CF
operators to build their own buildpacks, choosing the dependencies that
their organization wants to support or creating "online" buildpacks at
operators' discretion.
An example for #1 is that we'll go from packaging 34 versions of node v0.10.x
to only packaging two: 0.10.37 and 0.10.38.
An example for #2 is that we'll go from packaging 3 versions of nginx 1.5
in the PHP buildpack to only packaging one: 1.5.12.
An example for #3 is that we'll discontinue packaging ruby 1.9.3 in the
ruby-buildpack, which reached end-of-life in February 2015.
Outcomes
With these changes, the total buildpack size will be reduced greatly.
As an example, we expect the ruby-buildpack size to go from 922M to 338M.
We also want to set the expectation that, as new interpreter versions
are released, either for new features or (more urgently) for security
fixes, we'll release new buildpacks much more quickly than we do today. My
hope is that we'll be able to do it within 24 hours of a new release.
Planning
These changes will be relatively easy to make, since all the buildpacks
are now using a manifest.yml file to declare what's being packaged. We
expect to be able to complete this work within the next two weeks.
Stories are in the Tracker backlog under the Epic named
"skinny-buildpacks", which you can see here:
https://www.pivotaltracker.com/epic/show/1747328
------------------------------
Please let me know how these changes will impact you and your
organizations, and let me know of any counter-proposals or variations you'd
like to consider.
Thanks,
-mike
_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
--
Patrick Mueller
http://muellerware.org
_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
Patrick Mueller <pmuellr@...>
Ya, it doesn't seem to make a lot of sense to me to bundle the buildpacks
with their typical binaries. Take io.js for instance [1]; prolly not
required to change the buildpack as often as new releases of the io.js
itself.
[1] https://github.com/iojs/io.js/blob/master/CHANGELOG.md
toggle quoted message
Show quoted text
with their typical binaries. Take io.js for instance [1]; prolly not
required to change the buildpack as often as new releases of the io.js
itself.
[1] https://github.com/iojs/io.js/blob/master/CHANGELOG.md
On Tue, May 5, 2015 at 1:33 PM, Daniel Mikusa <dmikusa(a)pivotal.io> wrote:
I'm happy to see the size of the build packs dropping, but I have to ask
why do we bundle the build packs with a fixed set of binaries?
The build packs themselves are very small, it's the binaries that are
huge. It seems like it would make sense to handle them as separate
concerns.
I don't want to come off too harsh, but in addition to the size of the
build packs when bundled with binaries, there are some other disadvantages
to doing things this way.
- Binaries and build packs are updated at different rates. Binaries are
usually updated often, to pick up new runtimes versions & security fixes;
build packs are generally changed at a slower pace, as features or bug
fixes for them are needed. Bundling the two together, requires an operator
to update the build packs more often, just to get updated binaries. It's
been my experience that users don't (or forget) to update build packs which
means they're likely running with older, possibly insecure runtimes.
- It's difficult to bundle a set of runtime binaries that suite
everyone's needs, different users will update at different rates and will
want different sets of binaries. If build packs and binaries are packaged
together, users will end up needing to find a specific build pack bundle
that contains the runtime they want or users will need to build their own
custom bundles. If build packs and binaries are handled separately, there
will be more flexibility in what binaries a build pack has available as an
operator can manage binaries independently. Wayne's post seems to hit on
this point.
- At some point, I think this has already happened (jruby & java), build
packs are going to start having overlapping sets of binaries. If the
binaries are bundled with the build pack, there's no way that build packs
could ever share binaries.
My personal preference would be to see build packs bundled without
binaries and some other solution, which probably merits a separate thread,
for managing the binaries.
I'm curious to hear what others think or if I've missed something and
bundling build packs and binaries is clearly the way to go.
Dan
PS. If this is something that came up in the PMC, I apologize. I skimmed
the notes, but may have missed it.
On Mon, May 4, 2015 at 2:10 PM, Wayne E. Seguin <
wayneeseguin(a)starkandwayne.com> wrote:Because of very good compatibility between versions (post 1.X) I would_______________________________________________
like to make a motion to do the following:
Split the buildpack:
have the default golang buildpack track the latest golang version
Then handle older versions in one of two ways, either:
a) have a large secondary for older versions
or
b) have multiple, one for each version of golang, users can specify a
specific URL if they care about specific versions.
This would improve space/time considerations for operations. Personally I
would prefer b) because it allows you to enable supporting older go
versions out of the box by design but still keeping each golang buildpack
small.
~Wayne
Wayne E. Seguin <wayneeseguin(a)starkandwayne.com>
CTO ; Stark & Wayne, LLC
On May 4, 2015, at 12:40 , Mike Dalessio <mdalessio(a)pivotal.io> wrote:
Hi Wayne,
On Fri, May 1, 2015 at 1:29 PM, Wayne E. Seguin <
wayneeseguin(a)starkandwayne.com> wrote:What an incredible step in the right direction, Awesome!!!Thanks for asking this question.
Out of curiosity, why is the go buildpack still quite so large?
Currently we're including the following binary dependencies in
`go-buildpack`:
```
cache $ ls -lSh *_go*
-rw-r--r-- 1 flavorjones flavorjones 60M 2015-05-04 12:36
https___storage.googleapis.com_golang_go1.4.2.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 60M 2015-05-04 12:36
https___storage.googleapis.com_golang_go1.4.1.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 54M 2015-05-04 12:36
https___storage.googleapis.com_golang_go1.2.2.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 54M 2015-05-04 12:36
http___go.googlecode.com_files_go1.2.1.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 51M 2015-05-04 12:36
https___storage.googleapis.com_golang_go1.3.3.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 51M 2015-05-04 12:36
https___storage.googleapis.com_golang_go1.3.2.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 40M 2015-05-04 12:36
http___go.googlecode.com_files_go1.1.2.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 40M 2015-05-04 12:36
http___go.googlecode.com_files_go1.1.1.linux-amd64.tar.gz
```
One question we should ask, I think, is: should we still be supporting
golang 1.1 and 1.2? Dropping those versions would cut the size of the
buildpack in (approximately) half.
On May 1, 2015, at 11:54 , Mike Dalessio <mdalessio(a)pivotal.io> wrote:
Skinny buildpacks have been cut for go, nodejs, php, python and ruby
buildpacks.
| | current | previous |
|--------+---------+----------|
| go | 442MB | 633MB |
| nodejs | 69MB | 417MB |
| php | 804MB | 1.1GB |
| python | 454MB | 654MB |
| ruby | 365MB | 1.3GB |
|--------+---------+----------|
| total | 2.1GB | 4.1GB |
for an aggregate 51% reduction in size. Details follow.
Next Steps
I recognize that every cloud operator may have a different policy on
what versions of interpreters and libraries they want to support, based on
the specific requirements of their users.
These buildpacks reflect a "bare mininum" policy for a cloud to be
operable, and I do not expect these buildpacks to be adopted as-is by many
operators.
These buildpacks have not yet been added to cf-release, specifically so
that the community can prepare their own buildpacks if necessary.
Over the next few days, the buildpacks core team will ship documentation
and tooling to assist you in packaging specific dependencies for your
instance of CF. I'll start a new thread on this list early next week to
communicate this information.
Call to Action
In the meantime, please think about whether the policy implemented in
these buildpacks ("last two patches (or teenies) on all supported
major.minor releases") is suitable for your users; and if not, think about
what dependencies you'll ideally be supporting.
go-buildpack v1.3.0
Release notes are here
<https://github.com/cloudfoundry/go-buildpack/releases/tag/v1.3.0>.
Size reduced 30% from 633MB
<https://github.com/cloudfoundry/go-buildpack/releases/tag/v1.2.0> to
442MB <https://github.com/cloudfoundry/go-buildpack/releases/tag/v1.3.0>
.
Supports (full manifest here
<https://github.com/cloudfoundry/go-buildpack/blob/v1.3.0/manifest.yml>
):
- golang 1.4.{1,2}
- golang 1.3.{2,3}
- golang 1.2.{1,2}
- golang 1.1.{1,2}
nodejs-buildpack v1.3.0
Full release notes are here
<https://github.com/cloudfoundry/nodejs-buildpack/releases/tag/v1.3.0>.
Size reduced 83% from 417MB
<https://github.com/cloudfoundry/nodejs-buildpack/releases/tag/v1.2.1>
to 69MB
<https://github.com/cloudfoundry/nodejs-buildpack/releases/tag/v1.3.0>.
Supports (full manifest here
<https://github.com/cloudfoundry/nodejs-buildpack/blob/v1.3.0/manifest.yml>
):
- 0.8.{27,28}
- 0.9.{11,12}
- 0.10.{37,38}
- 0.11.{15,16}
- 0.12.{1,2}
php-buildpack v3.2.0
Full release notes are here
<https://github.com/cloudfoundry/php-buildpack/releases/tag/v3.2.0>.
Size reduced 27% from 1.1GB
<https://github.com/cloudfoundry/php-buildpack/releases/tag/v3.1.1> to
803MB
<https://github.com/cloudfoundry/php-buildpack/releases/tag/v3.2.0>.
Supports: (full manifest here
<https://github.com/cloudfoundry/php-buildpack/blob/v3.2.0/manifest.yml>
)
*PHP*:
- 5.6.{6,7}
- 5.5.{22,23}
- 5.4.{38,39}
*HHVM* (lucid64 stack):
- 3.2.0
*HHVM* (cflinuxfs2 stack):
- 3.5.{0,1}
- 3.6.{0,1}
*Apache HTTPD*:
- 2.4.12
*nginx*:
- 1.7.10
- 1.6.2
- 1.5.13
python-buildpack v1.3.0
Full release notes are here
<https://github.com/cloudfoundry/python-buildpack/releases/tag/v1.3.0>.
Size reduced 30% from 654MB
<https://github.com/cloudfoundry/python-buildpack/releases/tag/v1.2.0>
to 454MB
<https://github.com/cloudfoundry/python-buildpack/releases/tag/v1.3.0>.
Supports: (full manifest here
<https://github.com/cloudfoundry/python-buildpack/blob/v1.3.0/manifest.yml>
)
- 2.7.{8,9}
- 3.2.{4,5}
- 3.3.{5,6}
- 3.4.{2,3}
ruby-buildpack v1.4.0
Release notes are here
<https://github.com/cloudfoundry/ruby-buildpack/releases/tag/v1.4.0>.
Size reduced 71% from 1.3GB
<https://github.com/cloudfoundry/ruby-buildpack/releases/tag/v1.3.1> to
365MB
<https://github.com/cloudfoundry/ruby-buildpack/releases/tag/v1.4.0>.
Supports: (full manifest here
<https://github.com/cloudfoundry/ruby-buildpack/blob/v1.4.0/manifest.yml>
)
*MRI*:
- 2.2.{1,2}
- 2.1.{5,6}
- 2.0.0p645
*JRuby*:
- ruby-1.9.3-jruby-1.7.19
- ruby-2.0.0-jruby-1.7.19
- ruby-2.2.0-jruby-9.0.0.0.pre1
---------- Forwarded message ----------
From: Mike Dalessio <mdalessio(a)pivotal.io>
Date: Wed, Apr 8, 2015 at 11:10 AM
Subject: Addressing buildpack size
To: vcap-dev(a)cloudfoundry.org
Hello vcap-dev!
This email details a proposed change to how Cloud Foundry buildpacks are
packaged, with respect to the ever-increasing number of binary dependencies
being cached within them.
This proposal's permanent residence is here:
https://github.com/cloudfoundry-incubator/buildpack-packager/issues/4
Feel free to comment there or reply to this email.
------------------------------
Buildpack SizesWhere we are today
Many of you have seen, and possibly been challenged by, the enormous
sizes of some of the buildpacks that are currently shipping with cf-release.
Here's the state of the world right now, as of v205:
php-buildpack: 1.1G
ruby-buildpack: 922M
go-buildpack: 675M
python-buildpack: 654M
nodejs-buildpack: 403M
----------------------
total: 3.7G
These enormous sizes are the result of the current policy of packaging
every-version-of-everything-ever-supported ("EVOEES") within the buildpack.
Most recently, this problem was exacerbated by the fact that buildpacks
now contain binaries for two rootfses.
Why this is a problem
If continued, buildpacks will only continue to increase in size, leading
to longer and longer build and deploy times, longer test times, slacker
feedback loops, and therefore less frequent buildpack releases.
Additionally, this also means that we're shipping versions of
interpreters, web servers, and libraries that are deprecated, insecure, or
both. Feedback from CF users has made it clear that many companies view
this as an unnecessary security risk.
This policy is clearly unsustainable.
What we can do about it
There are many things being discussed to ameliorate the impact that
buildpack size is having on the operations of CF.
Notably, Onsi has proposed a change to buildpack caching, to improve
Diego staging times (link to proposal
<https://github.com/pivotal-cf-experimental/diego-dev-notes/blob/master/proposals/better-buildpack-caching.md>
).
However, there is an immediate solution available, which addresses both
the size concerns as well as the security concern: packaging fewer binary
dependencies within the buildpack.
The proposal
I'm proposing that we reduce the binary dependencies in each buildpack
in a very specific way.
Aside on terms I'll use below:
- Versions of the form "1.2.3" are broken down as:
MAJOR.MINOR.TEENY. Many language ecosystems refer to the "TEENY" as "PATCH"
interchangeably, but we're going to use "TEENY" in this proposal.
- We'll assume that TEENY gets bumped for API/ABI compatible changes.
- We'll assume that MINOR and MAJOR get bumped when there are
API/ABI *incompatible* changes.
I'd like to move forward soon with the following changes:
1. For language interpreters/compilers, we'll package the two
most-recent TEENY versions on each MAJOR.MINOR release.
2. For all other dependencies, we'll package only the single
most-recent TEENY version on each MAJOR.MINOR release.
3. We will discontinue packaging versions of dependencies that have
been deprecated.
4. We will no longer provide "EVOEES" buildpack releases.
5. We will no longer provide "online" buildpack releases, which
download dependencies from the public internet.
6. We will document the process, and provide tooling, for CF
operators to build their own buildpacks, choosing the dependencies that
their organization wants to support or creating "online" buildpacks at
operators' discretion.
An example for #1 is that we'll go from packaging 34 versions of node v0.10.x
to only packaging two: 0.10.37 and 0.10.38.
An example for #2 is that we'll go from packaging 3 versions of nginx 1.5
in the PHP buildpack to only packaging one: 1.5.12.
An example for #3 is that we'll discontinue packaging ruby 1.9.3 in the
ruby-buildpack, which reached end-of-life in February 2015.
Outcomes
With these changes, the total buildpack size will be reduced greatly. As
an example, we expect the ruby-buildpack size to go from 922M to 338M.
We also want to set the expectation that, as new interpreter versions
are released, either for new features or (more urgently) for security
fixes, we'll release new buildpacks much more quickly than we do today. My
hope is that we'll be able to do it within 24 hours of a new release.
Planning
These changes will be relatively easy to make, since all the buildpacks
are now using a manifest.yml file to declare what's being packaged. We
expect to be able to complete this work within the next two weeks.
Stories are in the Tracker backlog under the Epic named
"skinny-buildpacks", which you can see here:
https://www.pivotaltracker.com/epic/show/1747328
------------------------------
Please let me know how these changes will impact you and your
organizations, and let me know of any counter-proposals or variations you'd
like to consider.
Thanks,
-mike
_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
Mike Dalessio
Hey Dan,
On Tue, May 5, 2015 at 1:33 PM, Daniel Mikusa <dmikusa(a)pivotal.io> wrote:
separate concerns, and we're heading in that direction.
At one point very recently, we started doing some planning around how we
might cache buildpack assets in a structured way (like a blob store) and
seamlessly have everything Just Work™.
The first step towards separating these concerns was to extract the use of
dependencies out of the (generally upstream) buildpack code and into a
buildpack manifest file. Having done that, the dependencies are now
first-class artifacts that can be managed by operators.
We stopped there, at least for the time being, as it's not terribly clear
how to jam buildpack asset caching into the current API, CC buildpack
model, and staging process (though, again, the manifest is the best first
step, as it enables us to trap network calls and thus redirect them to a
cache either on disk or over the network).
It's also quite possible that the remaining pain will be further
ameliorated by the proposed Diego feature to attach persistent disk (on
which, presumably, the buildpacks and their assets are cached), which means
we're deferring further work until we've got more user feedback and data.
On Tue, May 5, 2015 at 1:33 PM, Daniel Mikusa <dmikusa(a)pivotal.io> wrote:
I'm happy to see the size of the build packs dropping, but I have to askYou've nailed it. Yes, it makes a ton of sense to handle binaries as
why do we bundle the build packs with a fixed set of binaries?
The build packs themselves are very small, it's the binaries that are
huge. It seems like it would make sense to handle them as separate
concerns.
separate concerns, and we're heading in that direction.
At one point very recently, we started doing some planning around how we
might cache buildpack assets in a structured way (like a blob store) and
seamlessly have everything Just Work™.
The first step towards separating these concerns was to extract the use of
dependencies out of the (generally upstream) buildpack code and into a
buildpack manifest file. Having done that, the dependencies are now
first-class artifacts that can be managed by operators.
We stopped there, at least for the time being, as it's not terribly clear
how to jam buildpack asset caching into the current API, CC buildpack
model, and staging process (though, again, the manifest is the best first
step, as it enables us to trap network calls and thus redirect them to a
cache either on disk or over the network).
It's also quite possible that the remaining pain will be further
ameliorated by the proposed Diego feature to attach persistent disk (on
which, presumably, the buildpacks and their assets are cached), which means
we're deferring further work until we've got more user feedback and data.
I don't want to come off too harsh, but in addition to the size of the
build packs when bundled with binaries, there are some other disadvantages
to doing things this way.
- Binaries and build packs are updated at different rates. Binaries are
usually updated often, to pick up new runtimes versions & security fixes;
build packs are generally changed at a slower pace, as features or bug
fixes for them are needed. Bundling the two together, requires an operator
to update the build packs more often, just to get updated binaries. It's
been my experience that users don't (or forget) to update build packs which
means they're likely running with older, possibly insecure runtimes.
- It's difficult to bundle a set of runtime binaries that suite
everyone's needs, different users will update at different rates and will
want different sets of binaries. If build packs and binaries are packaged
together, users will end up needing to find a specific build pack bundle
that contains the runtime they want or users will need to build their own
custom bundles. If build packs and binaries are handled separately, there
will be more flexibility in what binaries a build pack has available as an
operator can manage binaries independently. Wayne's post seems to hit on
this point.
- At some point, I think this has already happened (jruby & java), build
packs are going to start having overlapping sets of binaries. If the
binaries are bundled with the build pack, there's no way that build packs
could ever share binaries.
My personal preference would be to see build packs bundled without
binaries and some other solution, which probably merits a separate thread,
for managing the binaries.
I'm curious to hear what others think or if I've missed something and
bundling build packs and binaries is clearly the way to go.
Dan
PS. If this is something that came up in the PMC, I apologize. I skimmed
the notes, but may have missed it.
On Mon, May 4, 2015 at 2:10 PM, Wayne E. Seguin <
wayneeseguin(a)starkandwayne.com> wrote:Because of very good compatibility between versions (post 1.X) I would
like to make a motion to do the following:
Split the buildpack:
have the default golang buildpack track the latest golang version
Then handle older versions in one of two ways, either:
a) have a large secondary for older versions
or
b) have multiple, one for each version of golang, users can specify a
specific URL if they care about specific versions.
This would improve space/time considerations for operations. Personally I
would prefer b) because it allows you to enable supporting older go
versions out of the box by design but still keeping each golang buildpack
small.
~Wayne
Wayne E. Seguin <wayneeseguin(a)starkandwayne.com>
CTO ; Stark & Wayne, LLC
On May 4, 2015, at 12:40 , Mike Dalessio <mdalessio(a)pivotal.io> wrote:
Hi Wayne,
On Fri, May 1, 2015 at 1:29 PM, Wayne E. Seguin <
wayneeseguin(a)starkandwayne.com> wrote:What an incredible step in the right direction, Awesome!!!Thanks for asking this question.
Out of curiosity, why is the go buildpack still quite so large?
Currently we're including the following binary dependencies in
`go-buildpack`:
```
cache $ ls -lSh *_go*
-rw-r--r-- 1 flavorjones flavorjones 60M 2015-05-04 12:36
https___storage.googleapis.com_golang_go1.4.2.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 60M 2015-05-04 12:36
https___storage.googleapis.com_golang_go1.4.1.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 54M 2015-05-04 12:36
https___storage.googleapis.com_golang_go1.2.2.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 54M 2015-05-04 12:36
http___go.googlecode.com_files_go1.2.1.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 51M 2015-05-04 12:36
https___storage.googleapis.com_golang_go1.3.3.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 51M 2015-05-04 12:36
https___storage.googleapis.com_golang_go1.3.2.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 40M 2015-05-04 12:36
http___go.googlecode.com_files_go1.1.2.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 40M 2015-05-04 12:36
http___go.googlecode.com_files_go1.1.1.linux-amd64.tar.gz
```
One question we should ask, I think, is: should we still be supporting
golang 1.1 and 1.2? Dropping those versions would cut the size of the
buildpack in (approximately) half.
On May 1, 2015, at 11:54 , Mike Dalessio <mdalessio(a)pivotal.io> wrote:
Skinny buildpacks have been cut for go, nodejs, php, python and ruby
buildpacks.
| | current | previous |
|--------+---------+----------|
| go | 442MB | 633MB |
| nodejs | 69MB | 417MB |
| php | 804MB | 1.1GB |
| python | 454MB | 654MB |
| ruby | 365MB | 1.3GB |
|--------+---------+----------|
| total | 2.1GB | 4.1GB |
for an aggregate 51% reduction in size. Details follow.
Next Steps
I recognize that every cloud operator may have a different policy on
what versions of interpreters and libraries they want to support, based on
the specific requirements of their users.
These buildpacks reflect a "bare mininum" policy for a cloud to be
operable, and I do not expect these buildpacks to be adopted as-is by many
operators.
These buildpacks have not yet been added to cf-release, specifically so
that the community can prepare their own buildpacks if necessary.
Over the next few days, the buildpacks core team will ship documentation
and tooling to assist you in packaging specific dependencies for your
instance of CF. I'll start a new thread on this list early next week to
communicate this information.
Call to Action
In the meantime, please think about whether the policy implemented in
these buildpacks ("last two patches (or teenies) on all supported
major.minor releases") is suitable for your users; and if not, think about
what dependencies you'll ideally be supporting.
go-buildpack v1.3.0
Release notes are here
<https://github.com/cloudfoundry/go-buildpack/releases/tag/v1.3.0>.
Size reduced 30% from 633MB
<https://github.com/cloudfoundry/go-buildpack/releases/tag/v1.2.0> to
442MB <https://github.com/cloudfoundry/go-buildpack/releases/tag/v1.3.0>
.
Supports (full manifest here
<https://github.com/cloudfoundry/go-buildpack/blob/v1.3.0/manifest.yml>
):
- golang 1.4.{1,2}
- golang 1.3.{2,3}
- golang 1.2.{1,2}
- golang 1.1.{1,2}
nodejs-buildpack v1.3.0
Full release notes are here
<https://github.com/cloudfoundry/nodejs-buildpack/releases/tag/v1.3.0>.
Size reduced 83% from 417MB
<https://github.com/cloudfoundry/nodejs-buildpack/releases/tag/v1.2.1>
to 69MB
<https://github.com/cloudfoundry/nodejs-buildpack/releases/tag/v1.3.0>.
Supports (full manifest here
<https://github.com/cloudfoundry/nodejs-buildpack/blob/v1.3.0/manifest.yml>
):
- 0.8.{27,28}
- 0.9.{11,12}
- 0.10.{37,38}
- 0.11.{15,16}
- 0.12.{1,2}
php-buildpack v3.2.0
Full release notes are here
<https://github.com/cloudfoundry/php-buildpack/releases/tag/v3.2.0>.
Size reduced 27% from 1.1GB
<https://github.com/cloudfoundry/php-buildpack/releases/tag/v3.1.1> to
803MB
<https://github.com/cloudfoundry/php-buildpack/releases/tag/v3.2.0>.
Supports: (full manifest here
<https://github.com/cloudfoundry/php-buildpack/blob/v3.2.0/manifest.yml>
)
*PHP*:
- 5.6.{6,7}
- 5.5.{22,23}
- 5.4.{38,39}
*HHVM* (lucid64 stack):
- 3.2.0
*HHVM* (cflinuxfs2 stack):
- 3.5.{0,1}
- 3.6.{0,1}
*Apache HTTPD*:
- 2.4.12
*nginx*:
- 1.7.10
- 1.6.2
- 1.5.13
python-buildpack v1.3.0
Full release notes are here
<https://github.com/cloudfoundry/python-buildpack/releases/tag/v1.3.0>.
Size reduced 30% from 654MB
<https://github.com/cloudfoundry/python-buildpack/releases/tag/v1.2.0>
to 454MB
<https://github.com/cloudfoundry/python-buildpack/releases/tag/v1.3.0>.
Supports: (full manifest here
<https://github.com/cloudfoundry/python-buildpack/blob/v1.3.0/manifest.yml>
)
- 2.7.{8,9}
- 3.2.{4,5}
- 3.3.{5,6}
- 3.4.{2,3}
ruby-buildpack v1.4.0
Release notes are here
<https://github.com/cloudfoundry/ruby-buildpack/releases/tag/v1.4.0>.
Size reduced 71% from 1.3GB
<https://github.com/cloudfoundry/ruby-buildpack/releases/tag/v1.3.1> to
365MB
<https://github.com/cloudfoundry/ruby-buildpack/releases/tag/v1.4.0>.
Supports: (full manifest here
<https://github.com/cloudfoundry/ruby-buildpack/blob/v1.4.0/manifest.yml>
)
*MRI*:
- 2.2.{1,2}
- 2.1.{5,6}
- 2.0.0p645
*JRuby*:
- ruby-1.9.3-jruby-1.7.19
- ruby-2.0.0-jruby-1.7.19
- ruby-2.2.0-jruby-9.0.0.0.pre1
---------- Forwarded message ----------
From: Mike Dalessio <mdalessio(a)pivotal.io>
Date: Wed, Apr 8, 2015 at 11:10 AM
Subject: Addressing buildpack size
To: vcap-dev(a)cloudfoundry.org
Hello vcap-dev!
This email details a proposed change to how Cloud Foundry buildpacks are
packaged, with respect to the ever-increasing number of binary dependencies
being cached within them.
This proposal's permanent residence is here:
https://github.com/cloudfoundry-incubator/buildpack-packager/issues/4
Feel free to comment there or reply to this email.
------------------------------
Buildpack SizesWhere we are today
Many of you have seen, and possibly been challenged by, the enormous
sizes of some of the buildpacks that are currently shipping with cf-release.
Here's the state of the world right now, as of v205:
php-buildpack: 1.1G
ruby-buildpack: 922M
go-buildpack: 675M
python-buildpack: 654M
nodejs-buildpack: 403M
----------------------
total: 3.7G
These enormous sizes are the result of the current policy of packaging
every-version-of-everything-ever-supported ("EVOEES") within the buildpack.
Most recently, this problem was exacerbated by the fact that buildpacks
now contain binaries for two rootfses.
Why this is a problem
If continued, buildpacks will only continue to increase in size, leading
to longer and longer build and deploy times, longer test times, slacker
feedback loops, and therefore less frequent buildpack releases.
Additionally, this also means that we're shipping versions of
interpreters, web servers, and libraries that are deprecated, insecure, or
both. Feedback from CF users has made it clear that many companies view
this as an unnecessary security risk.
This policy is clearly unsustainable.
What we can do about it
There are many things being discussed to ameliorate the impact that
buildpack size is having on the operations of CF.
Notably, Onsi has proposed a change to buildpack caching, to improve
Diego staging times (link to proposal
<https://github.com/pivotal-cf-experimental/diego-dev-notes/blob/master/proposals/better-buildpack-caching.md>
).
However, there is an immediate solution available, which addresses both
the size concerns as well as the security concern: packaging fewer binary
dependencies within the buildpack.
The proposal
I'm proposing that we reduce the binary dependencies in each buildpack
in a very specific way.
Aside on terms I'll use below:
- Versions of the form "1.2.3" are broken down as:
MAJOR.MINOR.TEENY. Many language ecosystems refer to the "TEENY" as "PATCH"
interchangeably, but we're going to use "TEENY" in this proposal.
- We'll assume that TEENY gets bumped for API/ABI compatible changes.
- We'll assume that MINOR and MAJOR get bumped when there are
API/ABI *incompatible* changes.
I'd like to move forward soon with the following changes:
1. For language interpreters/compilers, we'll package the two
most-recent TEENY versions on each MAJOR.MINOR release.
2. For all other dependencies, we'll package only the single
most-recent TEENY version on each MAJOR.MINOR release.
3. We will discontinue packaging versions of dependencies that have
been deprecated.
4. We will no longer provide "EVOEES" buildpack releases.
5. We will no longer provide "online" buildpack releases, which
download dependencies from the public internet.
6. We will document the process, and provide tooling, for CF
operators to build their own buildpacks, choosing the dependencies that
their organization wants to support or creating "online" buildpacks at
operators' discretion.
An example for #1 is that we'll go from packaging 34 versions of node v0.10.x
to only packaging two: 0.10.37 and 0.10.38.
An example for #2 is that we'll go from packaging 3 versions of nginx 1.5
in the PHP buildpack to only packaging one: 1.5.12.
An example for #3 is that we'll discontinue packaging ruby 1.9.3 in the
ruby-buildpack, which reached end-of-life in February 2015.
Outcomes
With these changes, the total buildpack size will be reduced greatly. As
an example, we expect the ruby-buildpack size to go from 922M to 338M.
We also want to set the expectation that, as new interpreter versions
are released, either for new features or (more urgently) for security
fixes, we'll release new buildpacks much more quickly than we do today. My
hope is that we'll be able to do it within 24 hours of a new release.
Planning
These changes will be relatively easy to make, since all the buildpacks
are now using a manifest.yml file to declare what's being packaged. We
expect to be able to complete this work within the next two weeks.
Stories are in the Tracker backlog under the Epic named
"skinny-buildpacks", which you can see here:
https://www.pivotaltracker.com/epic/show/1747328
------------------------------
Please let me know how these changes will impact you and your
organizations, and let me know of any counter-proposals or variations you'd
like to consider.
Thanks,
-mike
_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
Mike Dalessio
Jack,
On Mon, May 4, 2015 at 2:43 PM, Jack Cai <greensight(a)gmail.com> wrote:
this on the BOSH manifest level. I'm sure we'll get there eventually, but
there is obviously quite a bit of work to get there.
The good news, though, is that the best first step has already been made,
which was to extract dependencies out of the upstream buildpack code, and
declare it in a buildpack manifest.
In the meantime, we'll do our best to make sure operator tools are
available and easy to use to manipulate the buildpacks manifests and create
custom buildpacks.
On Mon, May 4, 2015 at 2:43 PM, Jack Cai <greensight(a)gmail.com> wrote:
+1You're absolutely right, it would be tremendous if it were possible to do
Thanks for the great work!Over the next few days, the buildpacks core team will ship documentationand tooling to assist you in packaging specific dependencies for your
instance of CF. I'll start a new thread on this list early next week to
communicate this information.
I hope this will be easy to customize as part of a bosh release
configuration. Specifically, it would be even better if the cloud operator
can customize some of the binary download URLs in the configuration, so
that they can use their own binaries. As I know, many enterprises only use
legal-cleared binary versions of open source components, hosted inside
their firewall. I understand today this can be achieved by modifying the
manifest.yml in each buildpack. But it would greater if it can be done
through some build/package configuration.
this on the BOSH manifest level. I'm sure we'll get there eventually, but
there is obviously quite a bit of work to get there.
The good news, though, is that the best first step has already been made,
which was to extract dependencies out of the upstream buildpack code, and
declare it in a buildpack manifest.
In the meantime, we'll do our best to make sure operator tools are
available and easy to use to manipulate the buildpacks manifests and create
custom buildpacks.
Jack
On Mon, May 4, 2015 at 1:28 PM, Onsi Fakhouri <ofakhouri(a)pivotal.io>
wrote:the go community tends to move fast to adopt the latest versions of go.
i imagine we can drop 1.1 and 1.2 without impacting most people.
anyone on the list experience otherwise?
onsi
On Mon, May 4, 2015 at 9:40 AM, Mike Dalessio <mdalessio(a)pivotal.io>
wrote:Hi Wayne,_______________________________________________
On Fri, May 1, 2015 at 1:29 PM, Wayne E. Seguin <
wayneeseguin(a)starkandwayne.com> wrote:What an incredible step in the right direction, Awesome!!!Thanks for asking this question.
Out of curiosity, why is the go buildpack still quite so large?
Currently we're including the following binary dependencies in
`go-buildpack`:
```
cache $ ls -lSh *_go*
-rw-r--r-- 1 flavorjones flavorjones 60M 2015-05-04 12:36
https___storage.googleapis.com_golang_go1.4.2.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 60M 2015-05-04 12:36
https___storage.googleapis.com_golang_go1.4.1.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 54M 2015-05-04 12:36
https___storage.googleapis.com_golang_go1.2.2.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 54M 2015-05-04 12:36
http___go.googlecode.com_files_go1.2.1.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 51M 2015-05-04 12:36
https___storage.googleapis.com_golang_go1.3.3.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 51M 2015-05-04 12:36
https___storage.googleapis.com_golang_go1.3.2.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 40M 2015-05-04 12:36
http___go.googlecode.com_files_go1.1.2.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 40M 2015-05-04 12:36
http___go.googlecode.com_files_go1.1.1.linux-amd64.tar.gz
```
One question we should ask, I think, is: should we still be supporting
golang 1.1 and 1.2? Dropping those versions would cut the size of the
buildpack in (approximately) half._______________________________________________
On May 1, 2015, at 11:54 , Mike Dalessio <mdalessio(a)pivotal.io> wrote:
Skinny buildpacks have been cut for go, nodejs, php, python and ruby
buildpacks.
| | current | previous |
|--------+---------+----------|
| go | 442MB | 633MB |
| nodejs | 69MB | 417MB |
| php | 804MB | 1.1GB |
| python | 454MB | 654MB |
| ruby | 365MB | 1.3GB |
|--------+---------+----------|
| total | 2.1GB | 4.1GB |
for an aggregate 51% reduction in size. Details follow.
Next Steps
I recognize that every cloud operator may have a different policy on
what versions of interpreters and libraries they want to support, based on
the specific requirements of their users.
These buildpacks reflect a "bare mininum" policy for a cloud to be
operable, and I do not expect these buildpacks to be adopted as-is by many
operators.
These buildpacks have not yet been added to cf-release, specifically
so that the community can prepare their own buildpacks if necessary.
Over the next few days, the buildpacks core team will ship
documentation and tooling to assist you in packaging specific dependencies
for your instance of CF. I'll start a new thread on this list early next
week to communicate this information.
Call to Action
In the meantime, please think about whether the policy implemented in
these buildpacks ("last two patches (or teenies) on all supported
major.minor releases") is suitable for your users; and if not, think about
what dependencies you'll ideally be supporting.
go-buildpack v1.3.0
Release notes are here
<https://github.com/cloudfoundry/go-buildpack/releases/tag/v1.3.0>.
Size reduced 30% from 633MB
<https://github.com/cloudfoundry/go-buildpack/releases/tag/v1.2.0> to
442MB
<https://github.com/cloudfoundry/go-buildpack/releases/tag/v1.3.0>.
Supports (full manifest here
<https://github.com/cloudfoundry/go-buildpack/blob/v1.3.0/manifest.yml>
):
- golang 1.4.{1,2}
- golang 1.3.{2,3}
- golang 1.2.{1,2}
- golang 1.1.{1,2}
nodejs-buildpack v1.3.0
Full release notes are here
<https://github.com/cloudfoundry/nodejs-buildpack/releases/tag/v1.3.0>.
Size reduced 83% from 417MB
<https://github.com/cloudfoundry/nodejs-buildpack/releases/tag/v1.2.1>
to 69MB
<https://github.com/cloudfoundry/nodejs-buildpack/releases/tag/v1.3.0>.
Supports (full manifest here
<https://github.com/cloudfoundry/nodejs-buildpack/blob/v1.3.0/manifest.yml>
):
- 0.8.{27,28}
- 0.9.{11,12}
- 0.10.{37,38}
- 0.11.{15,16}
- 0.12.{1,2}
php-buildpack v3.2.0
Full release notes are here
<https://github.com/cloudfoundry/php-buildpack/releases/tag/v3.2.0>.
Size reduced 27% from 1.1GB
<https://github.com/cloudfoundry/php-buildpack/releases/tag/v3.1.1> to
803MB
<https://github.com/cloudfoundry/php-buildpack/releases/tag/v3.2.0>.
Supports: (full manifest here
<https://github.com/cloudfoundry/php-buildpack/blob/v3.2.0/manifest.yml>
)
*PHP*:
- 5.6.{6,7}
- 5.5.{22,23}
- 5.4.{38,39}
*HHVM* (lucid64 stack):
- 3.2.0
*HHVM* (cflinuxfs2 stack):
- 3.5.{0,1}
- 3.6.{0,1}
*Apache HTTPD*:
- 2.4.12
*nginx*:
- 1.7.10
- 1.6.2
- 1.5.13
python-buildpack v1.3.0
Full release notes are here
<https://github.com/cloudfoundry/python-buildpack/releases/tag/v1.3.0>.
Size reduced 30% from 654MB
<https://github.com/cloudfoundry/python-buildpack/releases/tag/v1.2.0>
to 454MB
<https://github.com/cloudfoundry/python-buildpack/releases/tag/v1.3.0>.
Supports: (full manifest here
<https://github.com/cloudfoundry/python-buildpack/blob/v1.3.0/manifest.yml>
)
- 2.7.{8,9}
- 3.2.{4,5}
- 3.3.{5,6}
- 3.4.{2,3}
ruby-buildpack v1.4.0
Release notes are here
<https://github.com/cloudfoundry/ruby-buildpack/releases/tag/v1.4.0>.
Size reduced 71% from 1.3GB
<https://github.com/cloudfoundry/ruby-buildpack/releases/tag/v1.3.1>
to 365MB
<https://github.com/cloudfoundry/ruby-buildpack/releases/tag/v1.4.0>.
Supports: (full manifest here
<https://github.com/cloudfoundry/ruby-buildpack/blob/v1.4.0/manifest.yml>
)
*MRI*:
- 2.2.{1,2}
- 2.1.{5,6}
- 2.0.0p645
*JRuby*:
- ruby-1.9.3-jruby-1.7.19
- ruby-2.0.0-jruby-1.7.19
- ruby-2.2.0-jruby-9.0.0.0.pre1
---------- Forwarded message ----------
From: Mike Dalessio <mdalessio(a)pivotal.io>
Date: Wed, Apr 8, 2015 at 11:10 AM
Subject: Addressing buildpack size
To: vcap-dev(a)cloudfoundry.org
Hello vcap-dev!
This email details a proposed change to how Cloud Foundry buildpacks
are packaged, with respect to the ever-increasing number of binary
dependencies being cached within them.
This proposal's permanent residence is here:
https://github.com/cloudfoundry-incubator/buildpack-packager/issues/4
Feel free to comment there or reply to this email.
------------------------------
Buildpack SizesWhere we are today
Many of you have seen, and possibly been challenged by, the enormous
sizes of some of the buildpacks that are currently shipping with cf-release.
Here's the state of the world right now, as of v205:
php-buildpack: 1.1G
ruby-buildpack: 922M
go-buildpack: 675M
python-buildpack: 654M
nodejs-buildpack: 403M
----------------------
total: 3.7G
These enormous sizes are the result of the current policy of packaging
every-version-of-everything-ever-supported ("EVOEES") within the buildpack.
Most recently, this problem was exacerbated by the fact that buildpacks
now contain binaries for two rootfses.
Why this is a problem
If continued, buildpacks will only continue to increase in size,
leading to longer and longer build and deploy times, longer test times,
slacker feedback loops, and therefore less frequent buildpack releases.
Additionally, this also means that we're shipping versions of
interpreters, web servers, and libraries that are deprecated, insecure, or
both. Feedback from CF users has made it clear that many companies view
this as an unnecessary security risk.
This policy is clearly unsustainable.
What we can do about it
There are many things being discussed to ameliorate the impact that
buildpack size is having on the operations of CF.
Notably, Onsi has proposed a change to buildpack caching, to improve
Diego staging times (link to proposal
<https://github.com/pivotal-cf-experimental/diego-dev-notes/blob/master/proposals/better-buildpack-caching.md>
).
However, there is an immediate solution available, which addresses both
the size concerns as well as the security concern: packaging fewer binary
dependencies within the buildpack.
The proposal
I'm proposing that we reduce the binary dependencies in each buildpack
in a very specific way.
Aside on terms I'll use below:
- Versions of the form "1.2.3" are broken down as:
MAJOR.MINOR.TEENY. Many language ecosystems refer to the "TEENY" as "PATCH"
interchangeably, but we're going to use "TEENY" in this proposal.
- We'll assume that TEENY gets bumped for API/ABI compatible
changes.
- We'll assume that MINOR and MAJOR get bumped when there are
API/ABI *incompatible* changes.
I'd like to move forward soon with the following changes:
1. For language interpreters/compilers, we'll package the two
most-recent TEENY versions on each MAJOR.MINOR release.
2. For all other dependencies, we'll package only the single
most-recent TEENY version on each MAJOR.MINOR release.
3. We will discontinue packaging versions of dependencies that have
been deprecated.
4. We will no longer provide "EVOEES" buildpack releases.
5. We will no longer provide "online" buildpack releases, which
download dependencies from the public internet.
6. We will document the process, and provide tooling, for CF
operators to build their own buildpacks, choosing the dependencies that
their organization wants to support or creating "online" buildpacks at
operators' discretion.
An example for #1 is that we'll go from packaging 34 versions of node v0.10.x
to only packaging two: 0.10.37 and 0.10.38.
An example for #2 is that we'll go from packaging 3 versions of nginx 1.5
in the PHP buildpack to only packaging one: 1.5.12.
An example for #3 is that we'll discontinue packaging ruby 1.9.3 in the
ruby-buildpack, which reached end-of-life in February 2015.
Outcomes
With these changes, the total buildpack size will be reduced greatly.
As an example, we expect the ruby-buildpack size to go from 922M to 338M.
We also want to set the expectation that, as new interpreter versions
are released, either for new features or (more urgently) for security
fixes, we'll release new buildpacks much more quickly than we do today. My
hope is that we'll be able to do it within 24 hours of a new release.
Planning
These changes will be relatively easy to make, since all the buildpacks
are now using a manifest.yml file to declare what's being packaged. We
expect to be able to complete this work within the next two weeks.
Stories are in the Tracker backlog under the Epic named
"skinny-buildpacks", which you can see here:
https://www.pivotaltracker.com/epic/show/1747328
------------------------------
Please let me know how these changes will impact you and your
organizations, and let me know of any counter-proposals or variations you'd
like to consider.
Thanks,
-mike
_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
Mike Dalessio
Hi Wayne,
Thanks for thinking about this problem.
On Mon, May 4, 2015 at 2:10 PM, Wayne E. Seguin <
wayneeseguin(a)starkandwayne.com> wrote:
the buildpacks, so it won't save that operation when DEAs roll. Let me know
if you're thinking of something else?
stack-specific.
So the Ruby buildpack, for example, wouldn't have to package binaries for
both `cflinuxfs2` and `lucid64` (though this is complicated by the
additional presence of stack-agnostic packages like JRuby).
But if we did this, then you only have to use the buildpacks for the
stack(s) in your CF deployment. Because, really, asking a buildpack to
contain binaries for every supported stack isn't really a scalable
practice; though we get away with it in a world with only 1 or 2 stacks.
Thanks for thinking about this problem.
On Mon, May 4, 2015 at 2:10 PM, Wayne E. Seguin <
wayneeseguin(a)starkandwayne.com> wrote:
Because of very good compatibility between versions (post 1.X) I wouldWhich operations did you have in mind? Currently the DEAs download *all*
like to make a motion to do the following:
Split the buildpack:
have the default golang buildpack track the latest golang version
Then handle older versions in one of two ways, either:
a) have a large secondary for older versions
or
b) have multiple, one for each version of golang, users can specify a
specific URL if they care about specific versions.
This would improve space/time considerations for operations.
the buildpacks, so it won't save that operation when DEAs roll. Let me know
if you're thinking of something else?
Personally I would prefer b) because it allows you to enable supportingI personally would like to see buildpacks have the option of being
older go versions out of the box by design but still keeping each golang
buildpack small.
stack-specific.
So the Ruby buildpack, for example, wouldn't have to package binaries for
both `cflinuxfs2` and `lucid64` (though this is complicated by the
additional presence of stack-agnostic packages like JRuby).
But if we did this, then you only have to use the buildpacks for the
stack(s) in your CF deployment. Because, really, asking a buildpack to
contain binaries for every supported stack isn't really a scalable
practice; though we get away with it in a world with only 1 or 2 stacks.
~Wayne
Wayne E. Seguin <wayneeseguin(a)starkandwayne.com>
CTO ; Stark & Wayne, LLC
On May 4, 2015, at 12:40 , Mike Dalessio <mdalessio(a)pivotal.io> wrote:
Hi Wayne,
On Fri, May 1, 2015 at 1:29 PM, Wayne E. Seguin <
wayneeseguin(a)starkandwayne.com> wrote:What an incredible step in the right direction, Awesome!!!Thanks for asking this question.
Out of curiosity, why is the go buildpack still quite so large?
Currently we're including the following binary dependencies in
`go-buildpack`:
```
cache $ ls -lSh *_go*
-rw-r--r-- 1 flavorjones flavorjones 60M 2015-05-04 12:36
https___storage.googleapis.com_golang_go1.4.2.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 60M 2015-05-04 12:36
https___storage.googleapis.com_golang_go1.4.1.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 54M 2015-05-04 12:36
https___storage.googleapis.com_golang_go1.2.2.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 54M 2015-05-04 12:36
http___go.googlecode.com_files_go1.2.1.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 51M 2015-05-04 12:36
https___storage.googleapis.com_golang_go1.3.3.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 51M 2015-05-04 12:36
https___storage.googleapis.com_golang_go1.3.2.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 40M 2015-05-04 12:36
http___go.googlecode.com_files_go1.1.2.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 40M 2015-05-04 12:36
http___go.googlecode.com_files_go1.1.1.linux-amd64.tar.gz
```
One question we should ask, I think, is: should we still be supporting
golang 1.1 and 1.2? Dropping those versions would cut the size of the
buildpack in (approximately) half.
On May 1, 2015, at 11:54 , Mike Dalessio <mdalessio(a)pivotal.io> wrote:
Skinny buildpacks have been cut for go, nodejs, php, python and ruby
buildpacks.
| | current | previous |
|--------+---------+----------|
| go | 442MB | 633MB |
| nodejs | 69MB | 417MB |
| php | 804MB | 1.1GB |
| python | 454MB | 654MB |
| ruby | 365MB | 1.3GB |
|--------+---------+----------|
| total | 2.1GB | 4.1GB |
for an aggregate 51% reduction in size. Details follow.
Next Steps
I recognize that every cloud operator may have a different policy on what
versions of interpreters and libraries they want to support, based on the
specific requirements of their users.
These buildpacks reflect a "bare mininum" policy for a cloud to be
operable, and I do not expect these buildpacks to be adopted as-is by many
operators.
These buildpacks have not yet been added to cf-release, specifically so
that the community can prepare their own buildpacks if necessary.
Over the next few days, the buildpacks core team will ship documentation
and tooling to assist you in packaging specific dependencies for your
instance of CF. I'll start a new thread on this list early next week to
communicate this information.
Call to Action
In the meantime, please think about whether the policy implemented in
these buildpacks ("last two patches (or teenies) on all supported
major.minor releases") is suitable for your users; and if not, think about
what dependencies you'll ideally be supporting.
go-buildpack v1.3.0
Release notes are here
<https://github.com/cloudfoundry/go-buildpack/releases/tag/v1.3.0>.
Size reduced 30% from 633MB
<https://github.com/cloudfoundry/go-buildpack/releases/tag/v1.2.0> to
442MB <https://github.com/cloudfoundry/go-buildpack/releases/tag/v1.3.0>.
Supports (full manifest here
<https://github.com/cloudfoundry/go-buildpack/blob/v1.3.0/manifest.yml>):
- golang 1.4.{1,2}
- golang 1.3.{2,3}
- golang 1.2.{1,2}
- golang 1.1.{1,2}
nodejs-buildpack v1.3.0
Full release notes are here
<https://github.com/cloudfoundry/nodejs-buildpack/releases/tag/v1.3.0>.
Size reduced 83% from 417MB
<https://github.com/cloudfoundry/nodejs-buildpack/releases/tag/v1.2.1>
to 69MB
<https://github.com/cloudfoundry/nodejs-buildpack/releases/tag/v1.3.0>.
Supports (full manifest here
<https://github.com/cloudfoundry/nodejs-buildpack/blob/v1.3.0/manifest.yml>
):
- 0.8.{27,28}
- 0.9.{11,12}
- 0.10.{37,38}
- 0.11.{15,16}
- 0.12.{1,2}
php-buildpack v3.2.0
Full release notes are here
<https://github.com/cloudfoundry/php-buildpack/releases/tag/v3.2.0>.
Size reduced 27% from 1.1GB
<https://github.com/cloudfoundry/php-buildpack/releases/tag/v3.1.1> to
803MB <https://github.com/cloudfoundry/php-buildpack/releases/tag/v3.2.0>
.
Supports: (full manifest here
<https://github.com/cloudfoundry/php-buildpack/blob/v3.2.0/manifest.yml>)
*PHP*:
- 5.6.{6,7}
- 5.5.{22,23}
- 5.4.{38,39}
*HHVM* (lucid64 stack):
- 3.2.0
*HHVM* (cflinuxfs2 stack):
- 3.5.{0,1}
- 3.6.{0,1}
*Apache HTTPD*:
- 2.4.12
*nginx*:
- 1.7.10
- 1.6.2
- 1.5.13
python-buildpack v1.3.0
Full release notes are here
<https://github.com/cloudfoundry/python-buildpack/releases/tag/v1.3.0>.
Size reduced 30% from 654MB
<https://github.com/cloudfoundry/python-buildpack/releases/tag/v1.2.0>
to 454MB
<https://github.com/cloudfoundry/python-buildpack/releases/tag/v1.3.0>.
Supports: (full manifest here
<https://github.com/cloudfoundry/python-buildpack/blob/v1.3.0/manifest.yml>
)
- 2.7.{8,9}
- 3.2.{4,5}
- 3.3.{5,6}
- 3.4.{2,3}
ruby-buildpack v1.4.0
Release notes are here
<https://github.com/cloudfoundry/ruby-buildpack/releases/tag/v1.4.0>.
Size reduced 71% from 1.3GB
<https://github.com/cloudfoundry/ruby-buildpack/releases/tag/v1.3.1> to
365MB
<https://github.com/cloudfoundry/ruby-buildpack/releases/tag/v1.4.0>.
Supports: (full manifest here
<https://github.com/cloudfoundry/ruby-buildpack/blob/v1.4.0/manifest.yml>
)
*MRI*:
- 2.2.{1,2}
- 2.1.{5,6}
- 2.0.0p645
*JRuby*:
- ruby-1.9.3-jruby-1.7.19
- ruby-2.0.0-jruby-1.7.19
- ruby-2.2.0-jruby-9.0.0.0.pre1
---------- Forwarded message ----------
From: Mike Dalessio <mdalessio(a)pivotal.io>
Date: Wed, Apr 8, 2015 at 11:10 AM
Subject: Addressing buildpack size
To: vcap-dev(a)cloudfoundry.org
Hello vcap-dev!
This email details a proposed change to how Cloud Foundry buildpacks are
packaged, with respect to the ever-increasing number of binary dependencies
being cached within them.
This proposal's permanent residence is here:
https://github.com/cloudfoundry-incubator/buildpack-packager/issues/4
Feel free to comment there or reply to this email.
------------------------------
Buildpack SizesWhere we are today
Many of you have seen, and possibly been challenged by, the enormous
sizes of some of the buildpacks that are currently shipping with cf-release.
Here's the state of the world right now, as of v205:
php-buildpack: 1.1G
ruby-buildpack: 922M
go-buildpack: 675M
python-buildpack: 654M
nodejs-buildpack: 403M
----------------------
total: 3.7G
These enormous sizes are the result of the current policy of packaging
every-version-of-everything-ever-supported ("EVOEES") within the buildpack.
Most recently, this problem was exacerbated by the fact that buildpacks
now contain binaries for two rootfses.
Why this is a problem
If continued, buildpacks will only continue to increase in size, leading
to longer and longer build and deploy times, longer test times, slacker
feedback loops, and therefore less frequent buildpack releases.
Additionally, this also means that we're shipping versions of
interpreters, web servers, and libraries that are deprecated, insecure, or
both. Feedback from CF users has made it clear that many companies view
this as an unnecessary security risk.
This policy is clearly unsustainable.
What we can do about it
There are many things being discussed to ameliorate the impact that
buildpack size is having on the operations of CF.
Notably, Onsi has proposed a change to buildpack caching, to improve
Diego staging times (link to proposal
<https://github.com/pivotal-cf-experimental/diego-dev-notes/blob/master/proposals/better-buildpack-caching.md>
).
However, there is an immediate solution available, which addresses both
the size concerns as well as the security concern: packaging fewer binary
dependencies within the buildpack.
The proposal
I'm proposing that we reduce the binary dependencies in each buildpack in
a very specific way.
Aside on terms I'll use below:
- Versions of the form "1.2.3" are broken down as: MAJOR.MINOR.TEENY.
Many language ecosystems refer to the "TEENY" as "PATCH" interchangeably,
but we're going to use "TEENY" in this proposal.
- We'll assume that TEENY gets bumped for API/ABI compatible changes.
- We'll assume that MINOR and MAJOR get bumped when there are API/ABI
*incompatible* changes.
I'd like to move forward soon with the following changes:
1. For language interpreters/compilers, we'll package the two
most-recent TEENY versions on each MAJOR.MINOR release.
2. For all other dependencies, we'll package only the single
most-recent TEENY version on each MAJOR.MINOR release.
3. We will discontinue packaging versions of dependencies that have
been deprecated.
4. We will no longer provide "EVOEES" buildpack releases.
5. We will no longer provide "online" buildpack releases, which
download dependencies from the public internet.
6. We will document the process, and provide tooling, for CF
operators to build their own buildpacks, choosing the dependencies that
their organization wants to support or creating "online" buildpacks at
operators' discretion.
An example for #1 is that we'll go from packaging 34 versions of node v0.10.x
to only packaging two: 0.10.37 and 0.10.38.
An example for #2 is that we'll go from packaging 3 versions of nginx 1.5
in the PHP buildpack to only packaging one: 1.5.12.
An example for #3 is that we'll discontinue packaging ruby 1.9.3 in the
ruby-buildpack, which reached end-of-life in February 2015.
Outcomes
With these changes, the total buildpack size will be reduced greatly. As
an example, we expect the ruby-buildpack size to go from 922M to 338M.
We also want to set the expectation that, as new interpreter versions are
released, either for new features or (more urgently) for security fixes,
we'll release new buildpacks much more quickly than we do today. My hope is
that we'll be able to do it within 24 hours of a new release.
Planning
These changes will be relatively easy to make, since all the buildpacks
are now using a manifest.yml file to declare what's being packaged. We
expect to be able to complete this work within the next two weeks.
Stories are in the Tracker backlog under the Epic named
"skinny-buildpacks", which you can see here:
https://www.pivotaltracker.com/epic/show/1747328
------------------------------
Please let me know how these changes will impact you and your
organizations, and let me know of any counter-proposals or variations you'd
like to consider.
Thanks,
-mike
_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
Daniel Mikusa
I'm happy to see the size of the build packs dropping, but I have to ask
why do we bundle the build packs with a fixed set of binaries?
The build packs themselves are very small, it's the binaries that are
huge. It seems like it would make sense to handle them as separate
concerns.
I don't want to come off too harsh, but in addition to the size of the
build packs when bundled with binaries, there are some other disadvantages
to doing things this way.
- Binaries and build packs are updated at different rates. Binaries are
usually updated often, to pick up new runtimes versions & security fixes;
build packs are generally changed at a slower pace, as features or bug
fixes for them are needed. Bundling the two together, requires an operator
to update the build packs more often, just to get updated binaries. It's
been my experience that users don't (or forget) to update build packs which
means they're likely running with older, possibly insecure runtimes.
- It's difficult to bundle a set of runtime binaries that suite
everyone's needs, different users will update at different rates and will
want different sets of binaries. If build packs and binaries are packaged
together, users will end up needing to find a specific build pack bundle
that contains the runtime they want or users will need to build their own
custom bundles. If build packs and binaries are handled separately, there
will be more flexibility in what binaries a build pack has available as an
operator can manage binaries independently. Wayne's post seems to hit on
this point.
- At some point, I think this has already happened (jruby & java), build
packs are going to start having overlapping sets of binaries. If the
binaries are bundled with the build pack, there's no way that build packs
could ever share binaries.
My personal preference would be to see build packs bundled without binaries
and some other solution, which probably merits a separate thread, for
managing the binaries.
I'm curious to hear what others think or if I've missed something and
bundling build packs and binaries is clearly the way to go.
Dan
PS. If this is something that came up in the PMC, I apologize. I skimmed
the notes, but may have missed it.
On Mon, May 4, 2015 at 2:10 PM, Wayne E. Seguin <
wayneeseguin(a)starkandwayne.com> wrote:
why do we bundle the build packs with a fixed set of binaries?
The build packs themselves are very small, it's the binaries that are
huge. It seems like it would make sense to handle them as separate
concerns.
I don't want to come off too harsh, but in addition to the size of the
build packs when bundled with binaries, there are some other disadvantages
to doing things this way.
- Binaries and build packs are updated at different rates. Binaries are
usually updated often, to pick up new runtimes versions & security fixes;
build packs are generally changed at a slower pace, as features or bug
fixes for them are needed. Bundling the two together, requires an operator
to update the build packs more often, just to get updated binaries. It's
been my experience that users don't (or forget) to update build packs which
means they're likely running with older, possibly insecure runtimes.
- It's difficult to bundle a set of runtime binaries that suite
everyone's needs, different users will update at different rates and will
want different sets of binaries. If build packs and binaries are packaged
together, users will end up needing to find a specific build pack bundle
that contains the runtime they want or users will need to build their own
custom bundles. If build packs and binaries are handled separately, there
will be more flexibility in what binaries a build pack has available as an
operator can manage binaries independently. Wayne's post seems to hit on
this point.
- At some point, I think this has already happened (jruby & java), build
packs are going to start having overlapping sets of binaries. If the
binaries are bundled with the build pack, there's no way that build packs
could ever share binaries.
My personal preference would be to see build packs bundled without binaries
and some other solution, which probably merits a separate thread, for
managing the binaries.
I'm curious to hear what others think or if I've missed something and
bundling build packs and binaries is clearly the way to go.
Dan
PS. If this is something that came up in the PMC, I apologize. I skimmed
the notes, but may have missed it.
On Mon, May 4, 2015 at 2:10 PM, Wayne E. Seguin <
wayneeseguin(a)starkandwayne.com> wrote:
Because of very good compatibility between versions (post 1.X) I would
like to make a motion to do the following:
Split the buildpack:
have the default golang buildpack track the latest golang version
Then handle older versions in one of two ways, either:
a) have a large secondary for older versions
or
b) have multiple, one for each version of golang, users can specify a
specific URL if they care about specific versions.
This would improve space/time considerations for operations. Personally I
would prefer b) because it allows you to enable supporting older go
versions out of the box by design but still keeping each golang buildpack
small.
~Wayne
Wayne E. Seguin <wayneeseguin(a)starkandwayne.com>
CTO ; Stark & Wayne, LLC
On May 4, 2015, at 12:40 , Mike Dalessio <mdalessio(a)pivotal.io> wrote:
Hi Wayne,
On Fri, May 1, 2015 at 1:29 PM, Wayne E. Seguin <
wayneeseguin(a)starkandwayne.com> wrote:What an incredible step in the right direction, Awesome!!!Thanks for asking this question.
Out of curiosity, why is the go buildpack still quite so large?
Currently we're including the following binary dependencies in
`go-buildpack`:
```
cache $ ls -lSh *_go*
-rw-r--r-- 1 flavorjones flavorjones 60M 2015-05-04 12:36
https___storage.googleapis.com_golang_go1.4.2.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 60M 2015-05-04 12:36
https___storage.googleapis.com_golang_go1.4.1.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 54M 2015-05-04 12:36
https___storage.googleapis.com_golang_go1.2.2.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 54M 2015-05-04 12:36
http___go.googlecode.com_files_go1.2.1.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 51M 2015-05-04 12:36
https___storage.googleapis.com_golang_go1.3.3.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 51M 2015-05-04 12:36
https___storage.googleapis.com_golang_go1.3.2.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 40M 2015-05-04 12:36
http___go.googlecode.com_files_go1.1.2.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 40M 2015-05-04 12:36
http___go.googlecode.com_files_go1.1.1.linux-amd64.tar.gz
```
One question we should ask, I think, is: should we still be supporting
golang 1.1 and 1.2? Dropping those versions would cut the size of the
buildpack in (approximately) half.
On May 1, 2015, at 11:54 , Mike Dalessio <mdalessio(a)pivotal.io> wrote:
Skinny buildpacks have been cut for go, nodejs, php, python and ruby
buildpacks.
| | current | previous |
|--------+---------+----------|
| go | 442MB | 633MB |
| nodejs | 69MB | 417MB |
| php | 804MB | 1.1GB |
| python | 454MB | 654MB |
| ruby | 365MB | 1.3GB |
|--------+---------+----------|
| total | 2.1GB | 4.1GB |
for an aggregate 51% reduction in size. Details follow.
Next Steps
I recognize that every cloud operator may have a different policy on what
versions of interpreters and libraries they want to support, based on the
specific requirements of their users.
These buildpacks reflect a "bare mininum" policy for a cloud to be
operable, and I do not expect these buildpacks to be adopted as-is by many
operators.
These buildpacks have not yet been added to cf-release, specifically so
that the community can prepare their own buildpacks if necessary.
Over the next few days, the buildpacks core team will ship documentation
and tooling to assist you in packaging specific dependencies for your
instance of CF. I'll start a new thread on this list early next week to
communicate this information.
Call to Action
In the meantime, please think about whether the policy implemented in
these buildpacks ("last two patches (or teenies) on all supported
major.minor releases") is suitable for your users; and if not, think about
what dependencies you'll ideally be supporting.
go-buildpack v1.3.0
Release notes are here
<https://github.com/cloudfoundry/go-buildpack/releases/tag/v1.3.0>.
Size reduced 30% from 633MB
<https://github.com/cloudfoundry/go-buildpack/releases/tag/v1.2.0> to
442MB <https://github.com/cloudfoundry/go-buildpack/releases/tag/v1.3.0>.
Supports (full manifest here
<https://github.com/cloudfoundry/go-buildpack/blob/v1.3.0/manifest.yml>):
- golang 1.4.{1,2}
- golang 1.3.{2,3}
- golang 1.2.{1,2}
- golang 1.1.{1,2}
nodejs-buildpack v1.3.0
Full release notes are here
<https://github.com/cloudfoundry/nodejs-buildpack/releases/tag/v1.3.0>.
Size reduced 83% from 417MB
<https://github.com/cloudfoundry/nodejs-buildpack/releases/tag/v1.2.1>
to 69MB
<https://github.com/cloudfoundry/nodejs-buildpack/releases/tag/v1.3.0>.
Supports (full manifest here
<https://github.com/cloudfoundry/nodejs-buildpack/blob/v1.3.0/manifest.yml>
):
- 0.8.{27,28}
- 0.9.{11,12}
- 0.10.{37,38}
- 0.11.{15,16}
- 0.12.{1,2}
php-buildpack v3.2.0
Full release notes are here
<https://github.com/cloudfoundry/php-buildpack/releases/tag/v3.2.0>.
Size reduced 27% from 1.1GB
<https://github.com/cloudfoundry/php-buildpack/releases/tag/v3.1.1> to
803MB <https://github.com/cloudfoundry/php-buildpack/releases/tag/v3.2.0>
.
Supports: (full manifest here
<https://github.com/cloudfoundry/php-buildpack/blob/v3.2.0/manifest.yml>)
*PHP*:
- 5.6.{6,7}
- 5.5.{22,23}
- 5.4.{38,39}
*HHVM* (lucid64 stack):
- 3.2.0
*HHVM* (cflinuxfs2 stack):
- 3.5.{0,1}
- 3.6.{0,1}
*Apache HTTPD*:
- 2.4.12
*nginx*:
- 1.7.10
- 1.6.2
- 1.5.13
python-buildpack v1.3.0
Full release notes are here
<https://github.com/cloudfoundry/python-buildpack/releases/tag/v1.3.0>.
Size reduced 30% from 654MB
<https://github.com/cloudfoundry/python-buildpack/releases/tag/v1.2.0>
to 454MB
<https://github.com/cloudfoundry/python-buildpack/releases/tag/v1.3.0>.
Supports: (full manifest here
<https://github.com/cloudfoundry/python-buildpack/blob/v1.3.0/manifest.yml>
)
- 2.7.{8,9}
- 3.2.{4,5}
- 3.3.{5,6}
- 3.4.{2,3}
ruby-buildpack v1.4.0
Release notes are here
<https://github.com/cloudfoundry/ruby-buildpack/releases/tag/v1.4.0>.
Size reduced 71% from 1.3GB
<https://github.com/cloudfoundry/ruby-buildpack/releases/tag/v1.3.1> to
365MB
<https://github.com/cloudfoundry/ruby-buildpack/releases/tag/v1.4.0>.
Supports: (full manifest here
<https://github.com/cloudfoundry/ruby-buildpack/blob/v1.4.0/manifest.yml>
)
*MRI*:
- 2.2.{1,2}
- 2.1.{5,6}
- 2.0.0p645
*JRuby*:
- ruby-1.9.3-jruby-1.7.19
- ruby-2.0.0-jruby-1.7.19
- ruby-2.2.0-jruby-9.0.0.0.pre1
---------- Forwarded message ----------
From: Mike Dalessio <mdalessio(a)pivotal.io>
Date: Wed, Apr 8, 2015 at 11:10 AM
Subject: Addressing buildpack size
To: vcap-dev(a)cloudfoundry.org
Hello vcap-dev!
This email details a proposed change to how Cloud Foundry buildpacks are
packaged, with respect to the ever-increasing number of binary dependencies
being cached within them.
This proposal's permanent residence is here:
https://github.com/cloudfoundry-incubator/buildpack-packager/issues/4
Feel free to comment there or reply to this email.
------------------------------
Buildpack SizesWhere we are today
Many of you have seen, and possibly been challenged by, the enormous
sizes of some of the buildpacks that are currently shipping with cf-release.
Here's the state of the world right now, as of v205:
php-buildpack: 1.1G
ruby-buildpack: 922M
go-buildpack: 675M
python-buildpack: 654M
nodejs-buildpack: 403M
----------------------
total: 3.7G
These enormous sizes are the result of the current policy of packaging
every-version-of-everything-ever-supported ("EVOEES") within the buildpack.
Most recently, this problem was exacerbated by the fact that buildpacks
now contain binaries for two rootfses.
Why this is a problem
If continued, buildpacks will only continue to increase in size, leading
to longer and longer build and deploy times, longer test times, slacker
feedback loops, and therefore less frequent buildpack releases.
Additionally, this also means that we're shipping versions of
interpreters, web servers, and libraries that are deprecated, insecure, or
both. Feedback from CF users has made it clear that many companies view
this as an unnecessary security risk.
This policy is clearly unsustainable.
What we can do about it
There are many things being discussed to ameliorate the impact that
buildpack size is having on the operations of CF.
Notably, Onsi has proposed a change to buildpack caching, to improve
Diego staging times (link to proposal
<https://github.com/pivotal-cf-experimental/diego-dev-notes/blob/master/proposals/better-buildpack-caching.md>
).
However, there is an immediate solution available, which addresses both
the size concerns as well as the security concern: packaging fewer binary
dependencies within the buildpack.
The proposal
I'm proposing that we reduce the binary dependencies in each buildpack in
a very specific way.
Aside on terms I'll use below:
- Versions of the form "1.2.3" are broken down as: MAJOR.MINOR.TEENY.
Many language ecosystems refer to the "TEENY" as "PATCH" interchangeably,
but we're going to use "TEENY" in this proposal.
- We'll assume that TEENY gets bumped for API/ABI compatible changes.
- We'll assume that MINOR and MAJOR get bumped when there are API/ABI
*incompatible* changes.
I'd like to move forward soon with the following changes:
1. For language interpreters/compilers, we'll package the two
most-recent TEENY versions on each MAJOR.MINOR release.
2. For all other dependencies, we'll package only the single
most-recent TEENY version on each MAJOR.MINOR release.
3. We will discontinue packaging versions of dependencies that have
been deprecated.
4. We will no longer provide "EVOEES" buildpack releases.
5. We will no longer provide "online" buildpack releases, which
download dependencies from the public internet.
6. We will document the process, and provide tooling, for CF
operators to build their own buildpacks, choosing the dependencies that
their organization wants to support or creating "online" buildpacks at
operators' discretion.
An example for #1 is that we'll go from packaging 34 versions of node v0.10.x
to only packaging two: 0.10.37 and 0.10.38.
An example for #2 is that we'll go from packaging 3 versions of nginx 1.5
in the PHP buildpack to only packaging one: 1.5.12.
An example for #3 is that we'll discontinue packaging ruby 1.9.3 in the
ruby-buildpack, which reached end-of-life in February 2015.
Outcomes
With these changes, the total buildpack size will be reduced greatly. As
an example, we expect the ruby-buildpack size to go from 922M to 338M.
We also want to set the expectation that, as new interpreter versions are
released, either for new features or (more urgently) for security fixes,
we'll release new buildpacks much more quickly than we do today. My hope is
that we'll be able to do it within 24 hours of a new release.
Planning
These changes will be relatively easy to make, since all the buildpacks
are now using a manifest.yml file to declare what's being packaged. We
expect to be able to complete this work within the next two weeks.
Stories are in the Tracker backlog under the Epic named
"skinny-buildpacks", which you can see here:
https://www.pivotaltracker.com/epic/show/1747328
------------------------------
Please let me know how these changes will impact you and your
organizations, and let me know of any counter-proposals or variations you'd
like to consider.
Thanks,
-mike
_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
Jack Cai
+1
Thanks for the great work!
instance of CF. I'll start a new thread on this list early next week to
communicate this information.
I hope this will be easy to customize as part of a bosh release
configuration. Specifically, it would be even better if the cloud operator
can customize some of the binary download URLs in the configuration, so
that they can use their own binaries. As I know, many enterprises only use
legal-cleared binary versions of open source components, hosted inside
their firewall. I understand today this can be achieved by modifying the
manifest.yml in each buildpack. But it would greater if it can be done
through some build/package configuration.
Jack
On Mon, May 4, 2015 at 1:28 PM, Onsi Fakhouri <ofakhouri(a)pivotal.io> wrote:
Thanks for the great work!
Over the next few days, the buildpacks core team will ship documentationand tooling to assist you in packaging specific dependencies for your
instance of CF. I'll start a new thread on this list early next week to
communicate this information.
I hope this will be easy to customize as part of a bosh release
configuration. Specifically, it would be even better if the cloud operator
can customize some of the binary download URLs in the configuration, so
that they can use their own binaries. As I know, many enterprises only use
legal-cleared binary versions of open source components, hosted inside
their firewall. I understand today this can be achieved by modifying the
manifest.yml in each buildpack. But it would greater if it can be done
through some build/package configuration.
Jack
On Mon, May 4, 2015 at 1:28 PM, Onsi Fakhouri <ofakhouri(a)pivotal.io> wrote:
the go community tends to move fast to adopt the latest versions of go. i
imagine we can drop 1.1 and 1.2 without impacting most people.
anyone on the list experience otherwise?
onsi
On Mon, May 4, 2015 at 9:40 AM, Mike Dalessio <mdalessio(a)pivotal.io>
wrote:Hi Wayne,_______________________________________________
On Fri, May 1, 2015 at 1:29 PM, Wayne E. Seguin <
wayneeseguin(a)starkandwayne.com> wrote:What an incredible step in the right direction, Awesome!!!Thanks for asking this question.
Out of curiosity, why is the go buildpack still quite so large?
Currently we're including the following binary dependencies in
`go-buildpack`:
```
cache $ ls -lSh *_go*
-rw-r--r-- 1 flavorjones flavorjones 60M 2015-05-04 12:36
https___storage.googleapis.com_golang_go1.4.2.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 60M 2015-05-04 12:36
https___storage.googleapis.com_golang_go1.4.1.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 54M 2015-05-04 12:36
https___storage.googleapis.com_golang_go1.2.2.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 54M 2015-05-04 12:36
http___go.googlecode.com_files_go1.2.1.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 51M 2015-05-04 12:36
https___storage.googleapis.com_golang_go1.3.3.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 51M 2015-05-04 12:36
https___storage.googleapis.com_golang_go1.3.2.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 40M 2015-05-04 12:36
http___go.googlecode.com_files_go1.1.2.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 40M 2015-05-04 12:36
http___go.googlecode.com_files_go1.1.1.linux-amd64.tar.gz
```
One question we should ask, I think, is: should we still be supporting
golang 1.1 and 1.2? Dropping those versions would cut the size of the
buildpack in (approximately) half._______________________________________________
On May 1, 2015, at 11:54 , Mike Dalessio <mdalessio(a)pivotal.io> wrote:
Skinny buildpacks have been cut for go, nodejs, php, python and ruby
buildpacks.
| | current | previous |
|--------+---------+----------|
| go | 442MB | 633MB |
| nodejs | 69MB | 417MB |
| php | 804MB | 1.1GB |
| python | 454MB | 654MB |
| ruby | 365MB | 1.3GB |
|--------+---------+----------|
| total | 2.1GB | 4.1GB |
for an aggregate 51% reduction in size. Details follow.
Next Steps
I recognize that every cloud operator may have a different policy on
what versions of interpreters and libraries they want to support, based on
the specific requirements of their users.
These buildpacks reflect a "bare mininum" policy for a cloud to be
operable, and I do not expect these buildpacks to be adopted as-is by many
operators.
These buildpacks have not yet been added to cf-release, specifically so
that the community can prepare their own buildpacks if necessary.
Over the next few days, the buildpacks core team will ship documentation
and tooling to assist you in packaging specific dependencies for your
instance of CF. I'll start a new thread on this list early next week to
communicate this information.
Call to Action
In the meantime, please think about whether the policy implemented in
these buildpacks ("last two patches (or teenies) on all supported
major.minor releases") is suitable for your users; and if not, think about
what dependencies you'll ideally be supporting.
go-buildpack v1.3.0
Release notes are here
<https://github.com/cloudfoundry/go-buildpack/releases/tag/v1.3.0>.
Size reduced 30% from 633MB
<https://github.com/cloudfoundry/go-buildpack/releases/tag/v1.2.0> to
442MB <https://github.com/cloudfoundry/go-buildpack/releases/tag/v1.3.0>
.
Supports (full manifest here
<https://github.com/cloudfoundry/go-buildpack/blob/v1.3.0/manifest.yml>
):
- golang 1.4.{1,2}
- golang 1.3.{2,3}
- golang 1.2.{1,2}
- golang 1.1.{1,2}
nodejs-buildpack v1.3.0
Full release notes are here
<https://github.com/cloudfoundry/nodejs-buildpack/releases/tag/v1.3.0>.
Size reduced 83% from 417MB
<https://github.com/cloudfoundry/nodejs-buildpack/releases/tag/v1.2.1>
to 69MB
<https://github.com/cloudfoundry/nodejs-buildpack/releases/tag/v1.3.0>.
Supports (full manifest here
<https://github.com/cloudfoundry/nodejs-buildpack/blob/v1.3.0/manifest.yml>
):
- 0.8.{27,28}
- 0.9.{11,12}
- 0.10.{37,38}
- 0.11.{15,16}
- 0.12.{1,2}
php-buildpack v3.2.0
Full release notes are here
<https://github.com/cloudfoundry/php-buildpack/releases/tag/v3.2.0>.
Size reduced 27% from 1.1GB
<https://github.com/cloudfoundry/php-buildpack/releases/tag/v3.1.1> to
803MB
<https://github.com/cloudfoundry/php-buildpack/releases/tag/v3.2.0>.
Supports: (full manifest here
<https://github.com/cloudfoundry/php-buildpack/blob/v3.2.0/manifest.yml>
)
*PHP*:
- 5.6.{6,7}
- 5.5.{22,23}
- 5.4.{38,39}
*HHVM* (lucid64 stack):
- 3.2.0
*HHVM* (cflinuxfs2 stack):
- 3.5.{0,1}
- 3.6.{0,1}
*Apache HTTPD*:
- 2.4.12
*nginx*:
- 1.7.10
- 1.6.2
- 1.5.13
python-buildpack v1.3.0
Full release notes are here
<https://github.com/cloudfoundry/python-buildpack/releases/tag/v1.3.0>.
Size reduced 30% from 654MB
<https://github.com/cloudfoundry/python-buildpack/releases/tag/v1.2.0>
to 454MB
<https://github.com/cloudfoundry/python-buildpack/releases/tag/v1.3.0>.
Supports: (full manifest here
<https://github.com/cloudfoundry/python-buildpack/blob/v1.3.0/manifest.yml>
)
- 2.7.{8,9}
- 3.2.{4,5}
- 3.3.{5,6}
- 3.4.{2,3}
ruby-buildpack v1.4.0
Release notes are here
<https://github.com/cloudfoundry/ruby-buildpack/releases/tag/v1.4.0>.
Size reduced 71% from 1.3GB
<https://github.com/cloudfoundry/ruby-buildpack/releases/tag/v1.3.1> to
365MB
<https://github.com/cloudfoundry/ruby-buildpack/releases/tag/v1.4.0>.
Supports: (full manifest here
<https://github.com/cloudfoundry/ruby-buildpack/blob/v1.4.0/manifest.yml>
)
*MRI*:
- 2.2.{1,2}
- 2.1.{5,6}
- 2.0.0p645
*JRuby*:
- ruby-1.9.3-jruby-1.7.19
- ruby-2.0.0-jruby-1.7.19
- ruby-2.2.0-jruby-9.0.0.0.pre1
---------- Forwarded message ----------
From: Mike Dalessio <mdalessio(a)pivotal.io>
Date: Wed, Apr 8, 2015 at 11:10 AM
Subject: Addressing buildpack size
To: vcap-dev(a)cloudfoundry.org
Hello vcap-dev!
This email details a proposed change to how Cloud Foundry buildpacks are
packaged, with respect to the ever-increasing number of binary dependencies
being cached within them.
This proposal's permanent residence is here:
https://github.com/cloudfoundry-incubator/buildpack-packager/issues/4
Feel free to comment there or reply to this email.
------------------------------
Buildpack SizesWhere we are today
Many of you have seen, and possibly been challenged by, the enormous
sizes of some of the buildpacks that are currently shipping with cf-release.
Here's the state of the world right now, as of v205:
php-buildpack: 1.1G
ruby-buildpack: 922M
go-buildpack: 675M
python-buildpack: 654M
nodejs-buildpack: 403M
----------------------
total: 3.7G
These enormous sizes are the result of the current policy of packaging
every-version-of-everything-ever-supported ("EVOEES") within the buildpack.
Most recently, this problem was exacerbated by the fact that buildpacks
now contain binaries for two rootfses.
Why this is a problem
If continued, buildpacks will only continue to increase in size, leading
to longer and longer build and deploy times, longer test times, slacker
feedback loops, and therefore less frequent buildpack releases.
Additionally, this also means that we're shipping versions of
interpreters, web servers, and libraries that are deprecated, insecure, or
both. Feedback from CF users has made it clear that many companies view
this as an unnecessary security risk.
This policy is clearly unsustainable.
What we can do about it
There are many things being discussed to ameliorate the impact that
buildpack size is having on the operations of CF.
Notably, Onsi has proposed a change to buildpack caching, to improve
Diego staging times (link to proposal
<https://github.com/pivotal-cf-experimental/diego-dev-notes/blob/master/proposals/better-buildpack-caching.md>
).
However, there is an immediate solution available, which addresses both
the size concerns as well as the security concern: packaging fewer binary
dependencies within the buildpack.
The proposal
I'm proposing that we reduce the binary dependencies in each buildpack
in a very specific way.
Aside on terms I'll use below:
- Versions of the form "1.2.3" are broken down as:
MAJOR.MINOR.TEENY. Many language ecosystems refer to the "TEENY" as "PATCH"
interchangeably, but we're going to use "TEENY" in this proposal.
- We'll assume that TEENY gets bumped for API/ABI compatible changes.
- We'll assume that MINOR and MAJOR get bumped when there are
API/ABI *incompatible* changes.
I'd like to move forward soon with the following changes:
1. For language interpreters/compilers, we'll package the two
most-recent TEENY versions on each MAJOR.MINOR release.
2. For all other dependencies, we'll package only the single
most-recent TEENY version on each MAJOR.MINOR release.
3. We will discontinue packaging versions of dependencies that have
been deprecated.
4. We will no longer provide "EVOEES" buildpack releases.
5. We will no longer provide "online" buildpack releases, which
download dependencies from the public internet.
6. We will document the process, and provide tooling, for CF
operators to build their own buildpacks, choosing the dependencies that
their organization wants to support or creating "online" buildpacks at
operators' discretion.
An example for #1 is that we'll go from packaging 34 versions of node v0.10.x
to only packaging two: 0.10.37 and 0.10.38.
An example for #2 is that we'll go from packaging 3 versions of nginx 1.5
in the PHP buildpack to only packaging one: 1.5.12.
An example for #3 is that we'll discontinue packaging ruby 1.9.3 in the
ruby-buildpack, which reached end-of-life in February 2015.
Outcomes
With these changes, the total buildpack size will be reduced greatly. As
an example, we expect the ruby-buildpack size to go from 922M to 338M.
We also want to set the expectation that, as new interpreter versions
are released, either for new features or (more urgently) for security
fixes, we'll release new buildpacks much more quickly than we do today. My
hope is that we'll be able to do it within 24 hours of a new release.
Planning
These changes will be relatively easy to make, since all the buildpacks
are now using a manifest.yml file to declare what's being packaged. We
expect to be able to complete this work within the next two weeks.
Stories are in the Tracker backlog under the Epic named
"skinny-buildpacks", which you can see here:
https://www.pivotaltracker.com/epic/show/1747328
------------------------------
Please let me know how these changes will impact you and your
organizations, and let me know of any counter-proposals or variations you'd
like to consider.
Thanks,
-mike
_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
Wayne E. Seguin
Because of very good compatibility between versions (post 1.X) I would like to make a motion to do the following:
Split the buildpack:
have the default golang buildpack track the latest golang version
Then handle older versions in one of two ways, either:
a) have a large secondary for older versions
or
b) have multiple, one for each version of golang, users can specify a specific URL if they care about specific versions.
This would improve space/time considerations for operations. Personally I would prefer b) because it allows you to enable supporting older go versions out of the box by design but still keeping each golang buildpack small.
~Wayne
Wayne E. Seguin <wayneeseguin(a)starkandwayne.com <mailto:wayneeseguin(a)starkandwayne.com>>
CTO ; Stark & Wayne, LLC
toggle quoted message
Show quoted text
Split the buildpack:
have the default golang buildpack track the latest golang version
Then handle older versions in one of two ways, either:
a) have a large secondary for older versions
or
b) have multiple, one for each version of golang, users can specify a specific URL if they care about specific versions.
This would improve space/time considerations for operations. Personally I would prefer b) because it allows you to enable supporting older go versions out of the box by design but still keeping each golang buildpack small.
~Wayne
Wayne E. Seguin <wayneeseguin(a)starkandwayne.com <mailto:wayneeseguin(a)starkandwayne.com>>
CTO ; Stark & Wayne, LLC
On May 4, 2015, at 12:40 , Mike Dalessio <mdalessio(a)pivotal.io> wrote:
Hi Wayne,
On Fri, May 1, 2015 at 1:29 PM, Wayne E. Seguin <wayneeseguin(a)starkandwayne.com <mailto:wayneeseguin(a)starkandwayne.com>> wrote:
What an incredible step in the right direction, Awesome!!!
Out of curiosity, why is the go buildpack still quite so large?
Thanks for asking this question.
Currently we're including the following binary dependencies in `go-buildpack`:
```
cache $ ls -lSh *_go*
-rw-r--r-- 1 flavorjones flavorjones 60M 2015-05-04 12:36 https___storage.googleapis.com_golang_go1.4.2.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 60M 2015-05-04 12:36 https___storage.googleapis.com_golang_go1.4.1.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 54M 2015-05-04 12:36 https___storage.googleapis.com_golang_go1.2.2.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 54M 2015-05-04 12:36 http___go.googlecode.com_files_go1.2.1.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 51M 2015-05-04 12:36 https___storage.googleapis.com_golang_go1.3.3.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 51M 2015-05-04 12:36 https___storage.googleapis.com_golang_go1.3.2.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 40M 2015-05-04 12:36 http___go.googlecode.com_files_go1.1.2.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 40M 2015-05-04 12:36 http___go.googlecode.com_files_go1.1.1.linux-amd64.tar.gz
```
One question we should ask, I think, is: should we still be supporting golang 1.1 and 1.2? Dropping those versions would cut the size of the buildpack in (approximately) half.On May 1, 2015, at 11:54 , Mike Dalessio <mdalessio(a)pivotal.io <mailto:mdalessio(a)pivotal.io>> wrote:
Skinny buildpacks have been cut for go, nodejs, php, python and ruby buildpacks.
| | current | previous |
|--------+---------+----------|
| go | 442MB | 633MB |
| nodejs | 69MB | 417MB |
| php | 804MB | 1.1GB |
| python | 454MB | 654MB |
| ruby | 365MB | 1.3GB |
|--------+---------+----------|
| total | 2.1GB | 4.1GB |
for an aggregate 51% reduction in size. Details follow.
Next Steps
I recognize that every cloud operator may have a different policy on what versions of interpreters and libraries they want to support, based on the specific requirements of their users.
These buildpacks reflect a "bare mininum" policy for a cloud to be operable, and I do not expect these buildpacks to be adopted as-is by many operators.
These buildpacks have not yet been added to cf-release, specifically so that the community can prepare their own buildpacks if necessary.
Over the next few days, the buildpacks core team will ship documentation and tooling to assist you in packaging specific dependencies for your instance of CF. I'll start a new thread on this list early next week to communicate this information.
Call to Action
In the meantime, please think about whether the policy implemented in these buildpacks ("last two patches (or teenies) on all supported major.minor releases") is suitable for your users; and if not, think about what dependencies you'll ideally be supporting.
go-buildpack v1.3.0
Release notes are here <https://github.com/cloudfoundry/go-buildpack/releases/tag/v1.3.0>.
Size reduced 30% from 633MB <https://github.com/cloudfoundry/go-buildpack/releases/tag/v1.2.0> to 442MB <https://github.com/cloudfoundry/go-buildpack/releases/tag/v1.3.0>.
Supports (full manifest here <https://github.com/cloudfoundry/go-buildpack/blob/v1.3.0/manifest.yml>):
golang 1.4.{1,2}
golang 1.3.{2,3}
golang 1.2.{1,2}
golang 1.1.{1,2}
nodejs-buildpack v1.3.0
Full release notes are here <https://github.com/cloudfoundry/nodejs-buildpack/releases/tag/v1.3.0>.
Size reduced 83% from 417MB <https://github.com/cloudfoundry/nodejs-buildpack/releases/tag/v1.2.1> to 69MB <https://github.com/cloudfoundry/nodejs-buildpack/releases/tag/v1.3.0>.
Supports (full manifest here <https://github.com/cloudfoundry/nodejs-buildpack/blob/v1.3.0/manifest.yml>):
0.8.{27,28}
0.9.{11,12}
0.10.{37,38}
0.11.{15,16}
0.12.{1,2}
php-buildpack v3.2.0
Full release notes are here <https://github.com/cloudfoundry/php-buildpack/releases/tag/v3.2.0>.
Size reduced 27% from 1.1GB <https://github.com/cloudfoundry/php-buildpack/releases/tag/v3.1.1> to 803MB <https://github.com/cloudfoundry/php-buildpack/releases/tag/v3.2.0>.
Supports: (full manifest here <https://github.com/cloudfoundry/php-buildpack/blob/v3.2.0/manifest.yml>)
PHP:
5.6.{6,7}
5.5.{22,23}
5.4.{38,39}
HHVM (lucid64 stack):
3.2.0
HHVM (cflinuxfs2 stack):
3.5.{0,1}
3.6.{0,1}
Apache HTTPD:
2.4.12
nginx:
1.7.10
1.6.2
1.5.13
python-buildpack v1.3.0
Full release notes are here <https://github.com/cloudfoundry/python-buildpack/releases/tag/v1.3.0>.
Size reduced 30% from 654MB <https://github.com/cloudfoundry/python-buildpack/releases/tag/v1.2.0> to 454MB <https://github.com/cloudfoundry/python-buildpack/releases/tag/v1.3.0>.
Supports: (full manifest here <https://github.com/cloudfoundry/python-buildpack/blob/v1.3.0/manifest.yml>)
2.7.{8,9}
3.2.{4,5}
3.3.{5,6}
3.4.{2,3}
ruby-buildpack v1.4.0
Release notes are here <https://github.com/cloudfoundry/ruby-buildpack/releases/tag/v1.4.0>.
Size reduced 71% from 1.3GB <https://github.com/cloudfoundry/ruby-buildpack/releases/tag/v1.3.1> to 365MB <https://github.com/cloudfoundry/ruby-buildpack/releases/tag/v1.4.0>.
Supports: (full manifest here <https://github.com/cloudfoundry/ruby-buildpack/blob/v1.4.0/manifest.yml>)
MRI:
2.2.{1,2}
2.1.{5,6}
2.0.0p645
JRuby:
ruby-1.9.3-jruby-1.7.19
ruby-2.0.0-jruby-1.7.19
ruby-2.2.0-jruby-9.0.0.0.pre1
---------- Forwarded message ----------
From: Mike Dalessio <mdalessio(a)pivotal.io <mailto:mdalessio(a)pivotal.io>>
Date: Wed, Apr 8, 2015 at 11:10 AM
Subject: Addressing buildpack size
To: vcap-dev(a)cloudfoundry.org <mailto:vcap-dev(a)cloudfoundry.org>
Hello vcap-dev!
This email details a proposed change to how Cloud Foundry buildpacks are packaged, with respect to the ever-increasing number of binary dependencies being cached within them.
This proposal's permanent residence is here:
https://github.com/cloudfoundry-incubator/buildpack-packager/issues/4 <https://github.com/cloudfoundry-incubator/buildpack-packager/issues/4>
Feel free to comment there or reply to this email.
Buildpack Sizes
Where we are today
Many of you have seen, and possibly been challenged by, the enormous sizes of some of the buildpacks that are currently shipping with cf-release.
Here's the state of the world right now, as of v205:
php-buildpack: 1.1G
ruby-buildpack: 922M
go-buildpack: 675M
python-buildpack: 654M
nodejs-buildpack: 403M
----------------------
total: 3.7G
These enormous sizes are the result of the current policy of packaging every-version-of-everything-ever-supported ("EVOEES") within the buildpack.
Most recently, this problem was exacerbated by the fact that buildpacks now contain binaries for two rootfses.
Why this is a problem
If continued, buildpacks will only continue to increase in size, leading to longer and longer build and deploy times, longer test times, slacker feedback loops, and therefore less frequent buildpack releases.
Additionally, this also means that we're shipping versions of interpreters, web servers, and libraries that are deprecated, insecure, or both. Feedback from CF users has made it clear that many companies view this as an unnecessary security risk.
This policy is clearly unsustainable.
What we can do about it
There are many things being discussed to ameliorate the impact that buildpack size is having on the operations of CF.
Notably, Onsi has proposed a change to buildpack caching, to improve Diego staging times (link to proposal <https://github.com/pivotal-cf-experimental/diego-dev-notes/blob/master/proposals/better-buildpack-caching.md>).
However, there is an immediate solution available, which addresses both the size concerns as well as the security concern: packaging fewer binary dependencies within the buildpack.
The proposal
I'm proposing that we reduce the binary dependencies in each buildpack in a very specific way.
Aside on terms I'll use below:
Versions of the form "1.2.3" are broken down as: MAJOR.MINOR.TEENY. Many language ecosystems refer to the "TEENY" as "PATCH" interchangeably, but we're going to use "TEENY" in this proposal.
We'll assume that TEENY gets bumped for API/ABI compatible changes.
We'll assume that MINOR and MAJOR get bumped when there are API/ABI incompatible changes.
I'd like to move forward soon with the following changes:
For language interpreters/compilers, we'll package the two most-recent TEENY versions on each MAJOR.MINOR release.
For all other dependencies, we'll package only the single most-recent TEENY version on each MAJOR.MINOR release.
We will discontinue packaging versions of dependencies that have been deprecated.
We will no longer provide "EVOEES" buildpack releases.
We will no longer provide "online" buildpack releases, which download dependencies from the public internet.
We will document the process, and provide tooling, for CF operators to build their own buildpacks, choosing the dependencies that their organization wants to support or creating "online" buildpacks at operators' discretion.
An example for #1 is that we'll go from packaging 34 versions of node v0.10.x to only packaging two: 0.10.37 and 0.10.38.
An example for #2 is that we'll go from packaging 3 versions of nginx 1.5 in the PHP buildpack to only packaging one: 1.5.12.
An example for #3 is that we'll discontinue packaging ruby 1.9.3 in the ruby-buildpack, which reached end-of-life in February 2015.
Outcomes
With these changes, the total buildpack size will be reduced greatly. As an example, we expect the ruby-buildpack size to go from 922M to 338M.
We also want to set the expectation that, as new interpreter versions are released, either for new features or (more urgently) for security fixes, we'll release new buildpacks much more quickly than we do today. My hope is that we'll be able to do it within 24 hours of a new release.
Planning
These changes will be relatively easy to make, since all the buildpacks are now using a manifest.yml file to declare what's being packaged. We expect to be able to complete this work within the next two weeks.
Stories are in the Tracker backlog under the Epic named "skinny-buildpacks", which you can see here:
https://www.pivotaltracker.com/epic/show/1747328 <https://www.pivotaltracker.com/epic/show/1747328>
Please let me know how these changes will impact you and your organizations, and let me know of any counter-proposals or variations you'd like to consider.
Thanks,
-mike
_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org <mailto:cf-dev(a)lists.cloudfoundry.org>
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev <https://lists.cloudfoundry.org/mailman/listinfo/cf-dev>
Onsi Fakhouri <ofakhouri@...>
the go community tends to move fast to adopt the latest versions of go. i
imagine we can drop 1.1 and 1.2 without impacting most people.
anyone on the list experience otherwise?
onsi
toggle quoted message
Show quoted text
imagine we can drop 1.1 and 1.2 without impacting most people.
anyone on the list experience otherwise?
onsi
On Mon, May 4, 2015 at 9:40 AM, Mike Dalessio <mdalessio(a)pivotal.io> wrote:
Hi Wayne,
On Fri, May 1, 2015 at 1:29 PM, Wayne E. Seguin <
wayneeseguin(a)starkandwayne.com> wrote:What an incredible step in the right direction, Awesome!!!Thanks for asking this question.
Out of curiosity, why is the go buildpack still quite so large?
Currently we're including the following binary dependencies in
`go-buildpack`:
```
cache $ ls -lSh *_go*
-rw-r--r-- 1 flavorjones flavorjones 60M 2015-05-04 12:36
https___storage.googleapis.com_golang_go1.4.2.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 60M 2015-05-04 12:36
https___storage.googleapis.com_golang_go1.4.1.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 54M 2015-05-04 12:36
https___storage.googleapis.com_golang_go1.2.2.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 54M 2015-05-04 12:36
http___go.googlecode.com_files_go1.2.1.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 51M 2015-05-04 12:36
https___storage.googleapis.com_golang_go1.3.3.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 51M 2015-05-04 12:36
https___storage.googleapis.com_golang_go1.3.2.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 40M 2015-05-04 12:36
http___go.googlecode.com_files_go1.1.2.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 40M 2015-05-04 12:36
http___go.googlecode.com_files_go1.1.1.linux-amd64.tar.gz
```
One question we should ask, I think, is: should we still be supporting
golang 1.1 and 1.2? Dropping those versions would cut the size of the
buildpack in (approximately) half._______________________________________________
On May 1, 2015, at 11:54 , Mike Dalessio <mdalessio(a)pivotal.io> wrote:
Skinny buildpacks have been cut for go, nodejs, php, python and ruby
buildpacks.
| | current | previous |
|--------+---------+----------|
| go | 442MB | 633MB |
| nodejs | 69MB | 417MB |
| php | 804MB | 1.1GB |
| python | 454MB | 654MB |
| ruby | 365MB | 1.3GB |
|--------+---------+----------|
| total | 2.1GB | 4.1GB |
for an aggregate 51% reduction in size. Details follow.
Next Steps
I recognize that every cloud operator may have a different policy on what
versions of interpreters and libraries they want to support, based on the
specific requirements of their users.
These buildpacks reflect a "bare mininum" policy for a cloud to be
operable, and I do not expect these buildpacks to be adopted as-is by many
operators.
These buildpacks have not yet been added to cf-release, specifically so
that the community can prepare their own buildpacks if necessary.
Over the next few days, the buildpacks core team will ship documentation
and tooling to assist you in packaging specific dependencies for your
instance of CF. I'll start a new thread on this list early next week to
communicate this information.
Call to Action
In the meantime, please think about whether the policy implemented in
these buildpacks ("last two patches (or teenies) on all supported
major.minor releases") is suitable for your users; and if not, think about
what dependencies you'll ideally be supporting.
go-buildpack v1.3.0
Release notes are here
<https://github.com/cloudfoundry/go-buildpack/releases/tag/v1.3.0>.
Size reduced 30% from 633MB
<https://github.com/cloudfoundry/go-buildpack/releases/tag/v1.2.0> to
442MB <https://github.com/cloudfoundry/go-buildpack/releases/tag/v1.3.0>.
Supports (full manifest here
<https://github.com/cloudfoundry/go-buildpack/blob/v1.3.0/manifest.yml>):
- golang 1.4.{1,2}
- golang 1.3.{2,3}
- golang 1.2.{1,2}
- golang 1.1.{1,2}
nodejs-buildpack v1.3.0
Full release notes are here
<https://github.com/cloudfoundry/nodejs-buildpack/releases/tag/v1.3.0>.
Size reduced 83% from 417MB
<https://github.com/cloudfoundry/nodejs-buildpack/releases/tag/v1.2.1>
to 69MB
<https://github.com/cloudfoundry/nodejs-buildpack/releases/tag/v1.3.0>.
Supports (full manifest here
<https://github.com/cloudfoundry/nodejs-buildpack/blob/v1.3.0/manifest.yml>
):
- 0.8.{27,28}
- 0.9.{11,12}
- 0.10.{37,38}
- 0.11.{15,16}
- 0.12.{1,2}
php-buildpack v3.2.0
Full release notes are here
<https://github.com/cloudfoundry/php-buildpack/releases/tag/v3.2.0>.
Size reduced 27% from 1.1GB
<https://github.com/cloudfoundry/php-buildpack/releases/tag/v3.1.1> to
803MB <https://github.com/cloudfoundry/php-buildpack/releases/tag/v3.2.0>
.
Supports: (full manifest here
<https://github.com/cloudfoundry/php-buildpack/blob/v3.2.0/manifest.yml>)
*PHP*:
- 5.6.{6,7}
- 5.5.{22,23}
- 5.4.{38,39}
*HHVM* (lucid64 stack):
- 3.2.0
*HHVM* (cflinuxfs2 stack):
- 3.5.{0,1}
- 3.6.{0,1}
*Apache HTTPD*:
- 2.4.12
*nginx*:
- 1.7.10
- 1.6.2
- 1.5.13
python-buildpack v1.3.0
Full release notes are here
<https://github.com/cloudfoundry/python-buildpack/releases/tag/v1.3.0>.
Size reduced 30% from 654MB
<https://github.com/cloudfoundry/python-buildpack/releases/tag/v1.2.0>
to 454MB
<https://github.com/cloudfoundry/python-buildpack/releases/tag/v1.3.0>.
Supports: (full manifest here
<https://github.com/cloudfoundry/python-buildpack/blob/v1.3.0/manifest.yml>
)
- 2.7.{8,9}
- 3.2.{4,5}
- 3.3.{5,6}
- 3.4.{2,3}
ruby-buildpack v1.4.0
Release notes are here
<https://github.com/cloudfoundry/ruby-buildpack/releases/tag/v1.4.0>.
Size reduced 71% from 1.3GB
<https://github.com/cloudfoundry/ruby-buildpack/releases/tag/v1.3.1> to
365MB
<https://github.com/cloudfoundry/ruby-buildpack/releases/tag/v1.4.0>.
Supports: (full manifest here
<https://github.com/cloudfoundry/ruby-buildpack/blob/v1.4.0/manifest.yml>
)
*MRI*:
- 2.2.{1,2}
- 2.1.{5,6}
- 2.0.0p645
*JRuby*:
- ruby-1.9.3-jruby-1.7.19
- ruby-2.0.0-jruby-1.7.19
- ruby-2.2.0-jruby-9.0.0.0.pre1
---------- Forwarded message ----------
From: Mike Dalessio <mdalessio(a)pivotal.io>
Date: Wed, Apr 8, 2015 at 11:10 AM
Subject: Addressing buildpack size
To: vcap-dev(a)cloudfoundry.org
Hello vcap-dev!
This email details a proposed change to how Cloud Foundry buildpacks are
packaged, with respect to the ever-increasing number of binary dependencies
being cached within them.
This proposal's permanent residence is here:
https://github.com/cloudfoundry-incubator/buildpack-packager/issues/4
Feel free to comment there or reply to this email.
------------------------------
Buildpack SizesWhere we are today
Many of you have seen, and possibly been challenged by, the enormous
sizes of some of the buildpacks that are currently shipping with cf-release.
Here's the state of the world right now, as of v205:
php-buildpack: 1.1G
ruby-buildpack: 922M
go-buildpack: 675M
python-buildpack: 654M
nodejs-buildpack: 403M
----------------------
total: 3.7G
These enormous sizes are the result of the current policy of packaging
every-version-of-everything-ever-supported ("EVOEES") within the buildpack.
Most recently, this problem was exacerbated by the fact that buildpacks
now contain binaries for two rootfses.
Why this is a problem
If continued, buildpacks will only continue to increase in size, leading
to longer and longer build and deploy times, longer test times, slacker
feedback loops, and therefore less frequent buildpack releases.
Additionally, this also means that we're shipping versions of
interpreters, web servers, and libraries that are deprecated, insecure, or
both. Feedback from CF users has made it clear that many companies view
this as an unnecessary security risk.
This policy is clearly unsustainable.
What we can do about it
There are many things being discussed to ameliorate the impact that
buildpack size is having on the operations of CF.
Notably, Onsi has proposed a change to buildpack caching, to improve
Diego staging times (link to proposal
<https://github.com/pivotal-cf-experimental/diego-dev-notes/blob/master/proposals/better-buildpack-caching.md>
).
However, there is an immediate solution available, which addresses both
the size concerns as well as the security concern: packaging fewer binary
dependencies within the buildpack.
The proposal
I'm proposing that we reduce the binary dependencies in each buildpack in
a very specific way.
Aside on terms I'll use below:
- Versions of the form "1.2.3" are broken down as: MAJOR.MINOR.TEENY.
Many language ecosystems refer to the "TEENY" as "PATCH" interchangeably,
but we're going to use "TEENY" in this proposal.
- We'll assume that TEENY gets bumped for API/ABI compatible changes.
- We'll assume that MINOR and MAJOR get bumped when there are API/ABI
*incompatible* changes.
I'd like to move forward soon with the following changes:
1. For language interpreters/compilers, we'll package the two
most-recent TEENY versions on each MAJOR.MINOR release.
2. For all other dependencies, we'll package only the single
most-recent TEENY version on each MAJOR.MINOR release.
3. We will discontinue packaging versions of dependencies that have
been deprecated.
4. We will no longer provide "EVOEES" buildpack releases.
5. We will no longer provide "online" buildpack releases, which
download dependencies from the public internet.
6. We will document the process, and provide tooling, for CF
operators to build their own buildpacks, choosing the dependencies that
their organization wants to support or creating "online" buildpacks at
operators' discretion.
An example for #1 is that we'll go from packaging 34 versions of node v0.10.x
to only packaging two: 0.10.37 and 0.10.38.
An example for #2 is that we'll go from packaging 3 versions of nginx 1.5
in the PHP buildpack to only packaging one: 1.5.12.
An example for #3 is that we'll discontinue packaging ruby 1.9.3 in the
ruby-buildpack, which reached end-of-life in February 2015.
Outcomes
With these changes, the total buildpack size will be reduced greatly. As
an example, we expect the ruby-buildpack size to go from 922M to 338M.
We also want to set the expectation that, as new interpreter versions are
released, either for new features or (more urgently) for security fixes,
we'll release new buildpacks much more quickly than we do today. My hope is
that we'll be able to do it within 24 hours of a new release.
Planning
These changes will be relatively easy to make, since all the buildpacks
are now using a manifest.yml file to declare what's being packaged. We
expect to be able to complete this work within the next two weeks.
Stories are in the Tracker backlog under the Epic named
"skinny-buildpacks", which you can see here:
https://www.pivotaltracker.com/epic/show/1747328
------------------------------
Please let me know how these changes will impact you and your
organizations, and let me know of any counter-proposals or variations you'd
like to consider.
Thanks,
-mike
_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
Mike Dalessio
Hi Wayne,
On Fri, May 1, 2015 at 1:29 PM, Wayne E. Seguin <
wayneeseguin(a)starkandwayne.com> wrote:
Currently we're including the following binary dependencies in
`go-buildpack`:
```
cache $ ls -lSh *_go*
-rw-r--r-- 1 flavorjones flavorjones 60M 2015-05-04 12:36
https___storage.googleapis.com_golang_go1.4.2.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 60M 2015-05-04 12:36
https___storage.googleapis.com_golang_go1.4.1.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 54M 2015-05-04 12:36
https___storage.googleapis.com_golang_go1.2.2.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 54M 2015-05-04 12:36
http___go.googlecode.com_files_go1.2.1.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 51M 2015-05-04 12:36
https___storage.googleapis.com_golang_go1.3.3.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 51M 2015-05-04 12:36
https___storage.googleapis.com_golang_go1.3.2.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 40M 2015-05-04 12:36
http___go.googlecode.com_files_go1.1.2.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 40M 2015-05-04 12:36
http___go.googlecode.com_files_go1.1.1.linux-amd64.tar.gz
```
One question we should ask, I think, is: should we still be supporting
golang 1.1 and 1.2? Dropping those versions would cut the size of the
buildpack in (approximately) half.
On Fri, May 1, 2015 at 1:29 PM, Wayne E. Seguin <
wayneeseguin(a)starkandwayne.com> wrote:
What an incredible step in the right direction, Awesome!!!Thanks for asking this question.
Out of curiosity, why is the go buildpack still quite so large?
Currently we're including the following binary dependencies in
`go-buildpack`:
```
cache $ ls -lSh *_go*
-rw-r--r-- 1 flavorjones flavorjones 60M 2015-05-04 12:36
https___storage.googleapis.com_golang_go1.4.2.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 60M 2015-05-04 12:36
https___storage.googleapis.com_golang_go1.4.1.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 54M 2015-05-04 12:36
https___storage.googleapis.com_golang_go1.2.2.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 54M 2015-05-04 12:36
http___go.googlecode.com_files_go1.2.1.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 51M 2015-05-04 12:36
https___storage.googleapis.com_golang_go1.3.3.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 51M 2015-05-04 12:36
https___storage.googleapis.com_golang_go1.3.2.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 40M 2015-05-04 12:36
http___go.googlecode.com_files_go1.1.2.linux-amd64.tar.gz
-rw-r--r-- 1 flavorjones flavorjones 40M 2015-05-04 12:36
http___go.googlecode.com_files_go1.1.1.linux-amd64.tar.gz
```
One question we should ask, I think, is: should we still be supporting
golang 1.1 and 1.2? Dropping those versions would cut the size of the
buildpack in (approximately) half.
On May 1, 2015, at 11:54 , Mike Dalessio <mdalessio(a)pivotal.io> wrote:
Skinny buildpacks have been cut for go, nodejs, php, python and ruby
buildpacks.
| | current | previous |
|--------+---------+----------|
| go | 442MB | 633MB |
| nodejs | 69MB | 417MB |
| php | 804MB | 1.1GB |
| python | 454MB | 654MB |
| ruby | 365MB | 1.3GB |
|--------+---------+----------|
| total | 2.1GB | 4.1GB |
for an aggregate 51% reduction in size. Details follow.
Next Steps
I recognize that every cloud operator may have a different policy on what
versions of interpreters and libraries they want to support, based on the
specific requirements of their users.
These buildpacks reflect a "bare mininum" policy for a cloud to be
operable, and I do not expect these buildpacks to be adopted as-is by many
operators.
These buildpacks have not yet been added to cf-release, specifically so
that the community can prepare their own buildpacks if necessary.
Over the next few days, the buildpacks core team will ship documentation
and tooling to assist you in packaging specific dependencies for your
instance of CF. I'll start a new thread on this list early next week to
communicate this information.
Call to Action
In the meantime, please think about whether the policy implemented in
these buildpacks ("last two patches (or teenies) on all supported
major.minor releases") is suitable for your users; and if not, think about
what dependencies you'll ideally be supporting.
go-buildpack v1.3.0
Release notes are here
<https://github.com/cloudfoundry/go-buildpack/releases/tag/v1.3.0>.
Size reduced 30% from 633MB
<https://github.com/cloudfoundry/go-buildpack/releases/tag/v1.2.0> to
442MB <https://github.com/cloudfoundry/go-buildpack/releases/tag/v1.3.0>.
Supports (full manifest here
<https://github.com/cloudfoundry/go-buildpack/blob/v1.3.0/manifest.yml>):
- golang 1.4.{1,2}
- golang 1.3.{2,3}
- golang 1.2.{1,2}
- golang 1.1.{1,2}
nodejs-buildpack v1.3.0
Full release notes are here
<https://github.com/cloudfoundry/nodejs-buildpack/releases/tag/v1.3.0>.
Size reduced 83% from 417MB
<https://github.com/cloudfoundry/nodejs-buildpack/releases/tag/v1.2.1> to
69MB
<https://github.com/cloudfoundry/nodejs-buildpack/releases/tag/v1.3.0>.
Supports (full manifest here
<https://github.com/cloudfoundry/nodejs-buildpack/blob/v1.3.0/manifest.yml>
):
- 0.8.{27,28}
- 0.9.{11,12}
- 0.10.{37,38}
- 0.11.{15,16}
- 0.12.{1,2}
php-buildpack v3.2.0
Full release notes are here
<https://github.com/cloudfoundry/php-buildpack/releases/tag/v3.2.0>.
Size reduced 27% from 1.1GB
<https://github.com/cloudfoundry/php-buildpack/releases/tag/v3.1.1> to
803MB <https://github.com/cloudfoundry/php-buildpack/releases/tag/v3.2.0>.
Supports: (full manifest here
<https://github.com/cloudfoundry/php-buildpack/blob/v3.2.0/manifest.yml>)
*PHP*:
- 5.6.{6,7}
- 5.5.{22,23}
- 5.4.{38,39}
*HHVM* (lucid64 stack):
- 3.2.0
*HHVM* (cflinuxfs2 stack):
- 3.5.{0,1}
- 3.6.{0,1}
*Apache HTTPD*:
- 2.4.12
*nginx*:
- 1.7.10
- 1.6.2
- 1.5.13
python-buildpack v1.3.0
Full release notes are here
<https://github.com/cloudfoundry/python-buildpack/releases/tag/v1.3.0>.
Size reduced 30% from 654MB
<https://github.com/cloudfoundry/python-buildpack/releases/tag/v1.2.0> to
454MB
<https://github.com/cloudfoundry/python-buildpack/releases/tag/v1.3.0>.
Supports: (full manifest here
<https://github.com/cloudfoundry/python-buildpack/blob/v1.3.0/manifest.yml>
)
- 2.7.{8,9}
- 3.2.{4,5}
- 3.3.{5,6}
- 3.4.{2,3}
ruby-buildpack v1.4.0
Release notes are here
<https://github.com/cloudfoundry/ruby-buildpack/releases/tag/v1.4.0>.
Size reduced 71% from 1.3GB
<https://github.com/cloudfoundry/ruby-buildpack/releases/tag/v1.3.1> to
365MB <https://github.com/cloudfoundry/ruby-buildpack/releases/tag/v1.4.0>
.
Supports: (full manifest here
<https://github.com/cloudfoundry/ruby-buildpack/blob/v1.4.0/manifest.yml>)
*MRI*:
- 2.2.{1,2}
- 2.1.{5,6}
- 2.0.0p645
*JRuby*:
- ruby-1.9.3-jruby-1.7.19
- ruby-2.0.0-jruby-1.7.19
- ruby-2.2.0-jruby-9.0.0.0.pre1
---------- Forwarded message ----------
From: Mike Dalessio <mdalessio(a)pivotal.io>
Date: Wed, Apr 8, 2015 at 11:10 AM
Subject: Addressing buildpack size
To: vcap-dev(a)cloudfoundry.org
Hello vcap-dev!
This email details a proposed change to how Cloud Foundry buildpacks are
packaged, with respect to the ever-increasing number of binary dependencies
being cached within them.
This proposal's permanent residence is here:
https://github.com/cloudfoundry-incubator/buildpack-packager/issues/4
Feel free to comment there or reply to this email.
------------------------------
Buildpack SizesWhere we are today
Many of you have seen, and possibly been challenged by, the enormous sizes
of some of the buildpacks that are currently shipping with cf-release.
Here's the state of the world right now, as of v205:
php-buildpack: 1.1G
ruby-buildpack: 922M
go-buildpack: 675M
python-buildpack: 654M
nodejs-buildpack: 403M
----------------------
total: 3.7G
These enormous sizes are the result of the current policy of packaging
every-version-of-everything-ever-supported ("EVOEES") within the buildpack.
Most recently, this problem was exacerbated by the fact that buildpacks
now contain binaries for two rootfses.
Why this is a problem
If continued, buildpacks will only continue to increase in size, leading
to longer and longer build and deploy times, longer test times, slacker
feedback loops, and therefore less frequent buildpack releases.
Additionally, this also means that we're shipping versions of
interpreters, web servers, and libraries that are deprecated, insecure, or
both. Feedback from CF users has made it clear that many companies view
this as an unnecessary security risk.
This policy is clearly unsustainable.
What we can do about it
There are many things being discussed to ameliorate the impact that
buildpack size is having on the operations of CF.
Notably, Onsi has proposed a change to buildpack caching, to improve Diego
staging times (link to proposal
<https://github.com/pivotal-cf-experimental/diego-dev-notes/blob/master/proposals/better-buildpack-caching.md>
).
However, there is an immediate solution available, which addresses both
the size concerns as well as the security concern: packaging fewer binary
dependencies within the buildpack.
The proposal
I'm proposing that we reduce the binary dependencies in each buildpack in
a very specific way.
Aside on terms I'll use below:
- Versions of the form "1.2.3" are broken down as: MAJOR.MINOR.TEENY.
Many language ecosystems refer to the "TEENY" as "PATCH" interchangeably,
but we're going to use "TEENY" in this proposal.
- We'll assume that TEENY gets bumped for API/ABI compatible changes.
- We'll assume that MINOR and MAJOR get bumped when there are API/ABI
*incompatible* changes.
I'd like to move forward soon with the following changes:
1. For language interpreters/compilers, we'll package the two
most-recent TEENY versions on each MAJOR.MINOR release.
2. For all other dependencies, we'll package only the single
most-recent TEENY version on each MAJOR.MINOR release.
3. We will discontinue packaging versions of dependencies that have
been deprecated.
4. We will no longer provide "EVOEES" buildpack releases.
5. We will no longer provide "online" buildpack releases, which
download dependencies from the public internet.
6. We will document the process, and provide tooling, for CF operators
to build their own buildpacks, choosing the dependencies that their
organization wants to support or creating "online" buildpacks at operators'
discretion.
An example for #1 is that we'll go from packaging 34 versions of node v0.10.x
to only packaging two: 0.10.37 and 0.10.38.
An example for #2 is that we'll go from packaging 3 versions of nginx 1.5
in the PHP buildpack to only packaging one: 1.5.12.
An example for #3 is that we'll discontinue packaging ruby 1.9.3 in the
ruby-buildpack, which reached end-of-life in February 2015.
Outcomes
With these changes, the total buildpack size will be reduced greatly. As
an example, we expect the ruby-buildpack size to go from 922M to 338M.
We also want to set the expectation that, as new interpreter versions are
released, either for new features or (more urgently) for security fixes,
we'll release new buildpacks much more quickly than we do today. My hope is
that we'll be able to do it within 24 hours of a new release.
Planning
These changes will be relatively easy to make, since all the buildpacks
are now using a manifest.yml file to declare what's being packaged. We
expect to be able to complete this work within the next two weeks.
Stories are in the Tracker backlog under the Epic named
"skinny-buildpacks", which you can see here:
https://www.pivotaltracker.com/epic/show/1747328
------------------------------
Please let me know how these changes will impact you and your
organizations, and let me know of any counter-proposals or variations you'd
like to consider.
Thanks,
-mike
_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
Wayne E. Seguin
What an incredible step in the right direction, Awesome!!!
Out of curiosity, why is the go buildpack still quite so large?
toggle quoted message
Show quoted text
Out of curiosity, why is the go buildpack still quite so large?
On May 1, 2015, at 11:54 , Mike Dalessio <mdalessio(a)pivotal.io> wrote:
Skinny buildpacks have been cut for go, nodejs, php, python and ruby buildpacks.
| | current | previous |
|--------+---------+----------|
| go | 442MB | 633MB |
| nodejs | 69MB | 417MB |
| php | 804MB | 1.1GB |
| python | 454MB | 654MB |
| ruby | 365MB | 1.3GB |
|--------+---------+----------|
| total | 2.1GB | 4.1GB |
for an aggregate 51% reduction in size. Details follow.
Next Steps
I recognize that every cloud operator may have a different policy on what versions of interpreters and libraries they want to support, based on the specific requirements of their users.
These buildpacks reflect a "bare mininum" policy for a cloud to be operable, and I do not expect these buildpacks to be adopted as-is by many operators.
These buildpacks have not yet been added to cf-release, specifically so that the community can prepare their own buildpacks if necessary.
Over the next few days, the buildpacks core team will ship documentation and tooling to assist you in packaging specific dependencies for your instance of CF. I'll start a new thread on this list early next week to communicate this information.
Call to Action
In the meantime, please think about whether the policy implemented in these buildpacks ("last two patches (or teenies) on all supported major.minor releases") is suitable for your users; and if not, think about what dependencies you'll ideally be supporting.
go-buildpack v1.3.0
Release notes are here <https://github.com/cloudfoundry/go-buildpack/releases/tag/v1.3.0>.
Size reduced 30% from 633MB <https://github.com/cloudfoundry/go-buildpack/releases/tag/v1.2.0> to 442MB <https://github.com/cloudfoundry/go-buildpack/releases/tag/v1.3.0>.
Supports (full manifest here <https://github.com/cloudfoundry/go-buildpack/blob/v1.3.0/manifest.yml>):
golang 1.4.{1,2}
golang 1.3.{2,3}
golang 1.2.{1,2}
golang 1.1.{1,2}
nodejs-buildpack v1.3.0
Full release notes are here <https://github.com/cloudfoundry/nodejs-buildpack/releases/tag/v1.3.0>.
Size reduced 83% from 417MB <https://github.com/cloudfoundry/nodejs-buildpack/releases/tag/v1.2.1> to 69MB <https://github.com/cloudfoundry/nodejs-buildpack/releases/tag/v1.3.0>.
Supports (full manifest here <https://github.com/cloudfoundry/nodejs-buildpack/blob/v1.3.0/manifest.yml>):
0.8.{27,28}
0.9.{11,12}
0.10.{37,38}
0.11.{15,16}
0.12.{1,2}
php-buildpack v3.2.0
Full release notes are here <https://github.com/cloudfoundry/php-buildpack/releases/tag/v3.2.0>.
Size reduced 27% from 1.1GB <https://github.com/cloudfoundry/php-buildpack/releases/tag/v3.1.1> to 803MB <https://github.com/cloudfoundry/php-buildpack/releases/tag/v3.2.0>.
Supports: (full manifest here <https://github.com/cloudfoundry/php-buildpack/blob/v3.2.0/manifest.yml>)
PHP:
5.6.{6,7}
5.5.{22,23}
5.4.{38,39}
HHVM (lucid64 stack):
3.2.0
HHVM (cflinuxfs2 stack):
3.5.{0,1}
3.6.{0,1}
Apache HTTPD:
2.4.12
nginx:
1.7.10
1.6.2
1.5.13
python-buildpack v1.3.0
Full release notes are here <https://github.com/cloudfoundry/python-buildpack/releases/tag/v1.3.0>.
Size reduced 30% from 654MB <https://github.com/cloudfoundry/python-buildpack/releases/tag/v1.2.0> to 454MB <https://github.com/cloudfoundry/python-buildpack/releases/tag/v1.3.0>.
Supports: (full manifest here <https://github.com/cloudfoundry/python-buildpack/blob/v1.3.0/manifest.yml>)
2.7.{8,9}
3.2.{4,5}
3.3.{5,6}
3.4.{2,3}
ruby-buildpack v1.4.0
Release notes are here <https://github.com/cloudfoundry/ruby-buildpack/releases/tag/v1.4.0>.
Size reduced 71% from 1.3GB <https://github.com/cloudfoundry/ruby-buildpack/releases/tag/v1.3.1> to 365MB <https://github.com/cloudfoundry/ruby-buildpack/releases/tag/v1.4.0>.
Supports: (full manifest here <https://github.com/cloudfoundry/ruby-buildpack/blob/v1.4.0/manifest.yml>)
MRI:
2.2.{1,2}
2.1.{5,6}
2.0.0p645
JRuby:
ruby-1.9.3-jruby-1.7.19
ruby-2.0.0-jruby-1.7.19
ruby-2.2.0-jruby-9.0.0.0.pre1
---------- Forwarded message ----------
From: Mike Dalessio <mdalessio(a)pivotal.io <mailto:mdalessio(a)pivotal.io>>
Date: Wed, Apr 8, 2015 at 11:10 AM
Subject: Addressing buildpack size
To: vcap-dev(a)cloudfoundry.org <mailto:vcap-dev(a)cloudfoundry.org>
Hello vcap-dev!
This email details a proposed change to how Cloud Foundry buildpacks are packaged, with respect to the ever-increasing number of binary dependencies being cached within them.
This proposal's permanent residence is here:
https://github.com/cloudfoundry-incubator/buildpack-packager/issues/4 <https://github.com/cloudfoundry-incubator/buildpack-packager/issues/4>
Feel free to comment there or reply to this email.
Buildpack Sizes
Where we are today
Many of you have seen, and possibly been challenged by, the enormous sizes of some of the buildpacks that are currently shipping with cf-release.
Here's the state of the world right now, as of v205:
php-buildpack: 1.1G
ruby-buildpack: 922M
go-buildpack: 675M
python-buildpack: 654M
nodejs-buildpack: 403M
----------------------
total: 3.7G
These enormous sizes are the result of the current policy of packaging every-version-of-everything-ever-supported ("EVOEES") within the buildpack.
Most recently, this problem was exacerbated by the fact that buildpacks now contain binaries for two rootfses.
Why this is a problem
If continued, buildpacks will only continue to increase in size, leading to longer and longer build and deploy times, longer test times, slacker feedback loops, and therefore less frequent buildpack releases.
Additionally, this also means that we're shipping versions of interpreters, web servers, and libraries that are deprecated, insecure, or both. Feedback from CF users has made it clear that many companies view this as an unnecessary security risk.
This policy is clearly unsustainable.
What we can do about it
There are many things being discussed to ameliorate the impact that buildpack size is having on the operations of CF.
Notably, Onsi has proposed a change to buildpack caching, to improve Diego staging times (link to proposal <https://github.com/pivotal-cf-experimental/diego-dev-notes/blob/master/proposals/better-buildpack-caching.md>).
However, there is an immediate solution available, which addresses both the size concerns as well as the security concern: packaging fewer binary dependencies within the buildpack.
The proposal
I'm proposing that we reduce the binary dependencies in each buildpack in a very specific way.
Aside on terms I'll use below:
Versions of the form "1.2.3" are broken down as: MAJOR.MINOR.TEENY. Many language ecosystems refer to the "TEENY" as "PATCH" interchangeably, but we're going to use "TEENY" in this proposal.
We'll assume that TEENY gets bumped for API/ABI compatible changes.
We'll assume that MINOR and MAJOR get bumped when there are API/ABI incompatible changes.
I'd like to move forward soon with the following changes:
For language interpreters/compilers, we'll package the two most-recent TEENY versions on each MAJOR.MINOR release.
For all other dependencies, we'll package only the single most-recent TEENY version on each MAJOR.MINOR release.
We will discontinue packaging versions of dependencies that have been deprecated.
We will no longer provide "EVOEES" buildpack releases.
We will no longer provide "online" buildpack releases, which download dependencies from the public internet.
We will document the process, and provide tooling, for CF operators to build their own buildpacks, choosing the dependencies that their organization wants to support or creating "online" buildpacks at operators' discretion.
An example for #1 is that we'll go from packaging 34 versions of node v0.10.x to only packaging two: 0.10.37 and 0.10.38.
An example for #2 is that we'll go from packaging 3 versions of nginx 1.5 in the PHP buildpack to only packaging one: 1.5.12.
An example for #3 is that we'll discontinue packaging ruby 1.9.3 in the ruby-buildpack, which reached end-of-life in February 2015.
Outcomes
With these changes, the total buildpack size will be reduced greatly. As an example, we expect the ruby-buildpack size to go from 922M to 338M.
We also want to set the expectation that, as new interpreter versions are released, either for new features or (more urgently) for security fixes, we'll release new buildpacks much more quickly than we do today. My hope is that we'll be able to do it within 24 hours of a new release.
Planning
These changes will be relatively easy to make, since all the buildpacks are now using a manifest.yml file to declare what's being packaged. We expect to be able to complete this work within the next two weeks.
Stories are in the Tracker backlog under the Epic named "skinny-buildpacks", which you can see here:
https://www.pivotaltracker.com/epic/show/1747328 <https://www.pivotaltracker.com/epic/show/1747328>
Please let me know how these changes will impact you and your organizations, and let me know of any counter-proposals or variations you'd like to consider.
Thanks,
-mike
_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org <mailto:cf-dev(a)lists.cloudfoundry.org>
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev <https://lists.cloudfoundry.org/mailman/listinfo/cf-dev>
Mike Dalessio
Skinny buildpacks have been cut for go, nodejs, php, python and ruby
buildpacks.
| | current | previous |
|--------+---------+----------|
| go | 442MB | 633MB |
| nodejs | 69MB | 417MB |
| php | 804MB | 1.1GB |
| python | 454MB | 654MB |
| ruby | 365MB | 1.3GB |
|--------+---------+----------|
| total | 2.1GB | 4.1GB |
for an aggregate 51% reduction in size. Details follow.
Next Steps
I recognize that every cloud operator may have a different policy on what
versions of interpreters and libraries they want to support, based on the
specific requirements of their users.
These buildpacks reflect a "bare mininum" policy for a cloud to be
operable, and I do not expect these buildpacks to be adopted as-is by many
operators.
These buildpacks have not yet been added to cf-release, specifically so
that the community can prepare their own buildpacks if necessary.
Over the next few days, the buildpacks core team will ship documentation
and tooling to assist you in packaging specific dependencies for your
instance of CF. I'll start a new thread on this list early next week to
communicate this information.
Call to Action
In the meantime, please think about whether the policy implemented in these
buildpacks ("last two patches (or teenies) on all supported major.minor
releases") is suitable for your users; and if not, think about what
dependencies you'll ideally be supporting.
go-buildpack v1.3.0
Release notes are here
<https://github.com/cloudfoundry/go-buildpack/releases/tag/v1.3.0>.
Size reduced 30% from 633MB
<https://github.com/cloudfoundry/go-buildpack/releases/tag/v1.2.0> to 442MB
<https://github.com/cloudfoundry/go-buildpack/releases/tag/v1.3.0>.
Supports (full manifest here
<https://github.com/cloudfoundry/go-buildpack/blob/v1.3.0/manifest.yml>):
- golang 1.4.{1,2}
- golang 1.3.{2,3}
- golang 1.2.{1,2}
- golang 1.1.{1,2}
nodejs-buildpack v1.3.0
Full release notes are here
<https://github.com/cloudfoundry/nodejs-buildpack/releases/tag/v1.3.0>.
Size reduced 83% from 417MB
<https://github.com/cloudfoundry/nodejs-buildpack/releases/tag/v1.2.1> to
69MB <https://github.com/cloudfoundry/nodejs-buildpack/releases/tag/v1.3.0>.
Supports (full manifest here
<https://github.com/cloudfoundry/nodejs-buildpack/blob/v1.3.0/manifest.yml>
):
- 0.8.{27,28}
- 0.9.{11,12}
- 0.10.{37,38}
- 0.11.{15,16}
- 0.12.{1,2}
php-buildpack v3.2.0
Full release notes are here
<https://github.com/cloudfoundry/php-buildpack/releases/tag/v3.2.0>.
Size reduced 27% from 1.1GB
<https://github.com/cloudfoundry/php-buildpack/releases/tag/v3.1.1> to 803MB
<https://github.com/cloudfoundry/php-buildpack/releases/tag/v3.2.0>.
Supports: (full manifest here
<https://github.com/cloudfoundry/php-buildpack/blob/v3.2.0/manifest.yml>)
*PHP*:
- 5.6.{6,7}
- 5.5.{22,23}
- 5.4.{38,39}
*HHVM* (lucid64 stack):
- 3.2.0
*HHVM* (cflinuxfs2 stack):
- 3.5.{0,1}
- 3.6.{0,1}
*Apache HTTPD*:
- 2.4.12
*nginx*:
- 1.7.10
- 1.6.2
- 1.5.13
python-buildpack v1.3.0
Full release notes are here
<https://github.com/cloudfoundry/python-buildpack/releases/tag/v1.3.0>.
Size reduced 30% from 654MB
<https://github.com/cloudfoundry/python-buildpack/releases/tag/v1.2.0> to
454MB <https://github.com/cloudfoundry/python-buildpack/releases/tag/v1.3.0>
.
Supports: (full manifest here
<https://github.com/cloudfoundry/python-buildpack/blob/v1.3.0/manifest.yml>)
- 2.7.{8,9}
- 3.2.{4,5}
- 3.3.{5,6}
- 3.4.{2,3}
ruby-buildpack v1.4.0
Release notes are here
<https://github.com/cloudfoundry/ruby-buildpack/releases/tag/v1.4.0>.
Size reduced 71% from 1.3GB
<https://github.com/cloudfoundry/ruby-buildpack/releases/tag/v1.3.1> to
365MB <https://github.com/cloudfoundry/ruby-buildpack/releases/tag/v1.4.0>.
Supports: (full manifest here
<https://github.com/cloudfoundry/ruby-buildpack/blob/v1.4.0/manifest.yml>)
*MRI*:
- 2.2.{1,2}
- 2.1.{5,6}
- 2.0.0p645
*JRuby*:
- ruby-1.9.3-jruby-1.7.19
- ruby-2.0.0-jruby-1.7.19
- ruby-2.2.0-jruby-9.0.0.0.pre1
---------- Forwarded message ----------
From: Mike Dalessio <mdalessio(a)pivotal.io>
Date: Wed, Apr 8, 2015 at 11:10 AM
Subject: Addressing buildpack size
To: vcap-dev(a)cloudfoundry.org
Hello vcap-dev!
This email details a proposed change to how Cloud Foundry buildpacks are
packaged, with respect to the ever-increasing number of binary dependencies
being cached within them.
This proposal's permanent residence is here:
https://github.com/cloudfoundry-incubator/buildpack-packager/issues/4
Feel free to comment there or reply to this email.
------------------------------
Buildpack SizesWhere we are today
Many of you have seen, and possibly been challenged by, the enormous sizes
of some of the buildpacks that are currently shipping with cf-release.
Here's the state of the world right now, as of v205:
php-buildpack: 1.1G
ruby-buildpack: 922M
go-buildpack: 675M
python-buildpack: 654M
nodejs-buildpack: 403M
----------------------
total: 3.7G
These enormous sizes are the result of the current policy of packaging
every-version-of-everything-ever-supported ("EVOEES") within the buildpack.
Most recently, this problem was exacerbated by the fact that buildpacks now
contain binaries for two rootfses.
Why this is a problem
If continued, buildpacks will only continue to increase in size, leading to
longer and longer build and deploy times, longer test times, slacker
feedback loops, and therefore less frequent buildpack releases.
Additionally, this also means that we're shipping versions of interpreters,
web servers, and libraries that are deprecated, insecure, or both. Feedback
from CF users has made it clear that many companies view this as an
unnecessary security risk.
This policy is clearly unsustainable.
What we can do about it
There are many things being discussed to ameliorate the impact that
buildpack size is having on the operations of CF.
Notably, Onsi has proposed a change to buildpack caching, to improve Diego
staging times (link to proposal
<https://github.com/pivotal-cf-experimental/diego-dev-notes/blob/master/proposals/better-buildpack-caching.md>
).
However, there is an immediate solution available, which addresses both the
size concerns as well as the security concern: packaging fewer binary
dependencies within the buildpack.
The proposal
I'm proposing that we reduce the binary dependencies in each buildpack in a
very specific way.
Aside on terms I'll use below:
- Versions of the form "1.2.3" are broken down as: MAJOR.MINOR.TEENY.
Many language ecosystems refer to the "TEENY" as "PATCH" interchangeably,
but we're going to use "TEENY" in this proposal.
- We'll assume that TEENY gets bumped for API/ABI compatible changes.
- We'll assume that MINOR and MAJOR get bumped when there are API/ABI
*incompatible* changes.
I'd like to move forward soon with the following changes:
1. For language interpreters/compilers, we'll package the two
most-recent TEENY versions on each MAJOR.MINOR release.
2. For all other dependencies, we'll package only the single most-recent
TEENY version on each MAJOR.MINOR release.
3. We will discontinue packaging versions of dependencies that have been
deprecated.
4. We will no longer provide "EVOEES" buildpack releases.
5. We will no longer provide "online" buildpack releases, which download
dependencies from the public internet.
6. We will document the process, and provide tooling, for CF operators
to build their own buildpacks, choosing the dependencies that their
organization wants to support or creating "online" buildpacks at operators'
discretion.
An example for #1 is that we'll go from packaging 34 versions of node v0.10.x
to only packaging two: 0.10.37 and 0.10.38.
An example for #2 is that we'll go from packaging 3 versions of nginx 1.5
in the PHP buildpack to only packaging one: 1.5.12.
An example for #3 is that we'll discontinue packaging ruby 1.9.3 in the
ruby-buildpack, which reached end-of-life in February 2015.
Outcomes
With these changes, the total buildpack size will be reduced greatly. As an
example, we expect the ruby-buildpack size to go from 922M to 338M.
We also want to set the expectation that, as new interpreter versions are
released, either for new features or (more urgently) for security fixes,
we'll release new buildpacks much more quickly than we do today. My hope is
that we'll be able to do it within 24 hours of a new release.
Planning
These changes will be relatively easy to make, since all the buildpacks are
now using a manifest.yml file to declare what's being packaged. We expect
to be able to complete this work within the next two weeks.
Stories are in the Tracker backlog under the Epic named
"skinny-buildpacks", which you can see here:
https://www.pivotaltracker.com/epic/show/1747328
------------------------------
Please let me know how these changes will impact you and your
organizations, and let me know of any counter-proposals or variations you'd
like to consider.
Thanks,
-mike
buildpacks.
| | current | previous |
|--------+---------+----------|
| go | 442MB | 633MB |
| nodejs | 69MB | 417MB |
| php | 804MB | 1.1GB |
| python | 454MB | 654MB |
| ruby | 365MB | 1.3GB |
|--------+---------+----------|
| total | 2.1GB | 4.1GB |
for an aggregate 51% reduction in size. Details follow.
Next Steps
I recognize that every cloud operator may have a different policy on what
versions of interpreters and libraries they want to support, based on the
specific requirements of their users.
These buildpacks reflect a "bare mininum" policy for a cloud to be
operable, and I do not expect these buildpacks to be adopted as-is by many
operators.
These buildpacks have not yet been added to cf-release, specifically so
that the community can prepare their own buildpacks if necessary.
Over the next few days, the buildpacks core team will ship documentation
and tooling to assist you in packaging specific dependencies for your
instance of CF. I'll start a new thread on this list early next week to
communicate this information.
Call to Action
In the meantime, please think about whether the policy implemented in these
buildpacks ("last two patches (or teenies) on all supported major.minor
releases") is suitable for your users; and if not, think about what
dependencies you'll ideally be supporting.
go-buildpack v1.3.0
Release notes are here
<https://github.com/cloudfoundry/go-buildpack/releases/tag/v1.3.0>.
Size reduced 30% from 633MB
<https://github.com/cloudfoundry/go-buildpack/releases/tag/v1.2.0> to 442MB
<https://github.com/cloudfoundry/go-buildpack/releases/tag/v1.3.0>.
Supports (full manifest here
<https://github.com/cloudfoundry/go-buildpack/blob/v1.3.0/manifest.yml>):
- golang 1.4.{1,2}
- golang 1.3.{2,3}
- golang 1.2.{1,2}
- golang 1.1.{1,2}
nodejs-buildpack v1.3.0
Full release notes are here
<https://github.com/cloudfoundry/nodejs-buildpack/releases/tag/v1.3.0>.
Size reduced 83% from 417MB
<https://github.com/cloudfoundry/nodejs-buildpack/releases/tag/v1.2.1> to
69MB <https://github.com/cloudfoundry/nodejs-buildpack/releases/tag/v1.3.0>.
Supports (full manifest here
<https://github.com/cloudfoundry/nodejs-buildpack/blob/v1.3.0/manifest.yml>
):
- 0.8.{27,28}
- 0.9.{11,12}
- 0.10.{37,38}
- 0.11.{15,16}
- 0.12.{1,2}
php-buildpack v3.2.0
Full release notes are here
<https://github.com/cloudfoundry/php-buildpack/releases/tag/v3.2.0>.
Size reduced 27% from 1.1GB
<https://github.com/cloudfoundry/php-buildpack/releases/tag/v3.1.1> to 803MB
<https://github.com/cloudfoundry/php-buildpack/releases/tag/v3.2.0>.
Supports: (full manifest here
<https://github.com/cloudfoundry/php-buildpack/blob/v3.2.0/manifest.yml>)
*PHP*:
- 5.6.{6,7}
- 5.5.{22,23}
- 5.4.{38,39}
*HHVM* (lucid64 stack):
- 3.2.0
*HHVM* (cflinuxfs2 stack):
- 3.5.{0,1}
- 3.6.{0,1}
*Apache HTTPD*:
- 2.4.12
*nginx*:
- 1.7.10
- 1.6.2
- 1.5.13
python-buildpack v1.3.0
Full release notes are here
<https://github.com/cloudfoundry/python-buildpack/releases/tag/v1.3.0>.
Size reduced 30% from 654MB
<https://github.com/cloudfoundry/python-buildpack/releases/tag/v1.2.0> to
454MB <https://github.com/cloudfoundry/python-buildpack/releases/tag/v1.3.0>
.
Supports: (full manifest here
<https://github.com/cloudfoundry/python-buildpack/blob/v1.3.0/manifest.yml>)
- 2.7.{8,9}
- 3.2.{4,5}
- 3.3.{5,6}
- 3.4.{2,3}
ruby-buildpack v1.4.0
Release notes are here
<https://github.com/cloudfoundry/ruby-buildpack/releases/tag/v1.4.0>.
Size reduced 71% from 1.3GB
<https://github.com/cloudfoundry/ruby-buildpack/releases/tag/v1.3.1> to
365MB <https://github.com/cloudfoundry/ruby-buildpack/releases/tag/v1.4.0>.
Supports: (full manifest here
<https://github.com/cloudfoundry/ruby-buildpack/blob/v1.4.0/manifest.yml>)
*MRI*:
- 2.2.{1,2}
- 2.1.{5,6}
- 2.0.0p645
*JRuby*:
- ruby-1.9.3-jruby-1.7.19
- ruby-2.0.0-jruby-1.7.19
- ruby-2.2.0-jruby-9.0.0.0.pre1
---------- Forwarded message ----------
From: Mike Dalessio <mdalessio(a)pivotal.io>
Date: Wed, Apr 8, 2015 at 11:10 AM
Subject: Addressing buildpack size
To: vcap-dev(a)cloudfoundry.org
Hello vcap-dev!
This email details a proposed change to how Cloud Foundry buildpacks are
packaged, with respect to the ever-increasing number of binary dependencies
being cached within them.
This proposal's permanent residence is here:
https://github.com/cloudfoundry-incubator/buildpack-packager/issues/4
Feel free to comment there or reply to this email.
------------------------------
Buildpack SizesWhere we are today
Many of you have seen, and possibly been challenged by, the enormous sizes
of some of the buildpacks that are currently shipping with cf-release.
Here's the state of the world right now, as of v205:
php-buildpack: 1.1G
ruby-buildpack: 922M
go-buildpack: 675M
python-buildpack: 654M
nodejs-buildpack: 403M
----------------------
total: 3.7G
These enormous sizes are the result of the current policy of packaging
every-version-of-everything-ever-supported ("EVOEES") within the buildpack.
Most recently, this problem was exacerbated by the fact that buildpacks now
contain binaries for two rootfses.
Why this is a problem
If continued, buildpacks will only continue to increase in size, leading to
longer and longer build and deploy times, longer test times, slacker
feedback loops, and therefore less frequent buildpack releases.
Additionally, this also means that we're shipping versions of interpreters,
web servers, and libraries that are deprecated, insecure, or both. Feedback
from CF users has made it clear that many companies view this as an
unnecessary security risk.
This policy is clearly unsustainable.
What we can do about it
There are many things being discussed to ameliorate the impact that
buildpack size is having on the operations of CF.
Notably, Onsi has proposed a change to buildpack caching, to improve Diego
staging times (link to proposal
<https://github.com/pivotal-cf-experimental/diego-dev-notes/blob/master/proposals/better-buildpack-caching.md>
).
However, there is an immediate solution available, which addresses both the
size concerns as well as the security concern: packaging fewer binary
dependencies within the buildpack.
The proposal
I'm proposing that we reduce the binary dependencies in each buildpack in a
very specific way.
Aside on terms I'll use below:
- Versions of the form "1.2.3" are broken down as: MAJOR.MINOR.TEENY.
Many language ecosystems refer to the "TEENY" as "PATCH" interchangeably,
but we're going to use "TEENY" in this proposal.
- We'll assume that TEENY gets bumped for API/ABI compatible changes.
- We'll assume that MINOR and MAJOR get bumped when there are API/ABI
*incompatible* changes.
I'd like to move forward soon with the following changes:
1. For language interpreters/compilers, we'll package the two
most-recent TEENY versions on each MAJOR.MINOR release.
2. For all other dependencies, we'll package only the single most-recent
TEENY version on each MAJOR.MINOR release.
3. We will discontinue packaging versions of dependencies that have been
deprecated.
4. We will no longer provide "EVOEES" buildpack releases.
5. We will no longer provide "online" buildpack releases, which download
dependencies from the public internet.
6. We will document the process, and provide tooling, for CF operators
to build their own buildpacks, choosing the dependencies that their
organization wants to support or creating "online" buildpacks at operators'
discretion.
An example for #1 is that we'll go from packaging 34 versions of node v0.10.x
to only packaging two: 0.10.37 and 0.10.38.
An example for #2 is that we'll go from packaging 3 versions of nginx 1.5
in the PHP buildpack to only packaging one: 1.5.12.
An example for #3 is that we'll discontinue packaging ruby 1.9.3 in the
ruby-buildpack, which reached end-of-life in February 2015.
Outcomes
With these changes, the total buildpack size will be reduced greatly. As an
example, we expect the ruby-buildpack size to go from 922M to 338M.
We also want to set the expectation that, as new interpreter versions are
released, either for new features or (more urgently) for security fixes,
we'll release new buildpacks much more quickly than we do today. My hope is
that we'll be able to do it within 24 hours of a new release.
Planning
These changes will be relatively easy to make, since all the buildpacks are
now using a manifest.yml file to declare what's being packaged. We expect
to be able to complete this work within the next two weeks.
Stories are in the Tracker backlog under the Epic named
"skinny-buildpacks", which you can see here:
https://www.pivotaltracker.com/epic/show/1747328
------------------------------
Please let me know how these changes will impact you and your
organizations, and let me know of any counter-proposals or variations you'd
like to consider.
Thanks,
-mike